NIST-SRE04-16 Dataset

Dataset Description

This is an aggregation of the NIST-SRE datasets from 2004 to 2016.

		Identities	Sample count
train		6213	71728
dev	references	80	120
	probes	5	1207
eval	references	802	1202
	probes	5	9294

GMM

To run the baseline, use the following command:

$ bob bio pipeline simple -d nist-sre04to16 -p gmm-nist -g dev -g eval -l sge -o results/gmm_nist

Then, to generate the scores, use:

$ bob bio metrics -e ./results/gmm_nist/scores-{dev,eval}.csv

Table 10 [Min. criterion: EER] Threshold on Development set: TODO

Development

Evaluation

On 1281 CPU nodes on the SGE Grid: TODO

ISV

To run the baseline, use the following command:

$ bob bio pipeline simple -d nist-sre04to16 -p isv-nist -g dev -g eval -l sge -o results/isv_nist

Then, to generate the scores, use:

$ bob bio metrics -e ./results/isv_nist/scores-{dev,eval}.csv

Table 11 [Min. criterion: EER] Threshold on Development set: TODO

Development

Evaluation

On 1281 CPU nodes on the SGE Grid: TODO

Speechbrain ECAPA-TDNN

To run the baseline, use the following command:

$ bob bio pipeline simple -d nist-sre04to16 -p speechbrain-ecapa-voxceleb -g dev -g eval -l sge -o results/speechbrain_nist

Then, to generate the scores, use:

$ bob bio metrics -e ./results/speechbrain_mobio_male/scores-{dev,eval}.csv

Table 12 [Min. criterion: EER ] Threshold on Development set: -3.860876e-01

	Development	Evaluation
Failure to Acquire	0.0%	0.0%
False Match Rate	12.9% (12434/96342)	11.4% (852522/7453619)
False Non Match Rate	12.8% (28/218)	23.7% (40/169)
False Accept Rate	12.9%	11.4%
False Reject Rate	12.8%	23.7%
Half Total Error Rate	12.9%	17.6%

On 701 CPU nodes on the SGE Grid: Ran in 55 minutes (no training).

Footnotes

1(1,2,3)

The number of nodes is a requested maximum amount and can vary depending on the number of jobs currently running on the grid as well as the scheduler’s load estimation. The execution time can then also vary.