NIST-SRE04-16 Dataset¶
Dataset Description¶
This is an aggregation of the NIST-SRE datasets from 2004 to 2016.
Identities |
Sample count |
||
train |
6213 |
71728 |
|
dev |
references |
80 |
120 |
probes |
5 |
1207 |
|
eval |
references |
802 |
1202 |
probes |
5 |
9294 |
GMM¶
To run the baseline, use the following command:
$ bob bio pipeline simple -d nist-sre04to16 -p gmm-nist -g dev -g eval -l sge -o results/gmm_nist
Then, to generate the scores, use:
$ bob bio metrics -e ./results/gmm_nist/scores-{dev,eval}.csv
Development |
Evaluation |
On 1281 CPU nodes on the SGE Grid: TODO
ISV¶
To run the baseline, use the following command:
$ bob bio pipeline simple -d nist-sre04to16 -p isv-nist -g dev -g eval -l sge -o results/isv_nist
Then, to generate the scores, use:
$ bob bio metrics -e ./results/isv_nist/scores-{dev,eval}.csv
Development |
Evaluation |
On 1281 CPU nodes on the SGE Grid: TODO
Speechbrain ECAPA-TDNN¶
To run the baseline, use the following command:
$ bob bio pipeline simple -d nist-sre04to16 -p speechbrain-ecapa-voxceleb -g dev -g eval -l sge -o results/speechbrain_nist
Then, to generate the scores, use:
$ bob bio metrics -e ./results/speechbrain_mobio_male/scores-{dev,eval}.csv
Development |
Evaluation |
|
---|---|---|
Failure to Acquire |
0.0% |
0.0% |
False Match Rate |
12.9% (12434/96342) |
11.4% (852522/7453619) |
False Non Match Rate |
12.8% (28/218) |
23.7% (40/169) |
False Accept Rate |
12.9% |
11.4% |
False Reject Rate |
12.8% |
23.7% |
Half Total Error Rate |
12.9% |
17.6% |
On 701 CPU nodes on the SGE Grid: Ran in 55 minutes (no training).
Footnotes