Mobio Dataset, female subjects

Dataset Description

Mobio is a collection of English voice recordings. The set of female subjects contains:

		Identities	Sample count
train		13	2406
dev	references	18	90
	probes		1890
eval	references	20	100
	probes		2100

GMM

To run the baseline, use the following command:

$ bob bio pipeline simple -d mobio-audio-female gmm-mobio -g dev -g eval -l sge -o results/gmm_mobio_female

Then, to generate the scores, use:

$ bob bio metrics -e ./results/gmm_mobio_female/scores-{dev,eval}.csv

Table 7 [Min. criterion: EER] Threshold on Development set: 7.550647e-01

	Development	Evaluation
Failure to Acquire	0.0%	0.0%
False Match Rate	20.6% (6632/32130)	7.8% (3093/39900)
False Non Match Rate	20.6% (390/1890)	26.8% (562/2100)
False Accept Rate	20.6%	7.8%
False Reject Rate	20.6%	26.8%
Half Total Error Rate	20.6%	17.3%

On 1281 CPU nodes on the SGE Grid: Ran in 15 minutes (11 minutes of training).

ISV

To run the baseline, use the following command:

$ bob bio pipeline simple -d mobio-audio-female -p isv-voxforge -g dev -g eval -l sge -o results/isv_mobio_female

Then, to generate the scores, use:

$ bob bio metrics -e ./results/isv_mobio_female/scores-{dev,eval}.csv

Table 8 [Min. criterion: EER] Threshold on Development set: 3.483318e-01

	Development	Evaluation
Failure to Acquire	0.0%	0.0%
False Match Rate	14.7% (4710/32130)	20.3% (8103/39900)
False Non Match Rate	14.7% (277/1890)	17.4% (366/2100)
False Accept Rate	14.7%	20.3%
False Reject Rate	14.7%	17.4%
Half Total Error Rate	14.7%	18.9%

On 1281 CPU nodes on the SGE Grid: Ran in 8 minutes (3 minutes of training).

Speechbrain ECAPA-TDNN

To run the baseline, use the following command:

$ bob bio pipeline simple -d mobio-audio-female -p speechbrain-ecapa-voxceleb -g dev -g eval -l sge -o results/speechbrain_mobio_female

Then, to generate the scores, use:

$ bob bio metrics -e ./results/speechbrain_mobio_female/scores-{dev,eval}.csv

Table 9 [Min. criterion: EER] Threshold on Development set: -5.091601e-01

	Development	Evaluation
Failure to Acquire	0.0%	0.0%
False Match Rate	1.9% (616/32130)	10.8% (4307/39900)
False Non Match Rate	1.9% (36/1890)	2.5% (52/2100)
False Accept Rate	1.9%	10.8%
False Reject Rate	1.9%	2.5%
Half Total Error Rate	1.9%	6.6%

On 1281 CPU nodes on the SGE Grid: 12 minutes (no training).

Footnotes

1(1,2,3)

The number of nodes is a requested maximum amount and can vary depending on the number of jobs currently running on the grid as well as the scheduler’s load estimation. The execution time can then also vary.