.. author: Yannick Dayer .. date: 2021-04-14 09:39:37 +02 .. _bob.bio.base.vulnerability_analysis: ====================== Vulnerability analysis ====================== When developing a biometric system, one could be interested in knowing how it performs when under attack. This is where the vulnerability to presentation attacks comes in. This analysis allows to test a variety of attacks and assess the cases where a biometric system is fooled. For more details on the vulnerability analysis of biometric systems, refer to [Chingovska_SPRINGER_2019]_. To evaluate a biometric system for vulnerability to presentation attack, three steps must be accomplished: - Preparing the data to feed the biometric system (using a ``CSVDataset``); - Running a pipeline to generate comparison scores (``bob bio pipelines``); - Analyzing the resulting scores to assess the performance of the system (``bob vulnerability`` commands). Using a ``CSVDataset`` ---------------------- A ``CSVDataset`` interface is needed to correctly feed the data to the PipelineSimple. You can use an already existing configuration (like ``replaymobile-img``), when present. If not, you will need to create your own. To do so, you need at least those three elements: - The raw data files (e.g. face images or videos); - A series of CSV protocol files; - A configuration file instantiating the ``CSVDataset`` with your parameters. In some specific cases (e.g. frames needing to be extracted from a video file), the base ``CSVDataset`` could not satisfy your needs, and a new class inheriting from it would have to be implemented. Data files ^^^^^^^^^^ The data files are never provided via the bob packages, as they contain private biometric information that can not be distributed. You have to retrieve them from their author (sometimes requiring a consent form to be approved) or create your own. CSV protocol files ^^^^^^^^^^^^^^^^^^ These files define which sample from the data files will be used and for which purpose (training, enrollment, or probe). A specific file structure must be observed, as well as some formatting rules in the files. File structure of the CSV protocol files ........................................ The CSV protocol files must be ordered in the following structure with the correct file names: .. code-block:: text | +-- | | | +-- dev | | | +-- for_models.csv | +-- for_probes.csv | +-- . | . +-- norm . | | | +-- train_world.csv | +-- dev | | | +-- for_models.csv | +-- for_probes.csv | +-- eval | +-- for_models.csv +-- for_probes.csv Each subfolder of the top-level folder contains a protocol definition whose name can be chosen and will be compared to the ``protocol_name`` parameter passed in the ``CSVDataset`` configuration. Each protocol folder must contain at least a ``dev`` folder and can contain a ``norm`` and/or an ``eval`` folder. The ``dev`` and ``eval`` folders each contain a ``for_models.csv`` file that will be used to enroll the samples listed in it, and a ``for_probes.csv`` file that contains the genuine and attack probes samples that will be compared to the enrolled models. CSV protocol files format ......................... Each CSV file must contain a header and its content must be separated by commas. The required columns for vulnerability analysis are ``path`` and ``reference_id``. Other column fields (metadata) will be processed by the ``CSVDataset`` class and passed through the samples to the pipeline. Each row of the CSV files will result in **one** sample being created (each frame of a video must be one row when processing images). The ``attack_type`` field can be omitted in the model files but must be present in the probe files. When the sample is not an attack, the ``attack_type`` field should be empty, otherwise, it represents the type of attack used. Minimal protocol definition CSV file for the enrollment (``for_models.csv``): .. code-block:: text PATH,REFERENCE_ID enroll/client01/im01,1 enroll/client01/im02,1 enroll/client02/im01,2 enroll/client02/im02,2 ... Minimal vulnerability probes protocol definition CSV file (``for_probes.csv``), where each row corresponds to **one comparison** (``is_sparse`` needs to be set in the ``CSVDataset``): .. code-block:: text PATH, REFERENCE_ID, ATTACK_TYPE, COMPARE_REFERENCE_ID probe/client01/im01,1,,1 probe/client01/im01,1,,2 probe/client01/im01,1,,3 probe/client01/im01,1,,4 probe/client01/im01,1,spoof,1 probe/client02/im01,2,,1 probe/client02/im01,2,,2 probe/client02/im01,2,,3 probe/client02/im01,2,,4 probe/client02/im01,2,spoof,2 ... Note that building the probe file this way (instead of one rew per probe) is recommended to prevent the computation of attack probes against unrelated references. The ``compare_reference_id`` specifies to which reference the probe will be compared to. The ``CSVDataset`` interface ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you want to use a dataset that is already configured for bob vulnerability you don't need to create the configuration file in this step; you can use the one provided in the bob package (see next chapter, using ``replaymobile-img``). Otherwise, once you have the protocol files and the data, you can define a configuration file to input in the PipelineSimple pipeline. This configuration file must assign a ``database`` object to an instance of :py:class:`bob.bio.base.database.CSVDataset` (or your own inheriting class if some special processing is needed for the loading of your samples). To use the default ``CSVDataset`` for your dataset, the python configuration file to use can be in the form: .. code-block:: python from bob.bio.base.database import CSVDataset, CSVToSampleLoaderBiometrics database = CSVDataset( dataset_protocol_path="path/to/definition/of/your_dataset_name", protocol_name="your_protocol_name", sample_loader=CSVToSampleLoaderBiometrics( data_loader=bob.io.base.load, dataset_original_directory="path/to/raw_data_folder", extension=".png", ), is_sparse=True, ) The ``is_sparse`` parameter is used in pair with the ``compare_reference_id`` column in the CSV definition files. It allows to define specifically which comparisons should be computed, by defining each row as one comparison between a probe sample and the reference specified in the ``compare_reference_id`` column. The custom configuration file (``custom_database_config.py``, here) has to be given as parameter to the PipelineSimple command: .. code-block:: sh $ bob bio pipeline simple custom_database_config.py facenet-sanderberg Computing the scores -------------------- To evaluate a biometric system, a series of samples coming from a dataset is compared to multiple enrolled models, and each comparison should result in a score. Generally, the score should be high if the compared samples come from the same individual represented in the model. In the case of vulnerability analysis, presentation attack samples are also compared to the enrolled model of the attacked individual and give a score. Ideally, the system should not recognize the attack sample and return a low score to those comparisons. But if the attack is well crafted enough to fool the system, these attack samples will pass as genuine and score higher. To compute the comparison scores of a dataset on a system, the PipelineSimple pipeline can be used. For a face presentation attack analysis (using the `replay mobile` dataset as an example) on a face recognition system (`Facenet` from Sandberg, in this case), use the following command: .. code-block:: sh $ bob bio pipeline simple -v --output ./results \ --group dev --group eval replaymobile-img facenet-sanderberg ``replaymobile-img`` points to a python configuration file defining a ``database`` object. ``facenet-sanderberg`` points to a python configuration file defining a ``pipeline`` object. .. note:: The data (face images) is **not** provided in the bob packages, as it contains private information. Only the protocol definition files and annotations are available and will be automatically downloaded unless specified, as they are anonymous. You must acquire the data files and specify their location using the ``bob config`` command, or specifying it in your own configuration files. When using ``replaymobile-img``, do: .. code-block:: sh $ bob config set bob.db.replaymobile.directory /your/path/to/replay-mobile/database Analyzing the scores -------------------- To evaluate the performance of the system, we can analyze the distribution of scores generated by the ``pipelines`` command. This analysis step includes a series of commands that generate plots and tables of metrics. Those are useful to assess the performance and vulnerability of the system or to compare multiple systems' performance. Each command is in the form: .. code-block:: sh $ bob vulnerability or alternatively: .. code-block:: sh $ bob vuln For a list of available commands, run: .. code-block:: sh $ bob vuln --help For more information on a specific command (available options, number of score files needed), you can use the integrated help option available for each command: .. code-block:: sh $ bob vuln metrics --help Metrics ^^^^^^^ The ``metrics`` command generates a list of useful metrics (FMR, FNMR, IAPMR, etc.) for a specific operating point (threshold value). It is possible to specify a value for the threshold, or a criterion can be used to compute this value automatically by minimizing an error rate. This command is useful to get a quick evaluation of a system on a single operating point. Example: .. code-block:: sh $ bob vuln metrics --eval results/scores-{dev,eval}.csv Output: .. code-block:: text [Min. criterion: EER ] Threshold on Development set `results/scores-dev.csv`: -4.150729e-01 ============================== ================== ================= .. Development Evaluation ============================== ================== ================= Licit Failure to Acquire 0.0% 0.4% Licit False Match Rate 0.1% (30/24000) 0.0% (0/12056) Licit False Non Match Rate 0.1% (2/1600) 0.2% (2/1096) Licit False Accept Rate 0.1% 0.0% Licit False Reject Rate 0.2% 0.6% Licit Half Total Error Rate 0.1% 0.1% Attack Presentation Match Rate 100.0% (2548/2549) 99.9% (1901/1902) ============================== ================== ================= Histograms ^^^^^^^^^^ The ``hist`` command plots the different distributions (positives, negatives, as well as spoof) of the scores, allowing to visualize if a biometric system can distinguish impostors and attacks from genuine samples. The threshold value for EER can be displayed. When using a development and evaluation set, the threshold value is computed only on the dev set and reported to the eval graph. Example: .. code-block:: sh $ bob vuln hist --eval results/scores-{dev,eval}.csv --output results/hist.pdf --figsize "6,5" Output: .. figure:: img/vuln_plots/hist.png :figwidth: 95% :align: center :alt: Histogram of vulnerability scores. Histogram of genuine, zero-effort impostor, and attack impostor scores. ROC and DET ^^^^^^^^^^^ The ``roc`` and ``det`` commands plot the FMR against the FNMR of a system allowing for example to assess the resulting FMR for a wanted minimum FNMR value. For vulnerability, the IAPMR is also plotted against the FNMR. Example: .. code-block:: sh $ bob vuln roc --eval results/scores-{dev,eval}.csv --output results/roc.pdf --figsize "6,4" Output: .. figure:: img/vuln_plots/roc.png :figwidth: 95% :align: center :alt: ROC of vulnerability scores. ROC of `dev` and `eval` groups. The annotated threshold value is chosen on `dev`. IAPMR vs FMR ^^^^^^^^^^^^ The ``fmr-iapmr`` command plots the IAPMR against the FMR, allowing to see the ratio of accepted attacks given an FMR value. Example: .. code-block:: sh $ bob vuln fmr-iapmr results/scores-{dev,eval}.csv --output results/fmr_iapmr.pdf --legends "replay-mobile" Output: .. figure:: img/vuln_plots/fmr_iapmr.png :figwidth: 75% :align: center :alt: FMR vs IAPMR of vulnerability scores. Plot of the IAPMR vs the FMR for various threshold values. EPSC ^^^^ The ``epsc`` command plots the WER and IAPMR for different values of :math:`\beta` and :math:`\omega` parameters used to compute this error rate. Since two variables are in play, one of them can be set and the other plotted, or a three-dimensional plot can be drawn. It is also possible to only draw the WER or the IAPMR plot. Example: .. code-block:: sh $ bob vuln epsc results/scores-{dev,eval}.csv --output results/epsc.pdf --fixed-params "0.5,0.6" --figsize "8,4" Output: .. figure:: img/vuln_plots/epsc.png :figwidth: 95% :align: center :alt: EPSC of vulnerability scores. EPSC with :math:`\omega` varying and :math:`\beta` set at 0.5 and 0.6. EPC ^^^ The ``epc`` command plots the EPC of the system with the IAPMR overlaid on top. Example: .. code-block:: sh $ bob vuln epc results/scores-{dev,eval}.csv --output results/epc.pdf Output: .. figure:: img/vuln_plots/epc.png :figwidth: 75% :align: center :alt: EPC of vulnerability scores. EPC of the system with the corresponding IAPMR curve. Evaluate ^^^^^^^^ The ``evaluate`` command creates a single report with multiple plots to display different aspects of the evaluation. Example: .. code-block:: sh $ bob vuln evaluate results/scores-{dev,eval}.csv --output results/report.pdf References ---------- .. [Chingovska_SPRINGER_2019] * Chingovska, Ivana and Mohammadi, Amir and Anjos, André and Marcel, Sébastien **Evaluation methodologies for biometric presentation attack detection**, 2019, Springer