.. vim: set fileencoding=utf-8 : .. Pavel Korshunov .. Thu 23 Jun 13:43:22 2016 .. _bob.report.deepfakes: =========================================================== Vulnerability assessment and detection of Deepfake videos =========================================================== Database ======== The database of Deepfake videos that is used in the experiments is DeepfakeTIMIT_. It is important to note that we do not provide the codebase we used to create DeepfakeTIMIT database. We have used an open souce GitHub_ code but we have heavily modified it to streamline the batch training and video generation of Deepfake models. And while we released the resulted video data to facilitate creation of Deepfake detectors, we would rather not make it even easier to generate such videos. Usage instructions ================== This package provides the source code for reproducing the experimental results presented in the above papers. The first part of the experiments demonstrates the vulnerability of state of the art face recognition systems to Deepfake videos, i.e., the systems get confused by the Deepfake video assuming the video contains the person who was swapped instead of the original speaker. And the second part of experiments shows the Deepfake detectors, best of which is based on image quality features and SVM classifier. Before running the experiments, please set the path to DeepfakeTIMIT database inside file `path_to_data.txt` like so: * [SAVI_DATA_DIRECTORY]=/path/to/the/folder/where/DeepfakeTIMIT/is Also, please note that the original `VidTIMIT` database is also necessary to run the experiments and the assumption is that `VidTIMIT` will be in the same folder where `DeepfakeTIMIT` is. So, inside `path_to_data.txt` please point to the folder with two subfolders: `DeepfakeTIMIT` and `VidTIMIT`. Also, note that video data in `VidTIMIT` is stored as a collection of images, so they should be converted to `AVI` files before advancing further. Vulnerability of face recognition --------------------------------- You can modify the parameters of the vulnerability experiments in these config files inside `bob.report.deepfakes.config` folder: `vuln_alg_cosine.py`, `vuln_db_vdtfaceswap.py`, `vuln_extr_facenet.py`, and `vuln_extr_vgg.py`. Run the experiments using `FaceNet`-based face recognition system as follows: .. code:: sh $ bin/verify.py bob/report/deepfakes/config/vuln_db_vdtfaceswap.py bob/report/deepfakes/config/vuln_extr_facenet.py \ bob/report/deepfakes/config/vuln_alg_cosine.py --protocol oneset-spoof $ bin/verify.py bob/report/deepfakes/config/vuln_db_vdtfaceswap.py bob/report/deepfakes/config/vuln_extr_facenet.py \ bob/report/deepfakes/config/vuln_alg_cosine.py --protocol oneset-licit The scores for `licit` (normal face recognition without Deepfakes) and `spoof` (when Deepfake videos are used as probes) scenarios will be located inside `./results/facenet-cosine` folder. Measure vulnerability of the system by running the following: .. code:: sh $ bob bio metrics -e ./results/facenet-cosine/oneset-licit/nonorm/scores-dev \ ./results/facenet-cosine/oneset-spoof/nonorm/scores-dev This command will take scores from `licit` scenario as a development scores and determine from them the threshold to separate genuine scores from zero effort impostors. Then, apply this threshold to the `spoof` scenario scores, where the two sets are genuine scores and scores from Deepfakes. The `Half Total Error Rate` metric (it's equal error rate in this case) should be `0.0%` for the Development column (the `licit` scenario) and `45.3%` (FAR is 90.6%) for Evaluation column. To run the experiments for VGG-based face recognition, instead of `vuln_extr_facenet.py` config, use `vuln_extr_vgg.py` when running `bin/verify.py` script. To run the experiments for lower qualitu Deepfakes, inside `vuln_db_vdtfaceswap.py` replace the line `db_name='deepfaketimit_hq'` with the line `db_name='deepfaketimit_lq'`. Detection with image quality features ------------------------------------- You can modify the parameters of the Deepfake detection experiments in these config files inside `bob.report.deepfakes.config` folder: `pad_db_vdtfaceswap.py`, `pad_extr_iqm_alg_pca_lda.py`, `pad_extr_iqm_alg_svm.py`, and `pad_extr_lin_alg_pca_lda.py`. To run an IQM-SVM-based Deepfake detector, run the following: .. code:: sh $ bin/spoof.py bob/report/deepfakes/config/pad_db_vdtfaceswap.py bob/report/deepfakes/config/pad_extr_iqm_alg_svm.py This script will produce the scores and place them inside `./results/pad-iqm-svm` folder. To measure the detection accuracy of the Deepfakes, please run the following command: .. code:: sh $ bob pad metrics -e ./results/pad-iqm-svm/train_dev/scores/scores-train pad-iqm-svm/train_dev/scores/scores-dev This command will take the scores computed for the training set (the corresponding score column is called `Development` because we only have two sets for this database), determine the threshold for them and use the threshold to compute the metrics for the development set (but the corresponding column from `bob pad metrics` is Evaluation). The resulted `HTER` metric should be 0% for the first set (Training/Development scores) and 12.4% for Development/Evaluation scores. .. Place your references here: .. _bob: https://www.idiap.ch/software/bob .. _GitHub: https://github.com/shaoanlu/faceswap-GAN .. _DeepfakeTIMIT: https://www.idiap.ch/dataset/deepfaketimit .. _installation: https://www.idiap.ch/software/bob/install .. _conda: https://conda.io