Manhob HCI-Tagging Database¶

This package contains an interface for the Mahnob HCI-Tagging dataset interface. It is presently used to benchmark and test Remote Photo-Plethysmography algorithms at Idiap. This package only uses the colored videos (from Camera 1, in AVI format) and the biological signals saved in BDF format.

If you decide to use this package, please consider citing Bob, as a software development environment and the authors of the dataset:

@article{soleymani-2012,
  author={Soleymani, M. and Lichtenauer, J. and Pun, T. and Pantic, M.},
  journal={Affective Computing, IEEE Transactions on},
  title={A Multimodal Database for Affect Recognition and Implicit Tagging},
  year={2012},
  volume={3},
  number={1},
  pages={42-55},
  doi={10.1109/T-AFFC.2011.25},
  month=Jan,
  }

Dependencies¶

This package makes use of the following important external dependencies (aside from Bob):

mne: For estimating the heart-rate in beats-per-minute using the Pam-Tompkins algorithm

Python-EDF tools: to read physiological sensor information out of BDF files

Development¶

This package can, optionally, automatically annotate the following key aspects of the Mahnob HCI-Tagging dataset:

Average heart-rate in beats-per-minute (BPM), using the Pam-Tompkins algorithm as implemented by mne.

Face bounding boxes, as detected by the default detector on bob.ip.facedetect.

Warning

Note this procedure is outdated by current metadata which is already shipped with this package. Only use it in case you know what you’re doing and/or want to modify/re-evaluate this package’s metadata.

For it to work properly, you’ll need to modify the method bob.db.hci_tagging.File.load_face_detection() to take it into account. As of today, it is set to load face detections from the HDF5 files distributed with this package.

The annotation procedure can be launched with the following command:

$ bob_dbmanage.py hci_tagging mkmeta

Each video, which is composed of a significant number of frames (hundreds), takes about 5 minutes to get completely processed. If are at Idiap, you can launch the job on the SGE queue using the following command-line:

$ jman sub -q q1d --io-big -t 3490 `which bob_dbmanage.py` hci_tagging mkmeta

API¶

bob.db.hci_tagging.resource_filename(package_or_requirement, resource_name)¶: Return a true filesystem path for specified resource

class bob.db.hci_tagging.Database[source]¶

Bases: object

objects(protocol='all', subset=None)[source]¶

Returns a list of unique File objects for the specific query by the user.

Parameters

protocol (str, optional) – If set, can take the value of either cvpr14 or all. cvpr14 subselects samples used by Li et al. on their CVPR``14 paper for heart-rate estimation. If all is set, the complete database is selected.
subset (str, optional) – If set, it could be either train, dev or test or a combination of them (i.e. a list). If not set (default), the files from all these sets are retrieved for the all protocol. Note that for the cvpr14 protocol, this has no effect, since no training, development and test set have been defined in this case.

Returns

A list of File objects.

Return type

list

class bob.db.hci_tagging.File(basedir, bdf, video, duration)¶

Bases: bob.db.base.File

Generic file container for HCI-Tagging files

Parameters

basedir (str) – The base directory for the data
bdf (str) – The name of the BDF file that accompanies the video file, containing the physiological signals.
video (str) – The name of the video file to be used
duration (int) – The time in seconds that corresponds to the estimated duration of the data (video and physiological signals).

default_extension()[source]¶

estimate_heartrate_in_bpm(directory)[source]¶

Estimates the person’s heart rate using the ECG sensor data

Parameters: directory (str) – A directory name that leads to the location the database is installed on the local disk

load(directory=None, extension='.avi')[source]¶

Loads the video for this file entry

Parameters: directory (str) – The path to the root of the database installation. This is the path leading to the directory Sessions of the database.
Returns: A 4D array of 8-bit unsigned integers corresponding to the input video for this file in (frame,channel,y,x) notation (Bob-style).
Return type: numpy.ndarray

load_drmf_keypoints()[source]¶

Loads the 66-keypoints coming from the Discriminative Response Map Fitting (DRMF) landmark detector. Raises if metadata file isn’t there.

Reference: http://ibug.doc.ic.ac.uk/resources/drmf-matlab-code-cvpr-2013/.

The code was written for Matlab. Data for the first frame of the colour video of this object was loaded on a compatible Matlab framework and the keypoints extracted taking as basis the currently available face bounding box, enlarged by 7% (so the key-point detector performs reasonably well). The extracted keypoints were then merged into this database access package so they are easy to load from python.

The points are in the form (y, x), as it is standard on Bob-based packages.

load_face_detection()[source]¶

Load bounding boxes for this file

This function loads bounding boxes for each frame of a video sequence. Bounding boxes are loaded from the package directory and are the ones provided with it. Bounding boxes generated from run_face_detector() (which should be exactly the same) are not used by this method.

Returns: A dictionary where the key is the frame number and the values are instances of bob.ip.facedetect.BoundingBox.
Return type: dict

load_heart_rate_in_bpm()[source]¶: Loads heart-rate from locally stored files, raises if it isn’t there

load_video(directory)[source]¶

Loads the colored video file associated to this object

Parameters: directory (str) – A directory name that will be prefixed to the returned result.

Returns

bob.io.video.reader: Preloaded and ready to be iterated by your code.

make_path(directory=None, extension=None)[source]¶

Wraps this files’ filename so that a complete path is formed

Keyword parameters:

directory: An optional directory name that will be prefixed to the returned result.
extension: An optional extension that will be suffixed to the returned filename. The extension normally includes the leading . character as in .png or .bmp. If not specified the default extension for the original file in the database will be used

Returns a string containing the newly generated file path.

run_face_detector(directory, max_frames=0)[source]¶

Runs bob.ip.facedetect stock detector on the selected frames.

Warning

This method is deprecated and serves only as a development basis to clean-up the load_face_detection(), which for now relies on .face files shipped with this package. Technically, the output of this method and the detected faces shipped should be the same as of today, 13 december 2016.

Parameters

directory (str) – A directory name that leads to the location the database is installed on the local disk
max_frames (int) – If set, delimits the maximum number of frames to treat from the associated video file. A value of zero (default), makes the detector run for all frames.

Returns

A dictionary where the key is the frame number and the values are instances of bob.ip.facedetect.BoundingBox.

Return type

dict

save(data, directory=None, extension='.hdf5')[source]¶

Saves the input data at the specified location and using the given extension.

Keyword parameters:

data: The data blob to be saved (normally a numpy.ndarray).
directory: If not empty or None, this directory is prefixed to the final file destination
extension: The extension of the filename - this will control the type of output and the codec for saving the input blob.