Manhob HCI-Tagging Database

This package contains an interface for the Mahnob HCI-Tagging dataset interface. It is presently used to benchmark and test Remote Photo-Plethysmography algorithms at Idiap. This package only uses the colored videos (from Camera 1, in AVI format) and the biological signals saved in BDF format.

If you decide to use this package, please consider citing Bob, as a software development environment and the authors of the dataset:

  author={Soleymani, M. and Lichtenauer, J. and Pun, T. and Pantic, M.},
  journal={Affective Computing, IEEE Transactions on},
  title={A Multimodal Database for Affect Recognition and Implicit Tagging},


This package makes use of the following important external dependencies (aside from Bob):

  • mne: For estimating the heart-rate in beats-per-minute using the Pam-Tompkins algorithm

  • Python-EDF tools: to read physiological sensor information out of BDF files


This package can, optionally, automatically annotate the following key aspects of the Mahnob HCI-Tagging dataset:

  • Average heart-rate in beats-per-minute (BPM), using the Pam-Tompkins algorithm as implemented by mne.

  • Face bounding boxes, as detected by the default detector on bob.ip.facedetect.


Note this procedure is outdated by current metadata which is already shipped with this package. Only use it in case you know what you’re doing and/or want to modify/re-evaluate this package’s metadata.

For it to work properly, you’ll need to modify the method bob.db.hci_tagging.File.load_face_detection() to take it into account. As of today, it is set to load face detections from the HDF5 files distributed with this package.

The annotation procedure can be launched with the following command:

$ hci_tagging mkmeta

Each video, which is composed of a significant number of frames (hundreds), takes about 5 minutes to get completely processed. If are at Idiap, you can launch the job on the SGE queue using the following command-line:

$ jman sub -q q1d --io-big -t 3490 `which` hci_tagging mkmeta


bob.db.hci_tagging.resource_filename(package_or_requirement, resource_name)

Return a true filesystem path for specified resource

class bob.db.hci_tagging.Database[source]

Bases: object

objects(protocol='all', subset=None)[source]

Returns a list of unique File objects for the specific query by the user.

  • protocol (str, optional) – If set, can take the value of either cvpr14 or all. cvpr14 subselects samples used by Li et al. on their CVPR``14 paper for heart-rate estimation. If all is set, the complete database is selected.

  • subset (str, optional) – If set, it could be either train, dev or test or a combination of them (i.e. a list). If not set (default), the files from all these sets are retrieved for the all protocol. Note that for the cvpr14 protocol, this has no effect, since no training, development and test set have been defined in this case.


A list of File objects.

Return type


class bob.db.hci_tagging.File(basedir, bdf, video, duration)

Bases: bob.db.base.File

Generic file container for HCI-Tagging files

  • basedir (str) – The base directory for the data

  • bdf (str) – The name of the BDF file that accompanies the video file, containing the physiological signals.

  • video (str) – The name of the video file to be used

  • duration (int) – The time in seconds that corresponds to the estimated duration of the data (video and physiological signals).


Estimates the person’s heart rate using the ECG sensor data


directory (str) – A directory name that leads to the location the database is installed on the local disk

load(directory=None, extension='.avi')[source]

Loads the video for this file entry


directory (str) – The path to the root of the database installation. This is the path leading to the directory Sessions of the database.


A 4D array of 8-bit unsigned integers corresponding to the input video for this file in (frame,channel,y,x) notation (Bob-style).

Return type



Loads the 66-keypoints coming from the Discriminative Response Map Fitting (DRMF) landmark detector. Raises if metadata file isn’t there.


The code was written for Matlab. Data for the first frame of the colour video of this object was loaded on a compatible Matlab framework and the keypoints extracted taking as basis the currently available face bounding box, enlarged by 7% (so the key-point detector performs reasonably well). The extracted keypoints were then merged into this database access package so they are easy to load from python.

The points are in the form (y, x), as it is standard on Bob-based packages.


Load bounding boxes for this file

This function loads bounding boxes for each frame of a video sequence. Bounding boxes are loaded from the package directory and are the ones provided with it. Bounding boxes generated from run_face_detector() (which should be exactly the same) are not used by this method.


A dictionary where the key is the frame number and the values are instances of bob.ip.facedetect.BoundingBox.

Return type



Loads heart-rate from locally stored files, raises if it isn’t there


Loads the colored video file associated to this object


directory (str) – A directory name that will be prefixed to the returned result.

Returns Preloaded and ready to be iterated by your code.

make_path(directory=None, extension=None)[source]

Wraps this files’ filename so that a complete path is formed

Keyword parameters:


An optional directory name that will be prefixed to the returned result.


An optional extension that will be suffixed to the returned filename. The extension normally includes the leading . character as in .png or .bmp. If not specified the default extension for the original file in the database will be used

Returns a string containing the newly generated file path.

run_face_detector(directory, max_frames=0)[source]

Runs bob.ip.facedetect stock detector on the selected frames.


This method is deprecated and serves only as a development basis to clean-up the load_face_detection(), which for now relies on .face files shipped with this package. Technically, the output of this method and the detected faces shipped should be the same as of today, 13 december 2016.

  • directory (str) – A directory name that leads to the location the database is installed on the local disk

  • max_frames (int) – If set, delimits the maximum number of frames to treat from the associated video file. A value of zero (default), makes the detector run for all frames.


A dictionary where the key is the frame number and the values are instances of bob.ip.facedetect.BoundingBox.

Return type


save(data, directory=None, extension='.hdf5')[source]

Saves the input data at the specified location and using the given extension.

Keyword parameters:


The data blob to be saved (normally a numpy.ndarray).


If not empty or None, this directory is prefixed to the final file destination


The extension of the filename - this will control the type of output and the codec for saving the input blob.