Li’s CVPR14 Python API

Signals extraction

bob.rppg.cvpr14.extract_utils.kp66_to_mask(image, keypoints, indent=10, plot=False)[source]

builds a mask on the lower part of the face

The mask is built using selected keypoints retrieved by a Discriminative Response Map Fitting (DRMF) algorithm. Note that the DRMF is not implemented here, and that the keypoints are loaded from file (and are not provided in the package).

Note also that this function is explicitly made for the keypoints set generated by the Matlab software downloaded from http://ibug.doc.ic.ac.uk/resources/drmf-matlab-code-cvpr-2013/

Update: this function also works when using bob.ip.dlib.DlibLandmarkExtraction

Parameters

  • image (numpy.ndarray) – The current frame.

  • keypoints (numpy.ndarray) – the set of 66 keypoints retrieved by DRMF.

  • indent (int) – The percentage of the facewidth [in pixels] by which selected keypoints are shifted inside the face to build the mask. THe facewidth is defined by the distance between the two keypoints located on the right and left edge of the face, at the eyes’ height.

  • plot (bool) – If set to True, plots the current face with the selected keypoints and the built mask.

Returns

  • mask (numpy.ndarray) – A boolean array of the size of the original image, where the region corresponding to the mask is True.

  • mask_points (list of tuple) – The points corresponding to vertices of the mask.

bob.rppg.cvpr14.extract_utils.get_mask(image, mask_points)[source]

returns a boolean array where the mask is True.

It turns mask points into a region of interest and returns the corresponding boolean array, of the same size as the image.

Taken from https://github.com/jdoepfert/roipoly.py/blob/master/roipoly.py

Parameters

  • image (numpy.ndarray) – The current frame.

  • mask_points (list of tuple) – The points corresponding to vertices of the mask.

Returns

mask – A boolean array of the size of the original image, where the region corresponding to the mask is True.

Return type

numpy.ndarray

bob.rppg.cvpr14.extract_utils.get_good_features_to_track(face, npoints, quality=0.01, min_distance=10, plot=False)[source]

applies the openCV function “good features to track”

Parameters

  • face (numpy.ndarray) – The cropped face image

  • npoints (int) – The maximum number of strong corners you want to detect

  • quality (float) – The minimum relative quality of the detected corners. Note that increasing this value decreases the number of detected corners. Defaluts to 0.01.

  • min_distance (int) – minimum euclidean distance between detected corners.

  • plot (bool) – if we should plot the currently selected features to track.

Returns

corners – the detected strong corners.

Return type

numpy.ndarray

bob.rppg.cvpr14.extract_utils.track_features(previous, current, previous_points, plot=False)[source]

projects the features from the previous frame in the current frame.

Parameters

  • previous (numpy.ndarray) – the previous frame.

  • current (numpy.ndarray) – the current frame.

  • previous_points (numpy.ndarray) – the set of keypoints to track (in the previous frame).

  • plot (bool) – Plots the keypoints projected on the current frame.

Returns

current_points – the set of keypoints in the current frame.

Return type

numpy.ndarray

bob.rppg.cvpr14.extract_utils.find_transformation(previous_points, current_points)[source]

finds the transformation matrix from previous points to current points.

The transformation matrix is found using estimateRigidTransform (fancier alternatives have been tried, but are not that stable).

Parameters

  • previous_points (numpy.ndarray) – Set of ‘starting’ 2d points

  • current_points (numpy.ndarray) – Set of ‘destination’ 2d points

Returns

transformation_matrix – the affine transformation matrix between the two sets of points.

Return type

numpy.ndarray

bob.rppg.cvpr14.extract_utils.get_current_mask_points(previous_mask_points, transfo_matrix)[source]

projects the previous mask points to get the current mask.

Parameters

  • previous_mask_points (numpy.ndarray) – The points forming the mask in the previous frame

  • transformation_matrix (numpy.ndarray) – the affine transformation matrix between the two sets of points.

Returns

current_mask_points – The points forming the mask in the current frame

Return type

numpy.ndarray

bob.rppg.cvpr14.extract_utils.compute_average_colors_mask(image, mask, plot=False)[source]

computes the average green color within a given mask.

Parameters

  • image (numpy.ndarray) – The image containing the face.

  • mask (numpy.ndarray) – A boolean array of the size of the original image, where the region corresponding to the mask is True.

  • plot (bool) – Plot the mask as an overlay on the original image.

Returns

color – The average RGB colors inside the mask ROI.

Return type

numpy.ndarray

bob.rppg.cvpr14.extract_utils.compute_average_colors_wholeface(image, plot=False)[source]

computes the average green color within the provided face image

Parameters

  • image (numpy.ndarray) – The cropped face image

  • plot (bool) – Plot the mask as an overlay on the original image.

Returns

color – The average green color inside the face

Return type

float

Illumination rectification

bob.rppg.cvpr14.illum_utils.rectify_illumination(face_color, bg_color, step, length)[source]

performs illumination rectification.

The correction is made on the face green values using the background green values, so as to remove global illumination variations in the face green color signal.

Parameters

  • face_color (numpy.ndarray) – The mean green value of the face across the video sequence.

  • bg_color (numpy.ndarray) – The mean green value of the background across the video sequence.

  • step (float) – Step size in the filter’s weight adaptation.

  • length (int) – Length of the filter.

Returns

rectified color – The mean green values of the face, corrected for illumination variations.

Return type

numpy.ndarray

bob.rppg.cvpr14.illum_utils.nlms(signal, desired_signal, n_filter_taps, step, initCoeffs=None, adapt=True)[source]

Normalized least mean square filter.

Based on adaptfilt 0.2: https://pypi.python.org/pypi/adaptfilt/0.2

Parameters

  • signal (numpy.ndarray) – The signal to be filtered.

  • desired_signal (numpy.ndarray) – The target signal.

  • n_filter_taps (int) – The number of filter taps (related to the filter order).

  • step (float) – Adaptation step for the filter weights.

  • initCoeffs (numpy.ndarray) – Initial values for the weights. Defaults to zero.

  • adapt (bool) – If True, adapt the filter weights. If False, only filters.

Returns

  • y (numpy.ndarray) – The filtered signal.

  • e (numpy.ndarray) – The error signal (difference between filtered and desired)

  • w (numpy.ndarray) – The found weights of the filter.

Motion correction

bob.rppg.cvpr14.motion_utils.build_segments(signal, length)[source]

builds an array containing segments of the signal.

The signal is divided into segments of provided length (no overlap) and the different segments are stacked.

Parameters

  • signal (numpy.ndarray) – The signal to be processed.

  • length (int) – The length of the segments.

Returns

  • segments (numpy.ndarray) – the segments composing the signal.

  • end_index (int) – The length of the signal (there may be a trail smaller than a segment at the end of the signal, that will be discarded).

bob.rppg.cvpr14.motion_utils.prune_segments(segments, threshold)[source]

remove segments.

Segments are removed if their standard deviation is higher than the provided threshold.

Parameters

  • segments (numpy.ndarray) – The set of segments.

  • threshold (float) – Threshold on the standard deviation.

Returns

  • pruned_segments (numpy.ndarray) – The set of “stable” segments.

  • gaps (list of length (# of retained segments)) – Boolean list that tells if a gap should be accounted for when building the final signal.

  • cut_index (list of tuples) – Contains the start and end index of each removed segment. Used for plotting purposes.

bob.rppg.cvpr14.motion_utils.build_final_signal(segments, gaps)[source]

builds the final signal with remaining segments.

Parameters

  • segments (numpy.ndarray) – The set of remaining segments.

  • gaps (list) – Boolean list that tells if a gap should be accounted for when building the final signal.

Returns

final_signal – The final signal.

Return type

numpy.ndarray

bob.rppg.cvpr14.motion_utils.build_final_signal_cvpr14(segments, gaps)[source]

builds the final signal with remaining segments.

Warning

This contains a bug !

Builds the final signal, but reproducing the bug found in the code provided by the authors of [li-cvpr-2014]. The bug is in the ‘collage’ of remaining segments. The gap is not always properly accounted for…

Parameters

  • segments (numpy.ndarray) – The set of remaining segments.

  • gaps (list) – Boolean list that tells if a gap should be accounted for when building the final signal.

Returns

final_signal – The final signal.

Return type

numpy.ndarray

Filtering

bob.rppg.cvpr14.filter_utils.detrend(signal, Lambda)[source]

applies a detrending filter.

This code is based on the following article “An advanced detrending method with application to HRV analysis”. Tarvainen et al., IEEE Trans on Biomedical Engineering, 2002.

Parameters

  • signal (numpy.ndarray) – The signal where you want to remove the trend.

  • Lambda (int) – The smoothing parameter.

Returns

filtered_signal – The detrended signal.

Return type

numpy.ndarray

bob.rppg.cvpr14.filter_utils.average(signal, window_size)[source]

Moving average filter.

Parameters

  • signal (numpy.ndarray) – The signal to filter.

  • window_size (int) – The size of the window to compute the average.

Returns

filtered_signal – The averaged signal.

Return type

numpy.ndarray