Facemap API Guide

Pose estimation

class facemap.pose.pose.Pose(filenames=None, bbox=[], bbox_set=False, resize=False, add_padding=False, gui=None, GUIobject=None, net=None, model_name=None)[source]

Pose estimation for single video processing

Parameters

filenames (2D-list) – List of filenames to be processed.
bbox (list) – Bounding box for cropping the video [x1, x2, y1, y2]. If not set, the entire frame is used.
bbox_set (bool) – Flag to indicate whether the bounding box has been set. Default is False.
resize (bool) – Flag to indicate whether the video needs to be resized
add_padding (bool) – Flag to indicate whether the video needs to be padded. Default is False.
gui (object) – GUI object.
GUIobject (object) – GUI mainwindow object.
net (object) – PyTorch model object.
model_name (str) – Name of the model to be used for pose estimation. Default is None which uses the pre-trained model.

load_model()[source]: Load model for keypoints prediction. Uses default model unless set_model is used to update the model name and load the model state.

predict_landmarks(video_id, frame_ind=None)[source]: Predict keypoints/landmarks for all frames in video and save output as .h5 file. If frame_ind is specified, only predict keypoints/landmarks for those frames. :param video_id: Index of video in filenames list to be used for prediction. :type video_id: int :param frame_ind: List of frame indices for keypoints/landmarks prediction. :type frame_ind: list

save_data_to_hdf5(data, video_id, selected_frame_ind=None)[source]

save_data_to_hdf5: Save data to an HDF5 file

Parameters

data (2D-array) – Data to save (nframes x nbodyparts x 3)
selected_frame_ind (list) – Indices of selected frames

save_dict_to_hdf5(h5file, path, data_dict)[source]

Saves dictionary to an HDF5 file. Adapted from https://github.com/talmolab/sleap/blob/391bc0421fe3820ddd6b5d07e31311d60b129fe3/sleap/util.py#L116 Calls itself recursively if items in dictionary are not np.ndarray, np.int64, np.float64, str, or bytes. Objects must be iterable. :param h5file: The HDF5 filename object to save the data to.

Assume it is open.

Parameters

path – The path to group save the dict under.
data_dict – The dict containing data to save.

Raises

ValueError – If type for item in dict cannot be saved.

Returns

None

save_model(model_filepath)[source]: Save model to file :param model_filepath: Path to save model weights :type model_filepath: str

set_model(model_selected=None)[source]

Set model to use for pose estimation

Parameters: model_selected (str, optional) – Path of trained model weights to use. Default value of None sets the model_name to Base model/facemap_model_state which uses pre-trained weights.

train(image_data, keypoints_data, num_epochs, batch_size, learning_rate, weight_decay, bbox)[source]

Train the model :param image_data: Array of images of shape (nframes, Ly, Lx) :type image_data: ND-array :param keypoints_data: Array of keypoints of shape (nframes, nkeypoints, 2) :type keypoints_data: ND-array :param num_epochs: Number of epochs for training :type num_epochs: int :param batch_size: Batch size for training :type batch_size: int :param learning_rate: Learning rate for training :type learning_rate: float :param weight_decay: Weight decay for training :type weight_decay: float

Returns: model – Trained/finetuned model
Return type: torch.nn.Module

SVD processing

facemap.process.process_blink_ROIs(t, nt0, img, ivid, rois, blind, blinks)[source]: docstring

facemap.process.process_pupil_ROIs(t, nt1, img, ivid, rois, pupind, pups, pupreflector)[source]: docstring

facemap.process.process_running(t, n, nt1, img, ivid, rois, runind, runs, rend)[source]: docstring

facemap.process.run(filenames, sbin=1, motSVD=True, movSVD=False, GUIobject=None, parent=None, proc=None, savepath=None)[source]

Process video files using SVD computation of motion and/or raw movie data. :param filenames: List of video files to process. Each element of the list is a list of

filenames for video(s) recorded simultaneously. For example, if two videos were recorded simultaneously, the list would be: [[‘video1.avi’, ‘video2.avi’]], and if the videos were recorded sequentially, the list would be: [[‘video1.avi’], [‘video2.avi’]].

Parameters

sbin (int) – Spatial binning factor. If sbin > 1, the movie will be spatially binned by a factor of sbin.
motSVD (bool) – If True, compute SVD of motion in the video i.e. the difference between consecutive frames.
movSVD (bool) – If True, compute SVD of raw movie data.
GUIobject (GUI object) – GUI object to update progress bar. If None, no progress bar will be updated.
parent (GUI object) – Parent GUI object to update progress bar. If None, no progress bar will be updated.
proc (dict) – Dictionary containing previously processed data. If provided, parameters from the saved data, such as sbin, rois, sy, sx, etc. will be used.
savepath (str) – Path to save processed data. If None, the processed data will be saved in the same directory as the first video file.

Returns

savename – Path to saved processed data.

Return type

str

Transforms

facemap.pose.transforms.adjust_bbox(prev_bbox, img_yx, div=16, extra=1)[source]

Takes a bounding box as an input and the original image size. Adjusts bounding box to be square instead of a rectangle. Uses longest dimension of prev_bbox for final image size that cannot exceed img_yx :param prev_bbox: bounding box positions in order x1, x2, y1, y2 :type prev_bbox: tuple of size (4,) :param img_yx: image size for y and x dimensions :type img_yx: tuple of size (2,)

Returns: bbox – bounding box positions in order x1, x2, y1, y2
Return type: tuple of size (4,)

facemap.pose.transforms.adjust_keypoints(xlabels, ylabels, crop_xy, padding, current_size, desired_size)[source]

Adjust raw keypoints (x,y coordinates) obtained from model to plot on original image :param xlabels: x coordinates of keypoints :type xlabels: ND-array :param ylabels: y coordinates of keypoints :type ylabels: ND-array :param crop_xy: initial coordinates of bounding box (x1,y1) for cropping :type crop_xy: tuple of size (2,) :param padding: padding values for bounding box (x1,x2,y1,y2) :type padding: tuple of size (4,)

Returns

xlabels (ND-array) – x coordinates of keypoints
ylabels (ND-array) – y coordinates of keypoints

facemap.pose.transforms.adjust_keypoints_for_padding(xlabels, ylabels, pads)[source]

Adjust keypoints for padding. Adds padding to the top and left of the image only. :param xlabels: x coordinates of keypoints :type xlabels: ND-array :param ylabels: y coordinates of keypoints :type ylabels: ND-array :param pads: padding values for bounding box (y1,y2,x1,x2) :type pads: tuple of size (4,)

Returns

xlabels (ND-array) – x coordinates of keypoints after padding
ylabels (ND-array) – y coordinates of keypoints after padding

facemap.pose.transforms.augment_data(image, keypoints, scale=False, scale_range=0.5, rotation=False, rotation_range=10, flip=True, contrast_adjust=True)[source]

Augments data by randomly scaling, rotating, flipping, and adjusting contrast :param image: image of size nchan x Ly x Lx :type image: ND-array :param keypoints: keypoints of size nkeypoints x 2 :type keypoints: ND-array :param scale: whether to scale the image :type scale: bool :param scale_range: range of scaling factor :type scale_range: float :param rotation: whether to rotate the image :type rotation: bool :param rotation_range: range of rotation angle :type rotation_range: float :param flip: whether to flip the image horizontally :type flip: bool :param contrast_adjust: whether to adjust contrast of image :type contrast_adjust: bool

Returns

image (ND-array) – image of size nchan x Ly x Lx
keypoints (ND-array) – keypoints of size nkeypoints x 2

facemap.pose.transforms.crop_image(im, bbox=None)[source]

Crop image to bounding box. :param im: image of size [(Lz) x Ly x Lx] :type im: ND-array :param bbox: bounding box positions in order x1, x2, y1, y2 :type bbox: tuple of size (4,)

Returns: im – cropped image of size [1 x Ly x Lx]
Return type: ND-array

facemap.pose.transforms.get_crop_resize_params(img, x_dims, y_dims, xy=(256, 256))[source]

Get cropped and resized image dimensions Input:-

img: image x_dims: min,max x pos y_dims: min,max y pos xy: final (desired) image size

Output:-: x1: (int) x dim start pos x2: (int) x dim stop pos y1: (int) y dim start pos y2: (int) y dim stop pos resize: (bool) whether to resize image

facemap.pose.transforms.get_cropped_imgs(imgs, bbox)[source]

Preproccesing of image involves: conversion to float32 in range0-1, normalize99, and padding image size to be compatible with UNet model input :param imgs: images of size [batch_size x nchan x Ly x Lx] :type imgs: ND-array :param bbox: bounding box positions in order x1, x2, y1, y2 :type bbox: tuple of size (4,)

Returns: cropped_imgs – images of size [batch_size x nchan x Ly’ x Lx’] where Ly’ = y2-y1 and Lx’=x2-x1
Return type: ND-array

facemap.pose.transforms.get_random_factor(factor_range)[source]

Get a random factor within the range provided. :param factor_range: factor range in order min, max :type factor_range: tuple of size (2,)

Returns: factor – random factor
Return type: float

facemap.pose.transforms.pad_img_to_square(img, bbox=None)[source]

Pad image to square. :param im: image of size [c x h x w] :type im: ND-array :param bbox: bounding box positions in order x1, x2, y1, y2 used for cropping image :type bbox: tuple of size (4,)

Returns

im (ND-array) – padded image of size [c x h x w]
(pad_w, pad_h) (tuple of int) – padding values for width and height

facemap.pose.transforms.pad_keypoints(keypoints, pad_h, pad_w)[source]

Pad keypoints using padding values for width and height. :param keypoints: keypoints of size [N x 2] :type keypoints: ND-array :param pad_h: height padding :type pad_h: int :param pad_w: width padding :type pad_w: int

Returns: keypoints – padded keypoints of size [N x 2]
Return type: ND-array

facemap.pose.transforms.preprocess_img(im, bbox, add_padding, resize, device=None)[source]

Preproccesing of image involves:

Conversion to float32 and normalize99

Cropping image to select bounding box (bbox) region

padding image size to be square

Resize image to 256x256 for model input

Parameters

bbox: tuple of size (4,): bounding box positions in order x1, x2, y1, y2
add_padding: bool: whether to add padding to image
resize: bool: whether to resize image

Returns

postpad_shape: tuple of size (2,): shape of padded image
pads: tuple of size (4,): padding values for (pad_y_top, pad_y_bottom, pad_x_left, pad_x_right)

facemap.pose.transforms.randomize_bbox_coordinates(bbox, im_shape, random_factor_range=(0.1, 0.3))[source]

Randomize bounding box by a random amount to increase/expand the bounding box region while staying within the image region. :param bbox: bounding box positions in order x1, x2, y1, y2 :type bbox: tuple of size (4,) :param im_shape: image shape in order Ly, Lx :type im_shape: tuple of size (2,) :param random_factor_range: range of random factor to use for expaning bounding box :type random_factor_range: tuple of size (2,)

Returns: bbox – randomized bounding box positions in order x1, x2, y1, y2
Return type: tuple of size (4,)

facemap.pose.transforms.rescale_keypoints(xlabels, ylabels, current_size, desired_size)[source]

Rescale keypoints to original image size :param xlabels: x coordinates of keypoints :type xlabels: ND-array :param ylabels: y coordinates of keypoints :type ylabels: ND-array :param current_size: current size of image (h,w) :type current_size: tuple of size (2,) :param desired_size: desired size of image (h,w) :type desired_size: tuple of size (2,)

Returns

xlabels (ND-array) – x coordinates of keypoints
ylabels (ND-array) – y coordinates of keypoints

facemap.pose.transforms.resize_image(im, resize_shape)[source]

Resize image to given height and width. :param im: image of size [Ly x Lx] :type im: ND-array :param resize_shape: desired shape of image :type resize_shape: tuple of size (2,)

Returns: im – resized image of size [h x w]
Return type: ND-array

facemap.pose.transforms.resize_keypoints(keypoints, desired_shape, original_shape)[source]

Resize keypoints to desired shape. :param keypoints: keypoints of size [batch x N x 2] :type keypoints: ND-array :param desired_shape: desired shape of image :type desired_shape: tuple of size (2,) :param original_shape: original shape of image :type original_shape: tuple of size (2,)

Returns: keypoints – keypoints of size [batch x N x 2]
Return type: ND-array