Category description and use cases
To allow collection managers to locate known persons in collections materials. If, for example, a collection has many images of someone important to their institution and suspects they appear in video footage but would like to confirm, or would like to know where in a video the person appears, a face recognition tool would provide that information.
Workflow example:
Output standard
Summary:
Element | Datatype | Obligation | Definition |
media | object | required | Wrapper for metadata about the source media file. |
media.filename | string | required | Filename of the source file. |
media.duration | string | required | The duration of the source file. |
media.frameRate | number | required | The frame rate of the video, in FPS. |
media.numFrames | number | required | The number of frames in the video. |
media.resolution | object | required | Resolution of the video. |
media.resolution.width | number | required | Width of the frame, in pixels. |
media.resolution.height | number | required | Height of the frame, in pixels. |
frames | array | required | List of frames containing identified faces. |
frames[*] | object | optional | A frame containing an identified face. |
frames[*].start | string (s.fff) | required | Time of the frame, in seconds. |
frames[*].objects | list | required | List of bounding boxes in the frame containing identified faces. |
frames[*].objects[*] | object | required | A bounding box in the frame containing an identified face. |
frames[*].objects[*].name | string | required | The name of the face within the bounding box. |
frames[*].objects[*].score | object | optional | A confidence or relevance score for the face. |
frames[*].objects[*].score.type | string (confidence | relevance) | required | The type of score, confidence or relevance. |
frames[*].objects[*].score.value | number | required | The score value, typically a number in the range of 0-1. |
frames[*].objects[*].vertices | object | optional | The top left (xmin, ymin) and bottom right (xmax, ymax) relative bounding coordinates. |
frames[*]objects[*].vertices.xmin | number | required | The top left x coordinate. |
frames[*]objects[*].vertices.ymin | number | required | The top left y coordinate. |
frames[*]objects[*].vertices.xmax | number | required | The bottom right x coordinate. |
frames[*]objects[*].vertices.ymax | number | required | The bottom right y coordinate. |
JSON Schema
Sample output
Recommended tool(s)
Python face_recognition
Official documentation: Library documentation | Custom code
Language: Python
Description: OpenCV-based face recognition library.
Cost: Free (open source)
Social impact: We retain full control of use of the images/face data.
Notes: Tests run on Charlie Nelms and Herman B Wells images/videos.
Installation & requirements
Install via pip (face_recognition).
Requires opencv-python
Parameters
Input formats
For training: Images labelled with person's name (currently via file path, but this should perhaps change-- discussion to have with dev)
For identifying: A model trained on the relevant people
Example Usage
See Colab notebook.
Example Output
List of timestamps where face was found
00:02:28 00:02:30 00:02:39 00:03:15 00:03:18 00:03:26 00:03:27 00:03:28 00:03:31 00:03:42
Evaluation summary
Precision, recall, and F1 scores for ground truth testing of five videos are in the project Google Drive.