Extend “facedetect” module in gstreamer into full speaker tracking system.

The aim of this project is to extend the OpenCV "facedetect" module in gstreamer (<[login to view URL]>) into a robust speaker tracking module. The facedetect module currently detects faces and emits the position of each face on the gstreamer bus. The project will production of the following; * Forking and renaming the "facedetect" module into a "speakertrack" module. * Adding "detection" and "locked" modes to the module. More details about these modes and how the transitions work can be found in the details section. * Creation of a small gstreamer based program which utilities the module and shows the current mode and position of the "active person" overlay on the input. * A number of test videos which can be fed into the above program to demonstrate it working. **When putting in an bid please include links to any previous C or Python code you have produced. Experience with gstreamer or OpenCV is not required but highly desirable. **Please include a set of milestones and timeline with your bid. ## Deliverables The aim of this project is to extend the OpenCV "facedetect" module in gstreamer (<[login to view URL]>) into a robust speaker tracking module. The facedetect module currently detects faces and emits the position of each face on the gstreamer bus. All code must be delivered as commits to your forked version of gst-plugins-bad git repository. The base git repository can be found at <[login to view URL]> This documentation shows how to build gstreamer; <[login to view URL]> The project will production of the following; * Forking and renaming the "facedetect" module into a "speakertrack" module. * Adding "detection" and "locked" modes to the module. More details about these modes and how the transitions work can be found in the details section. * Creation of a small gstreamer based program which utilities the module and shows the current mode and position of the "active person" overlay on the input. * A number of test videos which can be fed into the above program to demonstrate it working. This project is part of a larger project to create an automated video recording system, with the eventual goal of getting every talk at a User Group (such as Linux, Python, etc) recorded and live streamed without needing people with the knowledge how. Further work may be offered after completion of this project. **When putting in an bid please include links to any previous C or Python code you have produced. Experience with gstreamer or OpenCV is not required but highly desirable.** ### Adding "Detection" Mode Detection mode is where the module looks for a "gesture" in a camera's field of view. Once the gesture is detected, the module should transition into "locked mode" with the person performing the "gesture" becoming the active person. If there are multiple people in the camera's view the system should not transition. Possible gestures include a "high five" or "waving" at the camera in an aggressive manner. Other suggestions for the gesture are welcome. This process can take 15-20 seconds if needed. It will be done once at the beginning of the talk which would run for at least 15 minutes. You do not need to support "instant speaker changes". **Required output:** During this mode, no position information should be emitted gstreamer bus. The current mode of the module should be emitted onto the gstreamer bus. ### Adding "Locked" Mode Locked mode where the module emits the position of the face for the "active person". If the camera looses the "active person" for over 1 minute, it should transition back to "detection mode". The system should be resistant to the following problems; * Speaker turning away from the camera (while not drastically moving) so that their face is no longer seen. * The background is unevenly illuminated. * Very bright objects are in the background (such as a screen for example). * Someone walking in front of the camera causing a temporary loss of sight of the speaker for less then 15 seconds. * The back of people's heads appear at the bottom of the camera view. * The camera moves in a **controlled fashion**, such as panning in either X or Y direction and zoom. It is okay for the system to fail in the following cases; * Background has multiple moving people in it. * A different person walks between the speaker and the camera. * The speaker gives turns away from the camera and moves around. * The camera is dramatically bumped or moves in an uncontrolled fashion. **Required output:** The current mode of the module should be emitted onto the gstreamer bus. The position of the "active person"'s face should be emitted onto the gstreamer bus. The "size" of the "active person"'s face should be emitted onto the gstreamer bus. ## Further Requirements * Acceptable languages for development are Python and/or C++. * The module must successfully run on a dual Core i7 running at 2.4Ghz when feed 720p video input. * Run on Ubuntu Precise. * Should have reasonable level of test cases and the test cases should have more then 80% code coverage. # Legal All code must be delivered as commits to your forked version of gst-plugins-bad git repository. The base git repository can be found at <[login to view URL]> You will retain copyright of the code developed but a non-exclude license of the complete code for the system must be delivered under one of the following Open Source license; * GPL 3.0 - Full text at <[login to view URL]> * LGPL 3.0 - Full text at <[login to view URL]> * Apache 2 - Full text at <[login to view URL]> **If you are unable to release all the code to the system you must not bid.** You can use any existing code which is released under a compatible license including; * Gstreamer - <[login to view URL]> * OpenCV - [[login to view URL]][1]