Introduction
The goal of this application is to display a visual representation of the events happening in the Idiap Showroom in real-time. The room is equipped with a microphone array (8 microphones, located in the center of the room) and 4 cameras (one in each corner). A 3D representation of the room is projected on a screen.
The video/audio processing part of the application is made of separate processing blocks chained together. Most of these blocks run in their own thread. Some block examples: Video capture, distortion correction, background substraction, people detection, people tracking, audio capture, speakers localization, rendering, streams synchronization, ...
Screenshots
(Click for full size)
 |
The current 3D representation of the room. The dark yellow area on the floor indicates that a speaker was detected there. The more confident we are, the more bright yellow it is. The grid on the floor is the actual one used for the detection. In that sequence, each cell has 25x25cm. |
 |
The 4 video streams captured, after distortion correction (hence the strange curves at the borders). The red rectangles are the result of the people detection algorithm. |
 |
Composite view of one video stream and the corresponding virtual camera. My avatar (I'm the guy with the white shirt) is speaking, thanks to the speakers localization process. |
 |
Another perspective. Each change of view triggers an animation, either an animation of the camera of the 3D view, or an animation of the views themselves, or even a combination of the two. |
Details
- 4 video streams at 7.5fps
- 8 audio streams
- 38 processing blocks
- Libraries used: Ogre3D, OpenCV, pom (The Probabilistic Occupancy Map)
- Running on a MacPro 2.8GHz Quad-core
- 4 developers
Current features
- People tracking
- Speakers localization
Planned features/future work
- Improve some of the algorithms (background substraction, people tracking)
- Improve the 3D view (replace the avatar by a better one, add the missing furnitures, add some special effects, collision detection)
- Customization of the avatars appearence
- Speech recognition (to pilot the application: the toolbar at the top will the be removed, and the application will go fullscreen at that point)