Compressed Domain Video Analysis

Field of activity:	Video Signal Processing and Transmission
Research topic:	Video Analysis and Video Processing
Staff:

Description

In literature, there are several techniques for different post-processing steps for videos. Most of them operate in the so-called pixel domain. Pixel domain means that any processing is directly performed on the actual pixel values of a video image. Thereto all compressed video data has to be decoded before analysis algorithms can be performed. An example of a simplified processing chain is illustrated in Figure 1.

Figure 1: Pixel domain processing chain

The simplest way of analyzing video content is to watch it on an appropriate display. For example, a surveillance camera could transmit images of an area that is relevant for security to be evaluated by a watchman. Although this mode obviously finds its application in practice, it is not applicable for all systems, because of two major problems. The first problem is that at any time someone needs to keep track of the monitors. As a result this mode is indeed on the one hand real-time capable, but on the other hand quite expensive. A second major problem is that it is not scalable. If a surveillance system has a huge amount of cameras installed, it is nearly impossible to keep track of all of the monitors at the same time. So the efficieny of this mode will decrease with an increasing number of sources.

Beside a manual analysis of video content, an automated analysis became more and more important in the last years. The processing chain in Figure 1 shows a simplified procedure of an automated analysis in the pixel domain. At first, the received video content from the network has to be decoded. Thereby the decoded video frames are stored in a frame buffer to have access to them during the analysis procedure. Based on these video frames an analysis algorithm, e.g., object detection and tracking can be performed. A main advantage over a manual analysis is that this mode is usually highly scalable and less expensive. But due to the decoding process, the frame buffer operations, and the usually high computing time of pixel domain detection algorithms, this mode is not always real-time capable and has furthermore a high complexity.

Due to the limitations of pixel domain approaches, more and more attempts were made to transfer the video analysis procedures from pixel domain to compressed domain. Working within compressed domain means to work directly on compressed data. Figure 2 illustrates a processing chain within the compressed domain.

Figure 2: Compressed domain processing chain

Due to the omission of the preceding decoder it is now possible to work directly with the received data. At the same time, the now integrated syntax parser permits to extract single required elements from the data stream and to use them for analyzing. As a result, the analysis becomes less computationally intensive due to the reason that the costly decoding process must not be passed through completely at any time. Furthermore, this solution consumes less resources since it is not required anymore to store the video frames in a buffer. This leads to a technique that is compared to pixel domain techniques usually more efficient and appears more scalable.

The major task of this research project is to develop new methods and algorithms for a general, multi-layer system for analyzing compressed video data. Thereby a layer is defined by a single decoding step of the decoding process. Different analysis algorithms can be performed on different layers. The task is to give a general description of analysis classes and to determine the minimum decoding effort for each class. As a general rule, the complexity of an analysis algorithm decreases if it can be performed on a lower layer. Hence, the analysis will be faster and it is possible to process more video streams simultaneously, which leads to a higher scalability.

Cooperations

This research project is supported by Siemens Corporate Technology in Munich. Thereby, parts of the research work have also been published within the scope of the EU project FIWARE. Further details and related videos can be found on the website Compressed Domain Video Analysis in FIWARE.

Publications

2016-50 CRIS

M. Laumer, P. Amon, A. Hutter, A. Kaup    [link]   [doi]   [bib] Moving Object Detection in the H.264/AVC Compressed Domain APSIPA Transactions on Signal and Information Processing (ATSIP) Vol. 5, Online Publication, Num. e18, Pages: 1-20, Nov. 2016

2016-27

M. Laumer, P. Amon, A. Hutter    [bib] Apparatus and Method for Detecting a Moving Object CN 105516650 A, Apr. 2016

2016-26

M. Laumer, P. Amon, A. Hutter    [bib] Vorrichtung und Verfahren zum Detektieren eines sich bewegenden Objekts DE 10 2014 220 809 A1, Apr. 2016

2016-19

P. Wojaczek, M. Laumer, P. Amon, A. Hutter    [bib] Object Detection Device and Method for Detecting an Object Within a Video Sequence EP 2 988 273 A1, Feb. 2016

2015-12 CRIS

M. Laumer, P. Amon, A. Hutter, A. Kaup    [doi]   [bib] Compressed Domain Moving Object Detection by Spatio-Temporal Analysis of H.264/AVC Syntax Elements Picture Coding Symposium (PCS), Pages: 282-286, Cairns, Australia, May 2015

2015-2 CRIS

P. Wojaczek, M. Laumer, P. Amon, A. Hutter, A. Kaup    [doi]   [bib] Hybrid Person Detection and Tracking in H.264/AVC Video Streams Int. Conf. on Computer Vision Theory and Applications (VISAPP), Vol. 1, Pages: 478-485, Berlin, Germany, Mar. 2015

2013-78

M. Laumer, P. Amon, A. Hutter, A. Kaup    [bib] Method for Processing a Compressed Video Stream US 2013/0300940 A1, Nov. 2013

2013-73

M. Laumer, P. Amon    [bib] Methods and Devices for Object Detection in Coded Video Data WO 2013/160040 A1, Oct. 2013

2013-72

M. Laumer, P. Amon, A. Hutter, A. Kaup    [bib] Method for Processing a Compressed Video Stream CN 103299618 A, Sep. 2013

2013-71

M. Laumer, P. Amon    [bib] Methods and Devices for Object Detection in Coded Video Data EP 2 658 255 A1, Oct. 2013

2013-54

M. Laumer, P. Amon, A. Hutter, A. Kaup    [bib] Method for Processing a Compressed Video Stream EP 2 619 982 A2, Jul. 2013

2013-2 CRIS

M. Laumer, P. Amon, A. Hutter, A. Kaup    [doi]   [bib] Compressed Domain Moving Object Detection Based on H.264/AVC Macroblock Types Int. Conf. on Computer Vision Theory and Applications (VISAPP), Pages: 219-228, Barcelona, Spain, Feb. 2013

2012-64

M. Laumer, P. Amon, A. Hutter, A. Kaup    [bib] Method for Processing a Compressed Video Stream WO 2012/098078 A3, Oct. 2012

2012-63

M. Laumer, P. Amon, A. Hutter, A. Kaup    [bib] Method for Processing a Compressed Video Stream WO 2012/098078 A2, Jul. 2012

2011-48 CRIS

M. Laumer, P. Amon, A. Hutter, A. Kaup    [doi]   [bib] A Compressed Domain Change Detection Algorithm for RTP Streams in Video Surveillance Applications IEEE Int. Workshop on Multimedia Signal Processing (MMSP), Pages: 1-6, Hangzhou, China, Oct. 2011