Chair of
Multimedia Communications and Signal Processing
Prof. Dr.-Ing. André Kaup

Scalable Multi-View Video Coding

Field of activity: Video Signal Processing and Transmission
Research topic: Video Coding and Transmission
Staff: Dr.-Ing. Jens-Uwe Garbas

Background

Multiview video is the synchronous recording of a moving scene with a setup of several cameras. The utilization of video based rendering techniques allows, within certain limits, the generation of arbitrary photorealistic viewpoints of the recorded scene. This can be used for immersive video communication applications, such as free viewpoint television or three-dimensional television. Since the amount of data that has to be stored or transmitted increases proportional with the number of cameras, efficient compression of the multiview video data is crucial. Due to the heterogeneity of transmission scenarios and displaying devices, scalability of the coded bistream is also very desireable.

Project Description

The aim of this project is the finding of solutions for a highly efficient scalable compression of multiview video data. The problem is tackled by making use of the inherent scalability of the discrete wavelet transform. Since multiview video data has four dimensions, namely the two-dimensional video frame data as well as time and view dimensions, a four-dimensional wavelet transform is applied. In order to account for the perspective offset across the different views as well as the motion of the scene, disparity and motion compensation are integrated into the wavelet transform. This is realized by a lifting implementation that guarantees the invertibility of the transform even with the compensation steps. The wavelet coefficients are then embedded quantized and entropy coded by a similar entropy coding approach as employed in JPEG2000. Finally, a scalable bistream is generated, that allows scaling of the multiview video data in the view, time, spatial, and quality dimensions without any need for re-encoding or transcoding the data. It is shown, that the approach performs comparable to state-of-the-art non-scalable multiview video coding frameworks.

Further aspects the project include various enhancements of the proposed codec, such as nonlinear brightness and color correction, optimal adaptive wavelet packet transforms, enhanced spatial scalability by layered encoding, and the introduction of novel prediction modes for improving the motion and disparity compensated prediction steps.

Publications

2011-23
CRIS
J. Garbas, B. Pesquet-Popescu, A. Kaup
   [doi]   [bib]

Methods and Tools for Wavelet-Based Scalable Multiview Video Coding
IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT) Vol. 21, Num. 2, Pages: 113-126, Feb. 2011
2009-56
CRIS
J. Garbas, B. Pesquet-Popescu, A. Kaup
   [bib]

Optimized Anisotropic Spatial Transforms for Wavelet-Based Scalable Multi-View Video Coding
SPIE Visual Communications and Image Processing (VCIP), Vol. 7257, Jan. 2009
2009-33
CRIS
J. Garbas, A. Kaup
   [bib]

Analysis on Spatial Scalable Multiview Video Coding with Wavelets
IEEE International Workshop on Multimedia Signal Processing (MMSP), Rio de Janeiro, Brazil, Oct. 2009
2008-38
CRIS
J. Garbas, M. Trocan, B. Pesquet-Popescu, A. Kaup
   [bib]

Wavelet-Based Multi-View Video Coding with Joint Best Basis Wavelet Packets
IEEE International Conference on Image Processing (ICIP), Pages: 1232-1235, San Diego, USA, Oct. 2008
2007-31
CRIS
J. Garbas, A. Kaup
   [bib]

Inter-Scale Prediction of Motion Information for a Wavelet-Based Scalable Video Coder
Picture Coding Symposium, Lisbon, Portugal, Nov. 2007
2007-19
CRIS
J. Garbas, U. Fecker, A. Kaup
   [bib]

Wavelet-Based Multi-View Video Coding with Full Scalability and Illumination Compensation
15th Annual ACM International Conference on Multimedia 2007, Pages: 751-754, Augsburg, Germany, Sep. 2007
2007-14
CRIS
J. Garbas, A. Kaup
   [bib]

Wavelet-Based Multi-View Video Coding with Spatial Scalability
IEEE International Workshop on Multimedia Signal Processing, Pages: 422-425, Chania, Crete, Greece, Oct. 2007
2006-33
CRIS
J. Garbas, U. Fecker, T. Tröger, A. Kaup
   [doi]   [bib]

4D Scalable Multi-View Video Coding Using Disparity Compensated View Filtering and Motion Compensated Temporal Filtering
IEEE 8th. Intern. Workshop on Multimedia Signal Processing (MMSP), Pages: 54-58, Victoria, Canada, Oct. 2006