Chair of
Multimedia Communications and Signal Processing
Prof. Dr.-Ing. André Kaup
The content of the English page is outdated, please use the updated German version of our page at the moment: lms.tf.fau.de

Video processing for multi camera systems

Field of activity: Video Signal Processing and Transmission
Research topic: Video Analysis and Video Processing
Staff: M.Sc. Johannes Bauer

Full-Resolution View Synthesis

In modern TV domains, like 3D-TV or Free-Viewpoint-TV, a scene is usually taken by more than one camera. Thereby, all cameras have slightly different views of the scene. In many applications it is not possible to transmit all existing views to the receiver, due to the huge amount of data. Thus it is a key issue in the 3D video domain to reconstruct or synthesize an intermediate view from original available camera perspectives at the receiver side. One possibility to reconstruct missing views is the so called depth-image based rendering (DIBR).
In DIBR, two neighboring views and their corresponding depth information are used to synthesize a virtual camera perspective. The two neighboring views can thereby be seen as 'left reference view' and 'right reference view'. The pixels from both reference views are warped into the desired intermediate camera perspective using the corresponding depth information and known extrinsic and intrinsic camera parameters. This results in two synthesized versions of the desired view. One is obtained from the left reference view and the other one is obtained from the right reference view. The two warped images are merged afterwards. After merging, there might still exist some areas, for which no information could be found in both reference views. In order to fill those areas, a hole-filling algorithm is applied.
The quality of the resulting synthesized view can be measured both, subjectively and objectively. For an objective measurement well-known metrics like PSNR or SSIM can be used. In order to judge the subjective quality, a multiview auto-stereoscopic 3D display can be utilized at our chair.

Typical scenario: A desired intermediate camera perspective has to be synthesized by using two adjacent reference views:

(click to resize)

Mixed-Resolution View Synthesis

One special focus in this research area is the mixed-resolution view synthesis. Thereby a high- and a low-resolution reference view are used to synthesize the desired intermediate camera perspective. Due to disocclusion for instance, there might exist some areas in the synthesized view, for which no information can be found in the high-resolution reference image. Thus, this information has to be taken from the low-resolution reference. The insertion of upsampled and hence blurred regions from the low-resolution reference leads to annoying visible artifacts. The focus here is on different post-processing techniques in order to refine those areas and hence reduce the visible artifacts convincingly.

Overview of the mixed-resolution view synthesis algorithm:

(click to resize)

Publications

2017-2
CRIS
T. Richter, A. Kaup
   [bib]

Super-resolution for differently exposed mixed-resolution multi-view images adapted by a histogram matching method
IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, Mar. 2017
2015-40
CRIS
T. Richter, Habermann, Annelie, A. Kaup
   [doi]   [bib]

Super-Resolution for Mixed-Resolution Multiview Images Using a Relative Frequency Response Estimation Method
IEEE International Conference on Visual Communications and Image Processing (VCIP), Pages: 1-4, Singapore, Dec. 2015
2015-25
CRIS
T. Richter, J. Seiler, W. Schnurrer, M. Bätz, A. Kaup
   [doi]   [bib]

Combining Single-Image and Multiview Super-Resolution for Mixed-Resolution Image plus Depth Data
European Signal Processing Conf. (EUSIPCO), Pages: 1840-1844, Nice, France, Aug. 2015
2015-20
CRIS
T. Richter, J. Seiler, W. Schnurrer, A. Kaup
   [doi]   [bib]

Super-Resolution for Mixed-Resolution Multiview Image plus Depth Data using a Novel Two-Stage High-Frequency Extrapolation Method for Occluded Areas
IEEE Int. Conf. on Image Processing (ICIP), Pages: 1880-1884, Québec City, Canada, Sep. 2015
2015-14
CRIS
T. Richter, J. Seiler, W. Schnurrer, A. Kaup
   [doi]   [bib]

Robust Super-Resolution for Mixed-Resolution Multiview Image plus Depth Data
IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT) Vol. 26, Num. 5, Pages: 814-828, May 2015
2014-9
CRIS
T. Richter, M. Jonscher, W. Schnurrer, J. Seiler, A. Kaup
   [doi]   [bib]

Reconstruction of Multiview Images Taken with Non-Regular Sampling Sensors
IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Pages: 5789 - 5793, Florence, Italy, May 2014
2013-5
CRIS
T. Richter, E. Wige, A. Kaup
   [bib]

BM3D-verfeinerte Zwischenbildsynthese für unterschiedlich aufgelöste Referenzansichten
15. ITG-Fachtagung für Elektronische Medien, Dortmund, Deutschland, Feb. 2013
2012-38
CRIS
T. Richter, A. Kaup
   [doi]   [bib]

Multiview Super-Resolution Using High-Frequency Synthesis in Case of Temporally Subsampled Depth Information
IEEE Int. Conf. on Visual Communication and Image Processing (VCIP), Pages: 1-6, San Diego, USA, Nov. 2012
2012-28
CRIS
T. Richter, J. Seiler, W. Schnurrer, A. Kaup
   [doi]   [bib]

Robust Super-Resolution in a Multiview Setup Based on Refined High-Frequency Synthesis
IEEE International Workshop on Multimedia Signal Processing, Pages: 7-12, Banff, Canada, Sep. 2012
2012-2
CRIS
T. Richter, M. Schöberl, J. Seiler, T. Tröger, A. Kaup
   [doi]   [bib]

Mixed-Resolution View Synthesis Using Non-Local Means Refined Image Merging
SPIE Electronic Imaging - Stereoscopic Displays and Applications XXIII, Vol. 8288, San Francisco, CA, USA, Jan. 2012