Chair of
Multimedia Communications and Signal Processing
Prof. Dr.-Ing. André Kaup

Resolution Enhancement Techniques for Compressed Video Sequences

Field of activity: Video Signal Processing and Transmission
Research topic: Video Analysis and Video Processing
Staff:

A high spatial and temporal resolution as well as a high bit depth is desirable in many fields of application such as entertainment, automotive systems, or surveillance so as to be capable of preserving fine details, displaying fast motion in a smooth manner, and making under- or oversaturated regions distinguishable. Depending on the requirements, however, hardware performance, data rate, or storage space can be very limited. To meet the requirements, video sequences must be compressed or additionally downsampled. For alleviating object detection and recognition and to improve legibility of text, a higher spatial resolution is required. For facilitating object tracking, a higher temporal resolution is needed. Finally, in case of under- or oversaturated areas, a higher bit depth is essential. For each of these three cases, a higher resolution has to be reconstructed from a low resolution input. Each resolution enhancement can be realized by particular methods. A higher spatial resolution can be achieved by super-resolution methods that can be divided into single-image and multi-image approaches. As part of the research conducted in spatial resolution enhancement, a hybrid super-resolution method has been developed which combines both a single-image approach especially for static areas and regions reconstructed with erroneous motion information and a multi-image technique when valid motion vectors are available. Furthermore, for faster processing of color images, it has been shown that it is sufficient to apply an effective reconstruction approach only on the luminance values. The corresponding chrominance values can be reconstructed by a fast linear interpolation without perceivable quality loss. A higher temporal resolution can be achieved by employing framerate up-conversion techniques. These can be grouped into non-motion compensated approaches and motion compensated ones. In the context of this field of research, the performance of framerate up-conversion methods is evaluated on both regular and non-regular temporal sampling patterns. In particular, 3-D reconstruction approaches such as 3-D frequency selective extrapolation are of interest. Finally, a higher bit depth can be realized by merging images with different exposure times. The acquisition of these images is typically done in a temporal bracketing manner which makes the creation of high dynamic range video sequences difficult. However, there is also a method to generate high dynamic range video data from a stereo or multi-view setup by using a spatial bracketing approach instead. As part of this field of research, the performance of these reconstruction approaches is also evaluated for compressed input data. Moreover, modifications to these methods need to be conceived so as to improve the reconstruction quality or speed by making use of the side information extracted from the coding. As for the coding itself, the hybrid coder H.264/AVC and its successor H.265/HEVC are considered and side information such as block partitioning, quantization parameters, coding modes, or others are available for improving the above-mentioned methods.

 

Publications

2017-1
CRIS
M. Bätz, F. Brand, A. Eichenseer, A. Kaup
   [bib]

Motion Compensated Frame Rate Up-Conversion Using 3D Frequency Selective Extrapolation and a Multi-Layer Consistency Check
IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, Mar. 2017
2016-42
CRIS
M. Bätz, W. Schnurrer, J. Koloda, A. Eichenseer, A. Kaup
   [bib]

Joint Shape and Centroid Adaptive Frequency Selective Extrapolation for the Reconstruction of Arbitrarily Shaped Loss Areas
Picture Coding Symposium (PCS), Nuremberg, Germany, Dec. 2016
2016-35
CRIS
M. Bätz, J. Koloda, A. Eichenseer, A. Kaup
   [bib]

Multi-Image Super-Resolution Using a Locally Adaptive Denoising-Based Refinement
IEEE Workshop on Multimedia Signal Processing (MMSP), Montreal, Canada, Sep. 2016
2016-28
CRIS
M. Bätz, A. Eichenseer, A. Kaup
   [doi]   [bib]

Multi-Image Super-Resolution for Fisheye Video Sequences Using Subpixel Motion Estimation Based on Calibrated Re-Projection
European Signal Processing Conf. (EUSIPCO), Pages: 1872 - 1876, Budapest, Hungary, Aug. 2016
2016-24
CRIS
M. Bätz, A. Eichenseer, A. Kaup
   [doi]   [bib]

Multi-Image Super-Resolution Using a Dual Weighting Scheme Based on Voronoi Tessellation
IEEE Int. Conf. on Image Processing (ICIP), Pages: 2822 - 2826, Phoenix, AZ, USA, Sep. 2016
2015-21
CRIS
M. Bätz, A. Eichenseer, J. Seiler, M. Jonscher, A. Kaup
   [bib]

Hybrid Super-Resolution Combining Example-Based Single-Image and Interpolation-Based Multi-Image Reconstruction Approaches
IEEE Int. Conf. on Image Processing (ICIP), Pages: 58-62, Quebec City, Canada, Sep. 2015
2014-47
CRIS
M. Bätz, A. Eichenseer, M. Jonscher, J. Seiler, A. Kaup
   [bib]

Accelerated Hybrid Image Reconstruction for Non-Regular Sampling Color Sensors
IEEE Int. Conf. on Visual Communications and Image Processing (VCIP), Pages: 217-220, Valletta, Malta, Dec. 2014