Chair of
Multimedia Communications and Signal Processing
Prof. Dr.-Ing. André Kaup

Spatio-Temporal Prediction

Field of activity: Video Signal Processing and Transmission
Research topic: Video Coding and Transmission
Staff: Dr.-Ing. Jürgen Seiler
M.Sc. Nils Genser

Most of the actually used video coding standards are based on the idea of hybrid video coding. One of the most important parts of hybrid video codecs is the prediction. For this, the region of the signal actually coded is predicted from already completely transmitted areas. These can either be already transmitted previous frames or the regions from the actual frame that have been decoded already. As for prediction only regions are used that have been transmitted and thus are available at the decoder as well, the prediction can be carried out at the decoder in the same way as at the encoder. Due to this, only the prediction residual has to be transmitted instead of the original signal. Hence, prediction directly influences the coding efficiency. The better the signal is predicted, the less data hat to be transmitted for the residual.

Current video codecs carry out the prediction either in spatial or in temporal direction. For temporal prediction, the best fitting area is located in a previous frame and is used as predictor. For spatial prediction, the already transmitted parts of the frame are skillfully continued into the area to be predicted. Although actual codedcs can switch adaptively between spatial and temporal prediction, no combined spatio-temporal prediction is performed. Thus the available information is not exploited optimally.

But we work on how to exploit spatial as well as temporal information for prediction and therewith increase the coding efficieny. To achieve this, we perform a two-staged prediction. In the first stage a pure temporal prediction is carried out by motion compensation. Subsequent to this, the motion compensated signal is spatially refined. For this, a model of the signal is generated that fits the motion compensated signal as well as the already decoded neighboring areas. Thus, the model contains temporal as well as spatial information and can serve as an improved predictor for the signal being coded.

The subsequent block diagram shows a simplified hybrid video codec enhanced by spatial refinement.

 

For generating the model, different algorithms can be used. The algorithms developed so far are based on Selective Extrapolation, or denoising and deblocking algorithms and are able to lead to a significantly increased coding efficiency. The rate-distortion curve and the table belao show the performance of these three algorithms compared to pure motion compensation (MC).


 

  Crew Discovery City Discovery Orient Foreman Vimto
Mean rate reduction over MC          
MSA 7.69% 15.84% 13.41% 2.26% 14.98%
FSA 7.32% 10.38% 12.03% 3.20% 13.42%
RBA 6.20% 9.57% 8.73% 1.42% 12.61%
Mean PSNR-gain over MC          
MSA 0.39dB 1.16dB 0.61dB 0.09dB 0.74dB
FSA 0.37dB 0.74dB 0.55dB 0.13dB 0.66dB
RBA 0.31dB 0.68dB 0.39dB 0.06dB 0.62dB

 

 

Publications

2011-63
CRIS
J. Seiler
   [link]   [bib]

Signal Extrapolation Using Sparse Representations and its Applications in Video Communication
Verlag Dr. Hut, München, Dec. 2011
2011-14
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

A Fast Algorithm for Selective Signal Extrapolation with Arbitrary Basis Functions
EURASIP Journal on Advances in Signal Processing (JASP) Vol. 2011, 2011
2011-13
CRIS
J. Seiler, A. Kaup
   [link]   [bib]

Reusing the H.264/AVC Deblocking Filter for Efficient Spatio-Temporal Prediction in Video Coding
IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Pages: 1049-1052, Prague, Czech Republic, May 2011
2010-58
CRIS
J. Seiler, T. Richter, A. Kaup
   [doi]   [bib]

Spatio-Temporal Prediction in Video Coding by Non-Local Means Refined Motion Compensation
Picture Coding Symposium, Pages: 318-321, Nagoya, Japan, Dec. 2010
2010-56
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Complex-Valued Frequency Selective Extrapolation for Fast Image and Video Signal Extrapolation
IEEE Signal Processing Letters (IEEE SPL) Vol. 17, Num. 11, Pages: 949 - 952, Nov. 2010
2010-3
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Multiple Selection Approximation for Improved Spatio-Temporal Prediction in Video Coding
IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Pages: 886-889, Dallas, Texas, Mar. 2010
2009-11
CRIS
J. Seiler, H. Lakshman, A. Kaup
   [doi]   [bib]

Spatio-Temporal Prediction in Video Coding by Best Approximation
Picture Coding Symposium, Pages: 81-84, Chicago, USA, May 2009
2008-26
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Spatio-temporal prediction in video coding by spatially refined motion compensation
IEEE International Conference on Image Processing (ICIP), Pages: 2788-2791, San Diego, USA, Oct. 2008
2008-10
CRIS
J. Seiler, A. Kaup
   [doi]   [bib]

Fast Orthogonality Deficiency Compensation for Improved Frequency Selective Image Extrapolation
IEEE International Conference on Acoustics, Speech, and Signal Processing, Pages: 781-784, Las Vegas, Nevada, Mar. 2008
2007-35
CRIS
J. Seiler, K. Meisinger, A. Kaup
   [link]   [bib]

Orthogonality Deficiency Compensation for Improved Frequency Selective Image Extrapolation
Picture Coding Symposium, Lisbon, Portugal, Nov. 2007