Chair of
Multimedia Communications and Signal Processing
Prof. Dr.-Ing. André Kaup
The content of the English page is outdated, please use the updated German version of our page at the moment: lms.tf.fau.de

# Error Concealment of Image Data

Field of activity: Video Signal Processing and Transmission Video Coding and Transmission Dr.-Ing. Jürgen SeilerM.Sc. Nils Genser

If images or video sequences are transmitted over wireless channels or the internet, the risk of transmission errors is ubiquitous. This results in the problem that individual regions cannot be decoded and displayed correctly. But it is possible to estimate these lost areas from the other correctly received regions. To achieve this, we developed the Selective Extrapolation. This algorithm is able to reconstruct arbitrary image contents and can be applied to images as well as video sequences.

For estimating the lost signal parts, Selective Extrapolation generates a model of signal, defined over the complete considere area. For this, the model results from an iterative superposition of orthogonal basis functions. Prior to the model generation, a set of basis functions has to be selected. During our research we discovered, that the functions of the discrete Fourier-transform form a good basis function set. In this case, the algorithm is called Frequency Selective Extrapolation (FSE). In general, other basis function sets can be used as well. For generating the model, the known signal is projected onto all basis functions in every iteration. Resulting from the projections, one basis function is selected to be added to the model. This is the one, that maximizes the decrement of the approximation error energy. These steps are repeated for a predefined number of times. As the basis functions cover the known as well as unknown areas, the resulting model continues the signal into the distorted areas.

For a detailed description of Selective Extrapolation, please refer to 2008-10 and 2005-20.

### Examples for concealment of distorted image data

Original image
Distorted image
Concealed image

To demonstrate the abilies of Selective Extrapolation we provide a MATLAB-Example for error concealment by 2D Selective Extrapolation. This program is free of charge for scientific and personal use. Any commercial use is prohibited.

Source code (frequency_selective_extrapolation.m)
Example image (lena_y.tif)

### Error concealment of corrupted video data

If video sequences have to be concealed instead of imaged, correctly received previous frames can be used for model generation in addition to the correctly received regions from the actual frame. With that one obtains a three-dimensional data volume. For this a model is generated using three-dimensional basis functions. As the model generation uses correctly received regions from the actual frame as well as from previous frames, a very high quality of the concealed sequence can be achieved. In the case that Fourier basis functions are used for model generation, the algorithm is called 3D Frequency Selective Extrapolation (3D-FSE). A detailed description of 3D-FSE can be found in 2007-30.

The concealment quality can be further improved if the motion of the sequence is compensated prior to the model generation. In doing so, the different layers of the volume sonsist similar image contents which leads to a more precise model generation. This extension is called Motion Compensated Frequency Seletive Extrapolation (MC-FSE). For a detailed description of Selective Extrapolation, please refer to 2008-25.

Below, some example sequenzes to prove the abilities of 3D-FSE and MC-FSE:

Sequence "Discovery City"

Sequence "Discovery Orient"

AVI
YUV

AVI
YUV
Original
Distorted
Concealed by 3D-FSE
Concealed by MC-FSE

### Image restauration by Selective Extrapolation

Besides concealment of distortions resulting from transmission errors, Selective Extrapolation can also be used for image restauration. In doing so, defects or disturbing objects can be removed from images. To achieve this, the regions to be extrapolated are marked manually in a first step. For this a binary mask with zero at the regions to replace is generated. Then, the image is divided into blocks and the 2D Selective Extrapolation is applied to all blocks that contain regions to be extrapolated.

Two examples for image restauration by Selective Extrapolation:

We provide program fse_restauration which was used for creating these images free of charge for scientific and personal use. Any commercial use is prohibited. For compiling the source code, openmp and fftw is necessary. In addition to the command line program we provide a small graphical user interface. This can be used to adjust the individual parameters more easily and the GUI further supports more file formats as the command line program only supports ppm-files. The program was developed and tested using OpenSUSE11. The usage of the program is on your own risk.

Source code (fse_restauration.c)
Binary for OpenSUSE 11 (fse_restauration)
GUI (fse_restauration_gui.py)

#### Brief manunal for FSE-Restauration

In addition to the image to be processed, the program needs a mask that indicates all samples to be extrapolated. This mask can be created with an arbitrary image processing program. The image and the mask e.g. look like this:

These two images are selected either using the command line program or the GUI. Further an output-file has to be defined. For the extrapolation, additional parameters can be adjusted. These parameters are:

Number of iterations
FFT size
Block size
Support area width
Decay factor
Orthogonality deficiency compensation
Usage of fast FSE

All parameters can also be adjusted by using fse_restauration_gui.

For a detailed discussion of the individual parameters, please refer to the corresponding publications on Selective Extrapolation. For the example above we can obtain the following image:

The parameters for this image are:
25000 iterations
FFT size 128x128
Block size 32x32 pixel
Support area 32 pixel wide
Decay factor 0.9
Orthogonality deficiency compensation 0.8
Already concealed areas weighted by 0.4

## Publications

 2016-45CRIS J. Seiler, W. Schnurrer, S. Schöll, A. Kaup Optimized Processing Order for 3D Hole Filling in Video Sequences Using Frequency Selective Extrapolation accepted for Picture Coding Symposium (PCS), Nuremberg, Germany, Dec. 2016
 2015-34CRIS J. Seiler, M. Jonscher, M. Schöberl, A. Kaup Resampling Images to a Regular Grid from a Non-Regular Subset of Pixel Positions Using Frequency Selective Reconstruction IEEE Transactions on Image Processing (IEEE TIP) Vol. 24, Num. 11, Pages: 4540-4555, Nov. 2015
 2014-25CRIS J. Koloda, J. Seiler, A. Kaup, V. Sánchez, A.M. Peinado An error-based recursive filling ordering for image error concealment accepted for IEEE Int. Conf. on Image Processing (ICIP), Pages: 2517-2521, Paris, France, Oct. 2014
 2014-10CRIS J. Koloda, J. Seiler, A. Kaup, V. Sánchez, A.M. Peinado Frequency Selective Extrapolation with Residual Filtering for Image Error Concealment IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Pages: 1976-1980, Florence, Italy, May 2014
 2013-21CRIS J. Seiler, M. Schöberl, A. Kaup Spatio-Temporal Error Concealment in Video by Denoised Temporal Extrapolation Refinement IEEE Int. Conf. on Image Processing (ICIP), Melbourne, Australia, Sep. 2013
 2013-18CRIS W. Schnurrer, J. Seiler, A. Kaup Improving Block-Based Compensated Wavelet Lifting by Reconstructing Unconnected Pixels International Symposium on Signals, Circuits and Systems (ISSCS), Pages: 1-4, Iasi, Romania, Jul. 2013
 2011-63CRIS J. Seiler Signal Extrapolation Using Sparse Representations and its Applications in Video Communication Verlag Dr. Hut, München, Dec. 2011
 2011-43CRIS J. Seiler, A. Kaup Optimized and Parallelized Processing Order for Improved Frequency Selective Signal Extrapolation European Signal Processing Conf. (EUSIPCO), Pages: 269-273, Barcelona, Spain, Aug. 2011
 2011-14CRIS J. Seiler, A. Kaup A Fast Algorithm for Selective Signal Extrapolation with Arbitrary Basis Functions EURASIP Journal on Advances in Signal Processing (JASP) Vol. 2011, 2011
 2011-4CRIS J. Seiler, A. Kaup Motion Compensated Three-Dimensional Frequency Selective Extrapolation for Improved Error Concealment in Video Communication Journal of Visual Communication and Image Representation (VCIR) Vol. 22, Pages: 213-225, Mar. 2011
 2010-56CRIS J. Seiler, A. Kaup Complex-Valued Frequency Selective Extrapolation for Fast Image and Video Signal Extrapolation IEEE Signal Processing Letters (IEEE SPL) Vol. 17, Num. 11, Pages: 949 - 952, Nov. 2010
 2010-25CRIS J. Seiler, A. Kaup Content-Adaptive Motion Compensated Frequency Selective Extrapolation for Error Concealment in Video Communication IEEE International Conference on Image Processing (ICIP 2010), Pages: 469-472, Hong Kong, China, Sep. 2010
 2009-29CRIS J. Seiler, A. Kaup Multiple Selection Extrapolation for Spatial Error Concealment IEEE International Workshop on Multimedia Signal Processing (MMSP), Rio de Janeiro, Brazil, Oct. 2009
 2008-27CRIS J. Seiler, A. Kaup Adaptive Joint Spatio-Temporal Error Concealment for Video Communication IEEE International Workshop on Multimedia Signal Processing (MMSP 2008), Pages: 229-234, Cairns, Australia, Oct. 2008
 2008-25CRIS J. Seiler, A. Kaup Motion Compensated Frequency Selective Extrapolation for Error Concealment in Video Coding 16th European Signal Processing Conference (EUSIPCO), Lausanne, Switzerland, Aug. 2008
 2008-10CRIS J. Seiler, A. Kaup Fast Orthogonality Deficiency Compensation for Improved Frequency Selective Image Extrapolation IEEE International Conference on Acoustics, Speech, and Signal Processing, Pages: 781-784, Las Vegas, Nevada, Mar. 2008
 2008-3CRIS K. Meisinger Selective Signal Extrapolation and its Application in Image and Video Communications, PhD thesis Friedrich-Alexander-Universität Erlangen-Nürnberg, University Erlangen-Nuremberg, Germany, Jan. 2008
 2007-51 A. Kaup, K. Meisinger 3D Signalextrapolation DE 10 2007 002 545 A1, Jul. 2007
 2007-35CRIS J. Seiler, K. Meisinger, A. Kaup Orthogonality Deficiency Compensation for Improved Frequency Selective Image Extrapolation Picture Coding Symposium, Lisbon, Portugal, Nov. 2007
 2007-30CRIS K. Meisinger, A. Kaup Spatiotemporal Selective Extrapolation for 3-D Signals and its Applications in Video Communications IEEE Transactions on Image Processing (IEEE TIP) Vol. 16, Num. 9, Pages: 2348-2360, Sep. 2007
 2007-29CRIS K. Meisinger, T. Aach, A. Kaup Spatio-Temporal Defect Pixel Interpolation Using 3-D Frequency Selective Extrapolation IEEE International Conference on Image Processing, Vol. IV, Pages: 149-152, San Antonio, Texas, Sep. 2007
 2006-36CRIS K. Meisinger, S. Martin, A. Kaup Spatio-Temporal Concealment in H.264/AVC Video Coding by 3-D Selective Extrapolation 8th Int. Workshop on Multimedia Signal Processing (MMSP), Pages: 291-295, Victoria, BC, Canada, Oct. 2006
 2006-25CRIS K. Meisinger, A. Kaup 2D Frequency Selective Extrapolation for Spatial Error Concealment in H.264/AVC Video Coding IEEE International Conference on Image Processing (ICIP), Vol. -, Pages: 2233-2236, Atlanta, Oct. 2006
 2006-24CRIS K. Meisinger, S. Martin, A. Kaup Spatio-Temporal Selective Extrapolation for 3-D Signals Applied to Concealment in Video Communications 14th European Signal Processing Conference (EUSIPCO), Sep. 2006
 2005-35CRIS K. Meisinger, T. Tröger, M. Zeller, A. Kaup Automatische TV-Logo-Entfernung durch statistische modellbasierte Detektion und frequenzselektive Kompensation Elektronische Medien: 11. Dortmunder Fernsehseminar, ITG-Fachbericht 188, Pages: 211-214, Dortmund, Deutschland, Sep. 2005
 2005-32CRIS K. Meisinger, T. Tröger, M. Zeller, A. Kaup Automatic TV logo removal using statistical based logo detection and frequency selective inpainting 13th European Signal Processing Conference (EUSIPCO), Antalya, Turkey, Sep. 2005
 2005-20CRIS A. Kaup, K. Meisinger, T. Aach Frequency Selective Signal Extrapolation with Applications to Error Concealment in Image Communication AEÜ - International Journal of Electronics and Communication Vol. 59, Num. 3, Pages: 147-156, 2005
 2004-26CRIS K. Meisinger, J. Garbas, A. Kaup Error Control and Concealment of JPEG2000 Coded Image Data in Error Prone Environments Picture Coding Symposium (PCS), San Francisco, Dec. 2004
 2004-13CRIS K. Meisinger, A. Kaup Minimizing a Weighted Error Criterion for Spatial Error Concealment of Missing Image Data IEEE International Conference on Image Processing (ICIP), Pages: 813-816, Singapore, Oct. 2004
 2004-8CRIS K. Meisinger, A. Kaup Spatial Error Concealment of Corrupted Image Data using Frequency Selective Extrapolation Intern. Conf. on Acoustics, Speech and Signal Processing, Vol. III, Pages: 209-212, Montreal, Canada, May 2004
 2003-22CRIS K. Meisinger, A. Kaup Örtliche Fehlerverschleierung von gestört empfangenen Bilddaten durch frequenzselektive Extrapolation Elektronische Medien: 10. Dortmunder Fernsehseminar, ITG-Fachbericht 179, Pages: 189-194, Sep. 2003