Chair of
Multimedia Communications and Signal Processing
Prof. Dr.-Ing. André Kaup

Signal processing for digital camera systems

Field of activity: Video Signal Processing and Transmission
Research topic: Video Analysis and Video Processing
Staff:

Many algorithms in image processing start with ready-made images. Image generation and acquisition hat not been given mucht attention. The acquisition however is an important step and significant reduction in image quality can be observed. During acquisition, many decisions are made and baked into the image. In most cases a compensation or correction after the acquisition is hard or even impossible. The goal is to take these distortions into account and perform a reduction of distortions during acquisition.

In this project we will analyze image acquisition in digital camera systems. A special focus will be put on high quality moving picture images for media production. Distortions will be analyzed with methods of signal processing and quality improvements shall be developed.

Results - digital ND filter

A limitation in traditional cameras is the limited dynamic range. There is a lower threshold caused by image noise - objects from the scene below a certain intensity will not be visible among the noise. The upper limit is given by the full well capacity of a sensor - above the maximum number of electrons we can not distinguish object features. The following example shows the recording of an overexposed scene. A reconstruction of the image content is not possible.

Overexposed Recording
Overexposed Recording


In photography different methods are used to adjust the exposure and achieve a properly exposed image. The following examples show that these methods will alter the image and create often undesired results.

As a first example, we can reduce the exposure time. Unfortunatley, this will also modify the amount of motion blur. For the recoding of moving pictures motion blur is an essential component for achieving a smooth sequence. A sequence of sharp still images will be perceived as stuttering and rough. Modification of exposure time is not an option.

Reduced exposure time
Reduced exposure time

 

A further option is the adjustment of the lens aperture. This will both change the amount of light and the image reproduction. The lens aperture will adjust the depth-of-field, which is an important artistic element for direction the attention of the observer. The adjustment of the aperture is not desired in high quality image acquisition.

Reduced lens aperture diameter
Reduced lens aperture diameter


In professional imaging we can often find the use of optical neutral density filters. These filter discs can reduce the amount of light and enable the preservation of aperture and exposure time. Unfortunately, these filters are expensive, heavy and difficult to handle. A digital Option for reduction of sensitivity was previously not available.

The proposed method called digital ND filter offers the digital adjustment of the exposure. It allows the reduction of the sensor sensitivity while preserving apterture and exposure time. By recording multiple images during the original exposure time and the computational combination we can shift the effective dynamic range of the camera. The following example show the recording with reduced sensitivity without introducing distortions.

Adjustment with digital ND filter
Adjustment with digital ND filter

Publications

2014-62 J. Keinert, M. Schöberl, M. Wetzel, S. Foessel
   [link]   [bib]

Compressed data stream transmission using rate control
WO 2014/082680 A1, Jun. 2014
2014-58
CRIS
J. Keinert, M. Schöberl, M. Ziegler, F. Zilly, S. Foessel
   [link]   [bib]

High-Dynamic Range Video Cameras Based on Single Shot Non-Regular Sampling
SMPTE Motion Imaging Journal Vol. 8, Num. 123, Pages: 49-54, Nov. 2014
2014-52
CRIS
F. Zilly, M. Ziegler, J. Keinert, M. Schöberl, S. Foessel
   [link]   [bib]

Computational Imaging for Stop-Motion Animated Video Productions
The Best of IET and IBC Vol. 6, Pages: 14-19, Sep. 2014
2014-37 H. Sparenberg, S. Foessel, M. Martin, M. Schöberl
   [link]   [bib]

File System, Computer and Method
WO 2014 056 546 A1, Apr. 2014
2014-36
CRIS
F. Zilly, M. Ziegler, J. Keinert, M. Schöberl, S. Foessel
   [link]   [bib]

Computational Imaging for Stop-Motion Animated Video Productions
International Broadcast Convention (IBC) Conference, Amsterdam, Netherlands, Sep. 2014
2014-35
CRIS
M. Ziegler, F. Zilly, P. Schäfer, J. Keinert, M. Schöberl, S. Foessel
   [link]   [bib]

Dense lightfield reconstruction from multi aperture cameras
IEEE International Conference on Image Processing (ICIP), Oct. 2014
2014-1
CRIS
J. Keinert, M. Wetzel, M. Schöberl, P. Schäfer, F. Zilly, M. Bätz, S. Foessel, A. Kaup
   [doi]   [bib]

Cost-effective multi-camera array for high quality video with very high dynamic range
SPIE Electronic Imaging - Digital Photography X, Vol. SPIE 9023, San Francisco, CA, USA, Feb. 2014
2013-86 J. Ernst, M. Schöberl, A. Oberdörster
   [link]   [bib]

Optical Aliasing Filter, Pixel Sensor Arrangement and Digital Recording Device
US 8 593 562 B2, Nov. 2013
2013-77 M. Schöberl, J. Keinert, A. Kaup, S. Foessel
   [link]   [bib]

Camera Apparatus and Camera
PCT/EP2011/061756, Jan. 2013
2013-69
CRIS
S. Foessel, F. Zilly, M. Schöberl, P. Schäfer, M. Ziegler, J. Keinert
   [link]   [bib]

Light-field Acquisition and Processing System for Film Productions
SMPTE Annual Technical Conference, Hollywood, CA, USA, Oct. 2013
2013-68 M. Schöberl, A. Kaup
   [link]   [bib]

Bildsensor und Verfahren zum Aufnehmen eines Bildes
EP000002567539A1, Feb. 2013
2013-67 M. Schöberl, A. Kaup
   [link]   [bib]

Erkennung einer Änderung zwischen Bildern oder in einer Sequenz von Bildern
EP 2 420 062 B1, Jul. 2013
2013-66 A. Nowak, M. Schöberl
   [link]   [bib]

Creating a Digital Version of a Video From an Analog Magnetic Tape Recording
WO 2013/026908 A2, Feb. 2013
2013-65 M. Schöberl, A. Kaup
   [link]   [bib]

Detecting a Change Between Images or in a Sequence of Images
US8509549 B2, Aug. 2013
2013-63
CRIS
M. Schöberl
   [link]   [bib]

Modeling of Image Acquisition for Improving Digital Camera Systems
Dr. Hut, München, 2013
2013-62
CRIS
J. Keinert, M. Schöberl, M. Ziegler, F. Zilly, S. Foessel
   [link]   [bib]

High Dynamic Range Video Cameras Based on Single Shot Non-Regular Sampling
International Broadcast Convention Conference (IBC), Amsterdam, NL, Sep. 2013
2013-61
CRIS
J. Keinert, M. Schöberl, M. Ziegler, F. Zilly, S. Foessel
   [link]   [bib]

High dynamic range video cameras based on single shot non-regular sampling
The Best of IET and IBC Vol. 5, Pages: 31-37, Sep. 2013
2013-53
CRIS
F. Zilly, M. Schöberl, P. Schäfer, M. Ziegler, J. Keinert, S. Foessel
   [link]   [bib]

Light-field acquisition system allowing camera viewpoint and depth of field compositing in post-production
International Broadcast Convention Conference (IBC), Amsterdam, NL, Sep. 2013
2013-52
CRIS
F. Zilly, M. Schöberl, P. Schäfer, M. Ziegler, J. Keinert, S. Foessel
   [link]   [doi]   [bib]

Lightfield Media Production System using Sparse Angular Sampling
ACM SIGGRAPH, Annaheim, Jul. 2013
2013-26 M. Schöberl, J. Keinert
   [link]   [bib]

Vorrichtung zum Encodieren einer Mehrzahl von Symbolen in einen codieren Bitstrom, Vorrichtung zum Decodieren eines codierten Bitstroms und Verfahren
DE 10 2011 085 451 B3, Apr. 2013
2013-25 A. Nowak, M. Schöberl
   [link]   [bib]

Erzeugung einer digitalen Version eines Videos aus einer analogen Magnetbandaufzeichnung
DE 10 2011 081 409 B4, Feb. 2013
2013-24 M. Schöberl, A. Kaup
   [link]   [bib]

Image Sensor and Method of Capturing an Image
US 20130063622 A1, Mar. 2013
2013-3
CRIS
A. Brückner, M. Schöberl
   [link]   [doi]   [bib]

Diffraction and photometric limits in today's miniature digital camera systems
SPIE Photonics West - MOEMS and Miniaturized Systems XII, Num. 8616, Feb. 2013
2013-1
CRIS
M. Schöberl, J. Keinert, M. Ziegler, J. Seiler, M. Niehaus, G. Schuller, A. Kaup
   [link]   [doi]   [bib]

Evaluation of a high dynamic range video camera with non-regular sensor
SPIE Electronic Imaging - Digital Photography IX, Feb. 2013
2012-68 M. Schöberl, A. Kaup
   [link]   [bib]

Detecting a Change Between Images or in a Sequence of Images
US20120082343 A1, Apr. 2012
2012-67
CRIS
S. Foessel, M. Schöberl
   [bib]

Frame rate conversion from higher to lower frame rates in motion picture
SMPTE Annual Technical Conference - Symposium on High Frame Rates for Digital Cinema, Hollywood, CA, USA, Oct. 2012
2012-54 M. Schöberl, J. Seiler, A. Kaup
   [link]   [bib]

Bildsensor, Bildaufnahmesystem und Verfahren zum Aufnehmen eines Bildes
WO 2012/098117 A2, Jul. 2012
2012-52 H. Sparenberg, M. Schöberl, A. Nowak, S. Foessel
   [link]   [bib]

Video Recording System and Method for Video Recording
WO 2012/104284 A1, Aug. 2012
2012-47 H. Sparenberg, M. Schöberl, A. Nowak, S. Fössel
   [link]   [bib]

Videoaufnahmesystem und Verfahren zur Videoaufnahme
DE 10 2011 003 392 A1, Aug. 2012
2012-41 M. Schöberl, J. Seiler, A. Kaup
   [link]   [bib]

Bildsensor, Bildaufnahmesystem und Verfahren zum Aufnehmen eines Bildes
DE 10 2011 002 824 A1, Jul. 2012
2012-36
CRIS
M. Schöberl, A. Brückner, S. Foessel, A. Kaup
   [link]   [doi]   [bib]

Photometric Limits for Digital Camera Systems
SPIE Journal of Electronic Imaging Vol. 21, Num. 2, Jun. 2012
2012-22
CRIS
M. Schöberl, A. Belz, J. Seiler, S. Foessel, A. Kaup
   [link]   [doi]   [bib]

High Dynamic Range Video by Spatially Non-Regular Optical Filtering
IEEE International Conference on Image Processing (ICIP), Pages: 2757-2760, Sep. 2012
2012-21
CRIS
M. Schöberl, A. Belz, A. Nowak, J. Seiler, A. Kaup, S. Foessel
   [link]   [doi]   [bib]

Building a High Dynamic Range Video Sensor with Spatially Non-Regular Optical Filtering
SPIE Optics + Photonics, Applications of Digital Image Processing XXXV, Aug. 2012
2012-19
CRIS
M. Schöberl, J. Seiler, A. Kaup, A. Nowak
   [link]   [bib]

Auflösungserhöhung durch nichtregelmäßige Abtastung
Fernseh- und Kinotechnik Vol. 4, Apr. 2012
2012-18 M. Schöberl, A. Kaup
   [link]   [bib]

Bildsensor und Verfahren zum Aufnehmen eines Bildes
DE 10 2010 028 746 B4, Jun. 2012
2012-1
CRIS
M. Schöberl, J. Keinert, J. Seiler, S. Foessel, A. Kaup
   [link]   [doi]   [bib]

Defect Pixel Interpolation for Lossy Compression of Camera Raw Data
SPIE Electronic Imaging - Visual Information Processing and Communication II, Vol. 8305, San Francisco, CA, USA, Jan. 2012
2011-67 M. Schöberl, A. Kaup
   [link]   [bib]

Image Sensor and Method for Recording an Image
WO 2011/138 369 A1, Nov. 2011
2011-62
CRIS
M. Schöberl, J. Keinert, A. Kaup, S. Foessel
   [link]   [bib]

Analysis of Processing Pipelines in Digital RAW Cameras
Microelectronic Systems: Circuits, Systems and Applications, Editor(s): A. Heuberger, G. Elst, R. Hanke, Pages: 281-293, Springer, Heidelberg, Nov. 2011
2011-61 M. Schöberl, A. Kaup
   [link]   [bib]

Erkennung einer Änderung zwischen Bildern oder in einer Sequenz von Bildern
DE 10 2009 017 436 B4, Sep. 2011
2011-60 A. Oberdörster, M. Schöberl, J. Ernst
   [link]   [bib]

Sensorvorrichtung und Verfahren zum Erfassen von lokalen Intensitätsgradienten
DE 10 2010 002 599 B4, Dec. 2011
2011-59 M. Schöberl, A. Kaup
   [link]   [bib]

Bildsensor und Verfahren zum Aufnehmen eines Bildes
DE 10 2010 028 746 A1, Nov. 2011
2011-56 A. Oberdörster, M. Schöberl, J. Ernst
   [link]   [bib]

Sensorvorrichtung und Verfahren zum Erfassen von lokalen Intensitätsgradienten
DE 10 2010 002 599 A1, Sep. 2011
2011-49 J. Ernst, M. Schöberl, A. Oberdörster
   [link]   [bib]

Optical Aliasing Filter, Pixel Sensor Arrangement and Digital Recording Device
US 2011/0134294 A1, Jun. 2011
2011-45
CRIS
J. Ernst, M. Schöberl
   [link]   [bib]

Oberflächenstrukturierter Aliasfilter für digitale Kameras
Fernseh- und Kinotechnik Vol. 6, Pages: 317-322, Jun. 2011
2011-35 M. Schöberl, A. Kaup
   [link]   [bib]

Bildaufnahmevorrichtung und Verfahren zum Erzeugen einer Aufnahme
DE 10 2009 029 868 B4, Mar. 2011
2011-34 J. Ernst, M. Schöberl, A. Oberdörster
   [link]   [bib]

Optischer Aliasfilter, Pixelensoranordnung und digitale Aufnahmevorrichtung
EP 2 306 234 A2, Apr. 2011
2011-32
CRIS
M. Schöberl, J. Seiler, S. Foessel, A. Kaup
   [link]   [doi]   [bib]

Increasing Imaging Resolution by Covering Your Sensor
IEEE International Conference on Image Processing (ICIP), Pages: 1937-1940, Brussels, Belgium, Sep. 2011
2011-31
CRIS
M. Schöberl, W. Schnurrer, S. Foessel, A. Kaup
   [link]   [doi]   [bib]

Increasing Camera Dynamic Range Through In-Sensor Multi-Exposure White Balancing
IEEE International Conference on Image Processing (ICIP), Pages: 3226-3229, Brussels, Belgium, Sep. 2011
2011-9
CRIS
M. Schöberl, J. Seiler, B. Kasper, S. Foessel, A. Kaup
   [link]   [doi]   [bib]

Sparsity-based Defect Pixel Compensation for Arbitrary Camera Raw Images
IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Pages: 1257-1260, Prague, Czech Republic, May 2011
2011-1
CRIS
M. Schöberl, J. Ernst, W. Schnurrer, S. Foessel, A. Kaup
   [link]   [doi]   [bib]

Moving Refractive Optical Low Pass Filter for Digital Cameras
SPIE Electronic Imaging - Digital Photography VII, Vol. 7876, Pages: 78760Z, San Francisco, CA, USA, Jan. 2011
2010-78 M. Schöberl, A. Kaup
   [link]   [bib]

Bildaufnahmevorrichtung und Verfahren zum Erzeugen einer Aufnahme
WO 2010/149 591 A1, Dec. 2010
2010-77 M. Schöberl, A. Kaup
   [link]   [bib]

Bildaufnahmevorrichtung und Verfahren zum Erzeugen einer Aufnahme
DE 10 2009 029 868 A1, Dec. 2010
2010-73 M. Schöberl, A. Kaup
   [link]   [bib]

Erkennung einer Änderung zwischen Bildern oder in einer Sequenz von Bildern
DE 10 2009 017 436 A1, Nov. 2010
2010-65 M. Schöberl, A. Kaup
   [link]   [bib]

Detection of a Change Between Images or in a Sequence of Images
WO 2010/119 060 A1, Oct. 2010
2010-45
CRIS
M. Schöberl, W. Schnurrer, S. Fößel, A. Kaup
   [link]   [bib]

Dynamikerweiterung durch Weißabgleich im Bildsensor
16. Farbworkshop der German Color Group, Ilmenau, Pages: 108-121, Ilmenau, Oct. 2010
2010-22
CRIS
M. Schöberl, S. Fößel, A. Kaup
   [link]   [doi]   [bib]

Fixed Pattern Noise Column Drift Compensation (CDC) for Digital Moving Picture Cameras
IEEE International Conference on Image Processing (ICIP), Pages: 573-576, Hong Kong, Sep. 2010
2010-21
CRIS
M. Schöberl, W. Schnurrer, A. Oberdörster, S. Fößel, A. Kaup
   [link]   [doi]   [bib]

Dimensioning of Optical Birefringent Anti-Alias Filters for Digital Cameras
IEEE International Conference on Image Processing (ICIP), Pages: 4305-4308, Hong Kong, Sep. 2010
2010-15
CRIS
M. Schöberl, V. Bruns, S. Fößel, H. Bloss, A. Kaup
   [link]   [doi]   [bib]

Change Detection in JPEG 2000 Compressed Video
Electronics Letters Vol. 46, Num. 6, Pages: 409-410, Mar. 2010
2010-1
CRIS
M. Schöberl, A. Oberdörster, S. Fößel, H. Bloss, A. Kaup
   [link]   [doi]   [bib]

Digital Neutral Density Filter for Moving Picture Cameras
SPIE Electronic Imaging - Computational Imaging VIII, Vol. 7533, Pages: 75330L, San Jose, CA, USA, Jan. 2010
2009-30
CRIS
M. Schöberl, S. Fößel, H. Bloss, A. Kaup
   [link]   [doi]   [bib]

Modeling of Image Shutters and Motion Blur in Analog and Digital Camera Systems
IEEE International Conference on Image Processing (ICIP), Pages: 3457-3460, Cairo, Egypt, Nov. 2009
2009-20
CRIS
M. Schöberl, C. Senel, S. Fößel, H. Bloss, A. Kaup
   [bib]

Non-linear Dark Current Fixed Pattern Noise Compensation for Variable Frame Rate Moving Picture Cameras
17th European Signal Processing Conference (EUSIPCO), Pages: 268-272, Glasgow, Scotland, Aug. 2009