Welcome to The Neuromorphic Engineer
Applications » Imaging

Capture and display in wide-dynamic-range imaging

PDF version | Permalink

Moti Shefer

1 December 2005

By mimicking compression performed by the retina, the effective visual acuity of systems can be enhanced.

It is commonly accepted1 that the dynamic range (DR) of an image, i.e. the image color definition, is equal to the image maximum signal-to-noise ratio (max SNR). We also know that the maximum amount of image information is given by the product of resolution (cross-image definition) and color definition. Therefore, increasing the DR of images is by no mean less important than increasing their resolution. It is worth remembering however that DR and resolution cannot possibly be traded-off against each other, since they live in orthogonal sub-spaces of the image. Below, we analyze the two most commonly-used methods to increase dynamic range.


Purely logarithmic photo-response

Working with a purely log-response practically eliminates saturation altogether, thereby making the maximum signal unlimited. It can be shown however2 that to first order, the DR at the output of any 1-1 analytic mapping equals the DR at its input.

To realize this in the log case we recall that the small-signal gain of the log function is inversely proportional to the average input signal. This implies that, at this sensor's output, a local contrast whose magnitude is usually small relative to the average luminance, maintains a pretty-much constant SNR for all levels of average luminance. At the same time, however, such a local contrast becomes progressively ‘washed-out’ as the average luminance increases: a familiar and most undesirable property of the log transmission.


The multiple-exposure method

Multi-exposure (ME) means to apply shorter exposure times to originally saturated pixels, such that saturation is altogether avoided. This is generically illustrated in Figure 1, where the exposure periods are determined according to the simplest normalized series of

where k is a natural number. Note that the overall sensor's gain to light is proportional to the exposure period. Therefore, to avoid contrast reversals, the gain of segment 2 must be elevated by a factor of 2, and so on.

The RMS of the camera shot noise, however, is proportional to the square root of the exposure time. Therefore the RMS of the camera output noise in the continuous transmission arrangement above is elevated by a factor of

in segment 2, a factor
in segment 3, and so on. The DR of this arrangement is therefore given by
and the added DR in bits is

One can see that there is not much point in having the shortest exposure period less than a 1/3 of the default, since e.g., in 1/4-default the output noise is twice that of the default, thereby amplifying the minimum detectable local contrast by a factor of 2 and causing an already-significant 'noise-washout’ effect. This limits the added DR of the ME arrangement, in practice, to just a single bit.


The role of display DR

As it turns out,3 in normal indoor lighting we cannot see more than 5 to 6 bits per color (bpc) with most cathode-ray-tube, liquid-crystal, and plasma displays. The effect of this bottleneck is demonstrated by the upper half of Figure 2, an 8bpc RGB picture Room (courtesy of Vincent Laforet, Photographer in Residence at the New-York Times) where the maximum amount of lost visual content due to the display is about nine bits per pixel (24 minus 15).


Using multiple exposures to increase dynamic range.


The 8bpc version of Room before (upper half) and after (lower half) retinal dynamic-range compression to 5bpc.

To get around the display bottleneck, we consider a similar situation that takes place inside our retina.4 The neural channels that carry all sensory information to the cortex cannot possibly support more than 7 bits of DR. Nevertheless, the DR we actually perceive can easily exceed 30bpc. The mechanism that makes this possible has been analytically modeled5 and is called retinal dynamic-range compression (DRC).


Reconstruction of retinal DRC

The basis of retinal DRC is the spatial feedback automatic-gain-control (fb-AGC), described in Figure 3. Here, the acquired image Ei is multiplied by the scalar forward gain K and fed into the first input of an image pixel-wise multiplier. The output is then fed back into the multiplier's second input after having passed through a linear spatial low pass filter (LPF, to average) and being subtracted from unity.

The average transmission (DC) of this model5 is

which is known as Michaelis’ equation. Its graph (see Figure 4) is known as Weber's law.


The spatial feedback automatic-gain-control (fb-AGC) model.

The DR compression ratio (CR) is defined as the output/input ratio for Ei = 1/K (the knee-point). We then have: CR = K/2. The compression ratio is thus readily controlled through the parameter K. The fb-AGC gain for variations of low spatial frequencies (the ‘DC-gain’) is given by


The Weber's law curve.

The fb-AGC gain for local contrast variations (the ‘AC-gain’) is obtained by assuming that, for such variations, the LPF output output remains constant. Denoting this constant by Ē Eo we have

from which we get

Taking the quotient (GAC /GDC) as a measure of the detail enhancement of the retinal DRC, we see that the amount of this detail enhancement, or the ‘effective visual acuity’ increases linearly with the average luminance of the viewed scene: a well known property of human and animal vision. The lower half of Figure 2 is Room after retinal DRC to 5bpc.


Conclusions

On the capture side of wide-dynamic-range imaging, we have demonstrated that dealing with saturation alone cannot significantly increase the DR of image sensors. We therefore conclude that this can only be achieved via significant noise reduction. On the display side of the problem, Figure 2 above demonstrates how retinal DRC can increase the potential number of perceived distinguished colors—by a factor of 512 in the current example—and shows how this affects our watching experience.




Author

Moti Shefer
Trusight Ltd.


References
  1. A. Belenky, E. Artyomov, A. Fish and O. Yadid-Pecht, Wide dynamic range imaging, The Neuromorphic Engineer, 2004.

  2. M. Shefer, Increasing the dynamic range of image sensors: Critiques of three commonly practiced methods Sept., 2005. http://www.trusight.com/motis_place.htm

  3. M. Shefer, What's DR? Sept., 2005. http://www.trusight.com/motis_place.htm

  4. M. Shefer, Neuromorphic-retinal dynamic range compression as applied to image display Sept., 2005. http://www.trusight.com/motis_place.htm

  5. M. Shefer, AGC models for Retinal signal processing November, 1979. M.Sc. Thesis, EE Dept., Technion, I.I.T.

  6. S. Polyak, The Retina, University of Chicago Press, 1941.

  7. F. Ratliff, Mach Bands: Quantitative studies on neural networks in the Retina, Holden-Day Inc., 1965.

  8. G. Westheimer, Visual acuity and spatial modulation thresholds, Handbook of sensory physiology 7 (4), pp. 170-187, Springer Verlag, 1972.

  9. J. E. Dowling, B. Ahinger and W. L. Hadden, The interplexiform cell: A new type of Retinal neuron, Investigations in Ophthalmology 15, pp. 916-926, 1976.

  10. S. Chenand and R. Ginosar, Adaptive sensitivity CCD image sensor, Proc. SPIE 2415, pp. 303-309, 1995.


 
DOI:  10.2417/1200512.0032

@NeuroEng



Tell us what to cover!

If you'd like to write an article or know of someone else who is doing relevant and interesting stuff, let us know. E-mail the editor and suggest the subject for the article and, if you're suggesting someone else's work, tell us their name, affiliation, and e-mail.