A joint project between my laboratory at the Hong Kong University of Science and Technology, Ning Qian at Columbia University, and Meihua Tai at the Polytechnic University (both NY) has been set up to develop an active binocular-vision system where visual control is based upon the distributed populations of cortical neurons. This system consists of a six-degree-of-freedom binocular-vision head and custom hardware for rapidly computing, communicating and combining the outputs of retinotopic maps of model cortical neurons.

The development of neuromorphic systems for cortically-inspired visual processing leads naturally to the incorporation of active gaze control. For example, the disparity-selective model neurons that are constructed using the disparity energy model are only accurate within a small spatial range due to effects such as phase wrap around. As a result, the active control of camera gaze can bring different parts of the image into the required disparity range. Gibson anticipated this when he argued that perception arises through an active process that involves adjustments of the perceptual organ.¹ Appropriately, he likens active senses to tentacles or feelers. Although the visual senses have the potential to acquire environmental information purely passively—as evidenced by our ability to engineer algorithms using stereo heads with fixed camera parameters to extract environmental depth—there are many computational advantages of incorporating active gaze control into perceptual processing.²

Our binocular active-vision head (see Figure 1) has three degrees of freedom for each eye: horizontal and vertical rotation, as well as rotation around the line of sight (torsion). Because the most rapid eye movements are associated with saccades, during the design phase we took care to ensure that saccadic eye movements performed by the head can match or exceed those observed in primates. However, since it appears that visual perception is shut down during a saccade, we were not particularly concerned with matching exact trajectories.

The addition of torsional control distinguishes this binocular vision head from most of those previously developed. In humans, the eyes cannot only rotate horizontally or vertically, but also within about 10° around the line of sight. Active neural control of this torsional component may be important in reducing the motion of epipolar lines to enable stereopsis with smaller retina-fixed disparity search zones,³ as well as in quickly stabilizing the retinal image during gaze shifts where both eyes and head move.⁴

The six-degree-of-freedom active vision head.

The system also includes custom-designed hardware for computing the outputs of retinotopic arrays of artificial neurons (maps). These model the responses of populations of neurons within the visual cortex that are tuned to respond to different combinations of spatial/temporal frequency, orientation, and binocular disparity. For maximum expandability, we adopted a modular architecture. Computation is distributed among a number of identical boards, each of which (see Figure 2) contains a high-speed fixed-point digital signal processor (DSP) chip (the TI 6414 DSP) operating at 600MHz for computing the responses of the model neurons. Intra board communication is handled by a Xilinx Spartan III field-programmable gate array (FPGA) chip connected with low-voltage differential signalling serializers/deserializers. Each board supplements the on-chip memories of the DSP and FPGA with 8MB of synchronous dynamic random access memory (SDRAM) and 4MB of static RAM.

One of the boards used to compute cortical maps.

On these boards, computation and communication is similar to that previously developed for neuromorphic models of the retinotopic arrays of neurons tuned to different orientations in the primary visual cortex. In these earlier implementations, computation was performed using custom-designed mixed-signal analog-digital chips.⁵ However, our current system uses digital processing to enable rapid reconfiguration of the processing performed by each board, sacrificing low power consumption for enhanced flexibility. This enables more rapid experimentation with different models of bio-inspired processing. However, because the structure of the overall system is similar to that used by multi-chip neuromorphic networks, we expect that the processing performed by each board will easily be mapped onto mixed-signal neuromorphic VLSI chips. Thus, we view this system as an intermediate step between software simulations on a standard personal computer and multichip networks of custom-designed chips.

2. J. J. Gibson, The Senses Considered as Perceptual Systems, Greenwood Press, Westport, CT, 1966.
   
   
3. Y. Aloimonos, I. Weiss and A. Bandyopadhyay, Active vision, Int'l J. Comput. Vision 1, pp. 333-356, 1988.
   
   
4. K. Schreiber, J. D. Crawford, M. Fetter and D. Tweed, The motor side of depth vision, Nature 410, pp. 819-22, 2001.
   
   
5. D. Tweed, T. Haslwanter and M. Fetter, Optimizing gaze control in three dimensions, Science 281, pp. 1, 1363.
   
   
6. T. Y. W. Choi, P. A. Merolla, J. V. Arthur, K. A. Boahen and B. E. Shi, Neurmorphic Implementation of Orientation Hypercolumns, IEEE Trans. on Circuits and Systems-I: Regular Papers 52, pp. 1, 2005.