Shinsuke Shimojo

Integration Beyond Sensory Modalities

California Institute of Technology

Although brain localization and computational modularity of sensory modality are often claimed (sometimes even with regard to "innateness" of it), they tend to be overemphasized. Literature on neural plasticity of sensory systems in young animals and deprived humans, for example, is an abundant source for evidence against fixed brain localization and modularity. How much do we know, however, about perceptual integration across modalities in the normal adult humans? Our latest psychophysical findings also dispute localization/modularity in the strict sense. We have found that: (a) sounds can either improve, or degrade temporal order judgment in vision, depending on temporal relationship (1); (b) a sound (or a transient signal in visual or tactile modality) that is synchronized with visual crossing of two objects enhances bounce (collision) and suppresses streaming perception (2); and (c) a single flash can be seen as multiple flashes if it is accompanied with multiple sounds (3). These findings argue against the classical view of the human as a vision-dominated animal, and for crossmodal modulation at early cortical levels and transient signal as the key for perceptual integration. Our developmental study of the stream/bounce perception ((b) above) in human infants further indicates that infants first show behavioral evidence of adult-like categorical perception of a visual motion event modulated by a sound only after the age of 5-6 months, and that it is most likely constrained by development of spatial attention (4, 5). Perhaps the most classical view of crossmodal integration in the human was that it is primarily a vision-dominated animal. Due to a line of evidence indicating auditory modulation of vision in temporal domain, a modern "modality precision (appropriateness)" hypothesis was proposed. Our view yet differs from this; we believe that transient sensory signal is critical in crossmodal integration, regardless of whether the transient signal comes within the modality or across modalities (6).

References
(1) Scheier. C., Nijhawan, R., and Shimojo, S. (1999) Invest. Ophthal. Vis. Sci., 40(4), 4169.
(2) Watanabe, K. and Shimojo, S. (1998) Perception, 27, 1041-.
(3) Shams, L., Kamitani, Y. and Shimojo, S. (2000) Nature, 408, 788.
(4) Scheier, C., Lewkowicz, D., and Shimojo, S. (2000) Invest. Ophthal. Vis. Sci., 41(4), S327.
(5) Shimojo, S., Watanabe, K., and Scheier, C. In: Visual Attention and Cortical Circuits. J. Braun, C. Koch, and J. Davids (eds.), MIT Press (in press).
(6) Shimojo, S., and Shams, L. Sensory Modalities are Not Separate Modalities: Plasticity and Interaction Beyond Modalities. Current Opinion in Neurobiology, (in press).