Many of our daily activities are informed by vision, from reading and watching television to driving and sports. At the same time, visual perception during these activities is influenced by our knowledge, expectations, and goals. Visual cognitive neuroscience is the field of research that investigates the neural basis of the bidirectional interaction between visual perception and cognition, between seeing and thinking. Researchers in this field ask how the brain extracts meaningful information from visual input, how this information is represented in the visual cortex, and how visual cortex activity interfaces with cognitive functions such as memory and attention. While these questions have long been of interest to neuroscientists, advances in noninvasive brain imaging techniques have made it possible to study neural processes at the nexus of perception and cognition in humans and across the whole brain. This research has revealed cognitive influences on the functional organization of the visual cortex, has shown how visual cortex activity is modulated by cognitive states, and has started to shed light on the various ways in which the visual cortex supports cognitive tasks even without visual input, such as during visual imagery and visual working memory.
History
Much of our knowledge of the visual system originates from neurophysiological work in animals. Hubel and Wiesel recorded activity from single neurons in the visual cortex of anesthetized cats and monkeys (for a review, see Hubel & Wiesel, 1979). Subsequent neurophysiological investigations revealed organizing principles of the primate visual system, including a hierarchical organization: from neurons with small receptive fields (the portion of the visual field that a neuron responds to) that respond to specific visual features, to neurons with large receptive fields responding to more complex shapes, up to specific categories such as faces and hands (for a review, see Gross, 1994). While the possibility to use cognitive tasks in animals is limited and requires extensive animal training, some neurophysiological studies have also revealed the effects of attention and visual working memory on visual cortex activity (for a review, see Desimone & Duncan, 1995).
In parallel, neuropsychological studies have investigated the neural basis of visual cognition in patients with acquired brain damage, including from strokes or traumatic brain injuries. Patients with selective deficits in visual recognition tasks provided evidence for neural specialization. For example, patients with a selective impairment in recognizing faces after damage to the visual cortex (prosopagnosia; Bodamer, 1947) gave evidence that there were specialized neural mechanisms for face perception that had been compromised by the patients’ injuries. Another influential study reported a patient who was able to accurately grasp but not perceive objects (Goodale et al., 1991), supporting the neural distinction between vision for action (in the dorsal stream—a pathway running from the primary visual cortex to the parietal lobe) and vision for perception (in the ventral stream—a pathway running from the primary visual cortex to the temporal lobe; Goodale & Milner, 1992). Finally, studies have shown that brain damage can lead to impaired visual imagery with intact visual perception and vice versa, informing debates about the shared versus distinct neural mechanisms supporting imagery and perception (Behrmann et al., 1994; Farah, 1988).
Core concepts
Functional specialization in the visual cortex
Human neuroimaging studies have revealed particular circumscribed regions in the visual cortex that respond selectively to specific classes of visual stimuli, including faces [see Face Perception], bodies, scenes, tools, and words (Figure 1; Konkle & Caramazza, 2013; Kanwisher, 2010). These regions have been considered as nodes of domain-specific networks supporting core behaviors such as social cognition (faces, bodies), navigation (scenes), tool use (tools), and reading (words; Peelen & Downing, 2017). The reliable and selective response of these category-selective regions has enabled researchers to test cognitive models of attention, memory, expectation, and visual awareness (Downing et al., 2001). The discovery of category selectivity in the visual cortex has triggered many debates, including about the category specificity of the regions’ functions (Tarr & Gauthier, 2000; Ritchie et al., 2024), their innate versus acquired origins (Arcaro & Livingstone, 2021), the contributions of feedforward and feedback processes (Price & Devlin, 2011), and, more generally, the modularity of cognition (Coltheart, 1999).
Category-selective regions in the visual cortex of one example participant. Face-selective regions (in red) include the fusiform face area (FFA) and the occipital face area (OFA). Body-selective regions (in blue) include the fusiform body area (FBA) and the extrastriate body area (EBA). Scene-selective regions (in green) include the parahippocampal place area (PPA) and the transverse occipital sulcus (TOS). Adapted from Konkle & Caramazza (2013) under a CC-BY-NC-SA license. Icons by Flaticon.com.
Cognitive influences on visual cortex activity
Neural activity in the visual cortex is shaped by cognitive processes, including attention, motivation, emotion, and expectation. Attention, motivation, and emotion enhance visual processing in favor of behaviorally relevant, previously rewarded, or emotionally salient visual stimuli (Kastner & Ungerleider, 2000; Serences, 2008; Vuilleumier, 2005) [see Attention]. By contrast, expectation reduces visual cortex activity to expected stimuli (Summerfield & Egner, 2009). According to predictive processing theories, visual perception is an inference process, in which internal models (informed by knowledge, context, and experience) are generated to match and “explain away” visual input (Friston, 2005; Hohwy, 2016) [see The Free Energy Principle, Bayesian Models of Cognition]. On this account, feedforward visual processing reflects the unexplained (i.e., unexpected) visual input, which is used to update the internal model and thereby improve subsequent predictions. Together, the modulatory effects of attention, motivation, emotion, and expectation align visual cortex activity with the observer’s interpretation and subjective perception of the visual environment.
Role of the visual cortex in cognition
The visual cortex can be selectively activated in the absence of visual input, for example, when listening to the names of objects (Martin, 2007). In some visual cortex regions, such internally driven activity can also be observed in people who are born blind, raising the possibility that representations in these areas are not exclusively visual (Bi et al., 2016). Internally-driven visual cortex activity has also been observed during visual mental imagery, visual working memory, retrieval of autobiographical memory, and language comprehension (Pearson, 2019). These findings indicate that the visual cortex may play a broader role in cognition, although its specific and causal contribution to these processes is still debated (Binder & Desai, 2011; Xu, 2017).
Questions, controversies, and new developments
A prominent new development is the modeling of the visual cortex using deep neural networks (DNNs). Research has shown correspondence between the feedforward representations in DNNs and representations in visual cortex (Yamins & DiCarlo, 2016). However, differences remain (Bowers et al., 2022), and DNNs are not yet able to accurately model higher-level cognitive influences on visual cortex activity or tailor visual representations to the tasks humans perform in daily life (e.g., navigation, reading).
Other new developments include relating individual differences in visual cognition to individual differences in brain anatomy and function (Dubois & Adolphs, 2016) and using recent advances in neuroimaging methods to study visual cognition, for example, using high-resolution imaging to test the cortical layer specificity of cognitive influences on visual processing (Lawrence et al., 2019). Finally, researchers have started to use more naturalistic visual stimuli and tasks to make findings more generalizable to daily-life environments (Snow & Culham, 2021; Willems & Peelen, 2021).
Broader connections
Because vision is the dominant input modality for many types of behaviors, the field of visual cognitive neuroscience is closely connected to other areas of cognitive neuroscience, including sensorimotor neuroscience, the neuroscience of language, and social cognitive neuroscience. Furthermore, findings in the field of visual cognitive neuroscience, such as the effects of attention and expectation, may similarly apply to other sensory modalities (e.g., audition). Finally, internally driven activity in the visual cortex has been implicated in multiple cognitive processes, including memory, language, and the representation of concepts [see Concepts]. Thereby, research in visual cognitive neuroscience informs broader philosophical debates about the grounded nature of cognition (Barsalou, 2008) and the distinction between perception and cognition (Pylyshyn, 1999).
Further reading
Bracci, S., & Op de Beeck, H. P. (2023). Understanding human object vision: A picture is worth a thousand representations. Annual Review of Psychology, 74(1), 113–135. https://doi.org/10.1146/annurev-psych-032720-041031
Pearson, J. (2019). The human imagination: The cognitive neuroscience of visual mental imagery. Nature Reviews Neuroscience, 20(10), 624–634. https://doi.org/10.1038/s41583-019-0202-9
Peelen, M. V., Berlot, E., & de Lange, F. P. (2024). Predictive processing of scenes and objects. Nature Reviews Psychology, 3(1), 13–26. https://doi.org/10.1038/s44159-023-00254-0
Storm, J. F., Klink, P. C., Aru, J., Senn, W., Goebel, R., Pigorini, A., Avanzini, P., Vanduffel, W., Roelfsema, P. R., Massimini, M., Larkum, M. E., & Pennartz, C. M. A. (2024). An integrative, multiscale view on neural theories of consciousness. Neuron, 112(10), 1531–1552. https://doi.org/10.1016/j.neuron.2024.02.004
References
Arcaro, M. J., & Livingstone, M. S. (2021). On the relationship between maps and domains in inferotemporal cortex. Nature Reviews Neuroscience, 22(9), 573–583. https://doi.org/10.1038/s41583-021-00490-4
↩Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59(1), 617–645. https://doi.org/10.1146/annurev.psych.59.103006.093639
↩Behrmann, M., Moscovitch, M., & Winocur, G. (1994). Intact visual imagery and impaired visual perception in a patient with visual agnosia. Journal of Experimental Psychology: Human Perception and Performance, 20(5), 1068–1087. https://doi.org/10.1037/0096-1523.20.5.1068
↩Bi, Y., Wang, X., & Caramazza, A. (2016). Object domain and modality in the ventral visual pathway. Trends in Cognitive Sciences, 20(4), 282–290. https://doi.org/10.1016/j.tics.2016.02.002
↩Binder, J. R., & Desai, R. H. (2011). The neurobiology of semantic memory. Trends in Cognitive Sciences, 15(11), 527–536. https://doi.org/10.1016/j.tics.2011.10.001
↩Bodamer, J. (1947). Die prosop-agnosie. Archiv für Psychiatrie und Nervenkrankheiten, 179, 6–53. https://doi.org/10.1007/BF00352849
↩Bowers, J. S., Malhotra, G., Dujmović, M., Llera Montero, M., Tsvetkov, C., Biscione, V., Puebla, G., Adolfi, F., Hummel, J. E., Heaton, R. F., Evans, B. D., Mitchell, J., & Blything, R. (2022). Deep problems with neural network models of human vision. Behavioral and Brain Sciences, 46, e385. https://doi.org/10.1017/S0140525X22002813
↩Coltheart, M. (1999). Modularity and cognition. Trends in Cognitive Sciences, 3(3), 115–120. https://doi.org/10.1016/S1364-6613(99)01289-9
↩Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, 193–222. https://doi.org/10.1146/annurev.ne.18.030195.001205
↩Downing, P., Liu, J., & Kanwisher, N. (2001). Testing cognitive models of visual attention with fMRI and MEG. Neuropsychologia, 39(12), 1329–1342. https://doi.org/10.1016/S0028-3932(01)00121-X
↩Dubois, J., & Adolphs, R. (2016). Building a science of individual differences from fMRI. Trends in Cognitive Sciences, 20(6), 425–443. https://doi.org/10.1016/j.tics.2016.03.014
↩Farah, M. J. (1988). Is visual imagery really visual? Overlooked evidence from neuropsychology. Psychological Review, 95(3), 307–317. https://doi.org/10.1037/0033-295X.95.3.307
↩Friston, K. (2005). A theory of cortical responses. Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456), 815–836. https://doi.org/10.1098/rstb.2005.1622
↩Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15(1), 20–25. https://doi.org/10.1016/0166-2236(92)90344-8
↩Goodale, M. A., Milner, A. D., Jakobson, L. S., & Carey, D. P. (1991). A neurological dissociation between perceiving objects and grasping them. Nature, 349(6305), 154–156. https://doi.org/10.1038/349154a0
↩Gross, C. G. (1994). How inferior temporal cortex became a visual area. Cerebral Cortex, 4(5), 455–469. https://doi.org/10.1093/cercor/4.5.455
↩Hohwy, J. (2016). The self‐evidencing brain. Noûs, 50(2), 259–285. https://doi.org/10.1111/nous.12062
↩Hubel, D. H., & Wiesel, T. N. (1979). Brain mechanisms of vision. Scientific American, 241(3), 150–163. https://doi.org/10.1038/scientificamerican0979-150
↩Kanwisher, N. (2010). Functional specificity in the human brain: A window into the functional architecture of the mind. Proceedings of the National Academy of Sciences, 107(25), 11163–11170. https://doi.org/10.1073/pnas.1005062107
↩Kastner, S., & Ungerleider, L. G. (2000). Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience, 23(1), 315–341. https://doi.org/10.1146/annurev.neuro.23.1.315
↩Konkle, T., & Caramazza, A. (2013). Tripartite organization of the ventral stream by animacy and object size. Journal of Neuroscience, 33(25), 10235–10242. https://doi.org/10.1523/JNEUROSCI.0983-13.2013
↩Lawrence, S. J. D., Formisano, E., Muckli, L., & de Lange, F. P. (2019). Laminar fMRI: Applications for cognitive neuroscience. NeuroImage, 197, 785–791. https://doi.org/10.1016/j.neuroimage.2017.07.004
↩Martin, A. (2007). The representation of object concepts in the brain. Annual Review of Psychology, 58(1), 25–45. https://doi.org/10.1146/annurev.psych.57.102904.190143
↩Pearson, J. (2019). The human imagination: The cognitive neuroscience of visual mental imagery. Nature Reviews Neuroscience, 20(10), 624–634. https://doi.org/10.1038/s41583-019-0202-9
↩Peelen, M. V., & Downing, P. E. (2017). Category selectivity in human visual cortex: Beyond visual object recognition. Neuropsychologia, 105, 177–183. https://doi.org/10.1016/j.neuropsychologia.2017.03.033
↩Price, C. J., & Devlin, J. T. (2011). The interactive account of ventral occipitotemporal contributions to reading. Trends in Cognitive Sciences, 15(6), 246–253. https://doi.org/10.1016/j.tics.2011.04.001
↩Pylyshyn, Z. (1999). Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. Behavioral and Brain Sciences, 22(3), 341–365. https://doi.org/10.1017/S0140525X99002022
↩Ritchie, J. B., Wardle, S. G., Vaziri-Pashkam, M., Kravitz, D. J., & Baker, C. I. (2024). Rethinking category-selectivity in human visual cortex. arXiv. https://doi.org/10.48550/arXiv.2411.08251
↩Serences, J. T. (2008). Value-based modulations in human visual cortex. Neuron, 60(6), 1169–1181. https://doi.org/10.1016/j.neuron.2008.10.051
↩Snow, J. C., & Culham, J. C. (2021). The treachery of images: How realism influences brain and behavior. Trends in Cognitive Sciences, 25(6), 506–519. https://doi.org/10.1016/j.tics.2021.02.008
↩Summerfield, C., & Egner, T. (2009). Expectation (and attention) in visual cognition. Trends in Cognitive Sciences, 13, 403–409. https://doi.org/10.1016/j.tics.2009.06.003
↩Tarr, M. J., & Gauthier, I. (2000). FFA: A flexible fusiform area for subordinate-level visual processing automatized by expertise. Nature Neuroscience, 3(8), 764–769. https://doi.org/10.1038/77666
↩Vuilleumier, P. (2005). How brains beware: Neural mechanisms of emotional attention. Trends in Cognitive Sciences, 9(12), 585–594. https://doi.org/10.1016/j.tics.2005.10.011
↩Willems, R. M., & Peelen, M. V. (2021). How context changes the neural basis of perception and language. iScience, 24(5), 102392. https://doi.org/10.1016/j.isci.2021.102392
↩Xu, Y. (2017). Reevaluating the sensory account of visual working memory storage. Trends in Cognitive Sciences, 21(10), 794–815. https://doi.org/10.1016/j.tics.2017.06.013
↩Yamins, D. L. K., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3), 356–365. https://doi.org/10.1038/nn.4244
↩