As soon as you open your eyes, you perform something that is referred to as scene perception, a cognitive process by which the brain interprets and understands its visual environment. It involves recognizing, organizing, and extracting meaningful information from complex scenes, enabling humans to identify objects, their relationships, spatial layout, and relevant features within a given environment. This cognitive process is key to navigating and interacting effectively with our surroundings. A hallmark of scene perception is that it takes only a brief glimpse to extract rich meaning from complex visual input (see Figure 1). The remarkable speed and ease with which we process scenes is the product of swiftly integrating bottom-up visual information with top-down scene knowledge. It is this fascinating cognitive feat that will be covered here.

History

Although formal research on “scene perception” as a defined concept emerged in the 20th century, earlier philosophical and observational writings already touched on related ideas about visual perception and the human ability to interpret environments. Hermann von Helmholtz's (1867/1962) theory of unconscious inference, for instance, suggests that perception is an active process in which the brain makes implicit assumptions or inferences to interpret sensory information, particularly in complex or ambiguous scenes. This idea posits that our perception of scenes is not a direct reflection of the sensory input but results from the brain’s unconscious guessing and hypothesis testing based on prior knowledge and context.

In the early 1910s, Max Wertheimer and his colleagues Wolfgang Köhler and Kurt Koffka famously founded the Gestalt School (“Gestaltschule”) in Germany, which significantly contributed to understanding perceptual organization and gestalt principles (e.g., that “the whole is perceived as more than just the sum of its parts”). However, it was not until the 1970s and 80s that researchers like David Marr, Molly Potter, and Irv Biederman systematically put the feats of scene perception to a test. Their seminal works provided some core insights on scene perception, for example, that visual perception involves transforming retinal input into a series of increasingly abstract, computational representations that enable scene understanding (Marr, 1982; see also Groen et al., 2022); that the briefest of glances at a complex scene is enough to process its meaning (Potter, 1975; see also Greene & Oliva, 2009); and that scene semantics and syntax play a key role in the processing of objects therein (Biederman, 1976; see also Võ, 2021).

Core concepts

The scene grammar framework

The scene grammar framework (see Figure 2; Võ et al., 2019; Võ, 2021) proposes that objects within scenes are hierarchically organized and constrained by scene grammar—internalized rules that predict what objects tend to be where within a scene, how objects are positioned relative to one another, and how objects relate to the global scene context. Within a scene, so-called anchor objects (e.g., the shower or bathtub) guide the placement, identification, and use of associated objects (e.g., the shampoo inside the shower or the towels next to it), forming meaningful and functionally optimized clusters or phrases. This highly structured composition of objects in scenes helps the brain efficiently process complex visual environments, boosting efficient object recognition and search as well as scene understanding, all of which enable us to efficiently adapt and function in a diverse range of settings (Biederman, 1976; Draschkow & Võ, 2017; Henderson & Ferreira, 2004; Võ & Henderson, 2009; Võ & Wolfe, 2013).

What guides attention in real-world scenes?

A red poppy along a lush green hiking path might grab your attention, or a ball that your kid is launching your way. The view that attention is guided by low-level features like color contrast or movements was prominently featured in Itti & Koch's (2000) saliency model, but later accounts favored a greater role for top-down guidance of attention. For instance, Nuthmann & Henderson (2010) found preferred viewing location (PVL) close to the center of objects within naturalistic scenes, arguing that attentional selection in scenes is inherently object-based rather than saliency driven. Keeping visual saliency controlled, Võ and Henderson (2009), for instance, showed that objects violating scene grammar (e.g., a fire hydrant in a kitchen) keep attracting eye movements once fixated (it has been debated whether semantic or syntactic violations actually attract attention parafoveally, i.e., from the corner of your eye, or not; Võ & Henderson, 2009). And, more recently, deep neural networks like DeepGaze III (Kümmerer et al., 2022)—which combines image information with the history of previous fixations—have been shown to predict where observers might look next while freely viewing a scene. Most of the time, however, we do not just passively view scenes but instead search for objects and interact with them for the purpose of goal-directed actions. Try searching for the shampoo in Figure 3.

Most likely, scene grammar allowed you to ignore the left two-thirds of the scene (i.e., the toilet and sink phrases) and to limit your search to the shower phrase, likely to contain the target. And before deciding that the detergent is absent from this scene, you would likely open the cupboard and find it inside.

Questions, controversies, and new developments

As in many other disciplines, the rise of deep neural networks has opened up new possibilities for answering long-standing questions. A particular type of deep neural network—so-called generative adversarial networks, or GANs (Goodfellow et al., 2014), for instance—are becoming increasingly good at “hallucinating” images of scenes that do not actually exist. What exactly is the scene grammar that GANs have learned (see Kallmayer & Võ, 2024)? Lesioning GANs could serve as a useful testbed to investigate what actually makes a scene.

Lately, the classic notion of object affordances à la Gibson has been reintroduced to scene perception [see Affordances]. How do functions affect scene categorization (Greene et al., 2016), and which parts of a scene drive function understanding (Müller Karoza et al., 2025)? Making use of more interactable three-dimensional virtual environments will likely provide new answers to some old questions.

Although often merely used as an analogy, processing objects in scenes and words in sentences might actually share common, domain-general cognitive mechanisms. Do, for example, children with developmental language delay also show impediments in the processing of pictorial information (Bahn et al., 2025; Lindfors et al., 2025)? Investigating the commonalities and differences between scene perception and language processing has the potential to improve the understanding of both of these core cognitive abilities (for some initial thoughts on this matter, see also Henderson & Ferreira, 2004)

Broader connections

The study of scene perception is inherently interdisciplinary. As briefly outlined above, knowing a scene’s grammar will strongly affect visual search in naturalistic environments providing a highly efficient, top-down guidance of attention that allows detecting objects in scenes even if they are occluded or even hidden from view [see Visual Search; Attention]. Also, people interested in memory have been intrigued by what features of a scene make it more or less memorable (Guo & Bainbridge, 2025), and recently deep convolutional neural networks trained on large datasets have been shown to successfully predict memorability scores for new images (for a review, see Rust & Mehrpour, 2020).

Acknowledgments

This work was supported by the Deutsche Forschungsgemeinschaft (DFG) under Germany’s Excellence Strategy (EXC 3066/1 “The Adaptive Mind,” Project No. 533717223) as well as by the DFG Research Unit FOR 5368.

Further reading

• Bartnik, C. G., & Groen, I. I. A. (2023). Visual perception in the human brain: How the brain perceives and understands real-world scenes. Oxford Research Encyclopedia of Neuroscience. https://doi.org/10.1093/acrefore/9780190264086.013.437

• Hansen, B. C., Greene, M. R., Lewinsohn, H. A. S., Kris, A. E., Smyth, S., & Tang, B. (2025). Brain-guided convolutional neural networks reveal task-specific representations in scene processing. Scientific Reports, 15, 13025. https://doi.org/10.1038/s41598-025-96307-w

• Wiesmann, S. L., & Võ, M. L.-H. (2025). Flexible usage of object and global scene information during human scene categorization. Journal of Experimental Psychology: Human Perception and Performance. https://doi.org/10.1037/xhp0001342