Augmented reality (AR) is a technology that provides perceptual experiences that blend the user’s real-world context with computer-generated virtual elements to form a unified scene. AR experiences situate the virtual in relation to the real world and then dynamically track changes in perspective to sustain the illusion that the virtual elements are “out there” in the world. AR is primarily a visual technology, but like the related technology of virtual reality (VR), spatial audio and haptic elements can be incorporated. The impact of AR experiences on users, especially on cognitive load and education, is a growing area of study in cognitive science.

History

The origins of AR overlap with that of VR: in the lab of Ivan Sutherland in Utah in 1968 (Lanier, 2017). The first prototypes overlaid computer graphics selectively on top of the user’s field of view to create a jointly real and virtual visual experience. The major challenge of AR is matching the virtual elements to the real-world context in such a way that it looks like they belong together, and it takes some element of computer vision (Hoff et al., 1996)—via a camera attached to a computing device—to retain that match as the world changes around the user or as the user changes their perspective on the world. The advent of smartphones in the late 2000s created the opportunity for widespread, consumer-facing AR.

Core concepts

There exists a wide disparity in the function and efficacy of different technologies that are labeled as AR and inconsistencies in the terminology used to refer to different variants. In general, it matters most how the device senses its relation to the world (tracking), how it represents the virtual elements (rendering), and what kind of display technology is used.

Tracking

Consumer AR deployments in the early 2010s used a detection trigger image—initially a QR code, later any image with distinctive contrast—to track the relative position of the real-world target area or object to the camera. Later, SLAM (simultaneous localization and mapping) technologies dispensed with the need for trigger images and instead tracked general context features such as the horizontal surfaces of tables or the ground. The Pokémon Go smartphone game became a worldwide phenomenon in 2016 using this technology, and since then, both Apple and Google have embedded AR engines (called AR Kit and AR Core, respectively) into their iOS and Android operating systems to make development and distribution of AR experiences easier.

Rendering

Tracking information, from a trigger or from SLAM, is used to render the appropriate point of view of a three-dimensional model and overlay this on the camera feed that the user sees. These calculations are undertaken by the graphics processor within the device and crucially require games/physics engine software (e.g., Unity) to generate 30 to 60 frames of animation per second to sustain the AR illusion whilst the camera or target are in motion.

Display: Augmented versus mixed reality

Most AR deployments mediate all of the user’s AR experience—the virtual and the real—via a screen. This is also the approach taken by recent Meta Quest headsets (Quest Pro and Quest 3 are especially capable versions) and the Apple Vision Pro device. In the current nomenclature, this is usually referred to as mixed reality. Apple refer to theirs as spatial computing.

In contrast, the approach taken by Microsoft with its HoloLens (2016) and by the rival Magic Leap One (2018) was to use transparent lenses that allowed the user to see the real world naturally and only project the virtual elements. This relatively unmediated version of the technology was initially branded mixed reality by Microsoft but has since become more commonly referred to as (true) AR.

The trajectory of the technology is towards (true) AR, but the devices to date have not become commercially viable. The technical challenges are considerable, relating primarily to optics (field of view, brightness of image) and secondarily to the size, weight, heat output, and cost of devices. Mixed reality via a screen represents a more achievable interim step and is the most common form of AR today, with an estimated 1.4 billion AR devices of varying levels of sophistication in the world today (Statista).

Questions, controversies, and new developments

The impact of AR on learning outcomes has been studied for almost three decades (Garzón, 2021) and remains a prominent and growing topic (Avila-Garzon et al., 2021). Areas of particular concern to cognitive scientists include spatial judgement (Shelton & Hedley, 2004), information retention (Huang et al., 2019), and in embodied approaches to learning (Abrahamson, 2014).

In both educational and industrial applications of AR, the topic of cognitive load (within cognitive load theory) is ubiquitous. Understanding the load experienced by users across different devices, tasks, and applications is vital to understanding the net impact of such adoption (Buchner et al., 2021). How load is measured and the different types of load (intrinsic, extrinsic, germane) remain challenges in the literature (Buchner et al., 2022)

Clinical applications of AR are also being studied (Vinci et al., 2020), in particular, the role of AR in exposure-based therapies (Riva et al., 2016).

As with the concept of VR before it (Savickaite et al., 2022), the lack of consensus regarding what counts as AR makes for inconsistent nomenclature across studies. This observation in turn makes literature reviews challenging and headline claims about AR ambiguous across significantly different technological approaches: mobile, wearable via screens, and wearable via transparent lenses.

Broader connections

Within cognitive science, there are clear connections with the broader topics of cognitive load (Sweller et al., 2019), education and learning (Poupard et al., 2024; Quintero et al., 2019), exposure therapy (Wechsler et al., 2019), and vision (Scarfe & Glennerster, 2015). Virtual content is often characterized as illusory [see Virtual Reality]. However, it is a controversial matter within philosophy whether the virtual content in AR (and VR) experiences are genuinely real and thus whether or not we should think of such overall experiences as genuinely illusory (see Chalmers, 2017). This question has potential ramifications in the law, especially as it regards to the apparent theft of virtual objects (Wildman & McDonnell, 2020).

On a practical level, access to high-end AR technology is not ubiquitous, and the cost and other technical barriers limit the scope of work. There are specific considerations around access to AR technology in education, including the need for inclusive design to cater to learners with diverse abilities, such as those with sensory impairments or neurodivergence, and the imperative to address equality issues arising from affordability to prevent disparities in educational opportunities (McDonnell et al., 2024). Sophisticated AR devices and the exciting use cases that they promise require sophisticated camera arrays and sensors to understand their environment and, in the case of head-worn devices, to understand the user via eye tracking, retina sensing, and more. These technical issues raise significant privacy and data security questions around AR technology for researchers and policy makers (Colburn et al., 2024).

Further reading

• Lanier, J. (2017). Dawn of the new everything: Encounters with reality and virtual reality. Henry Holt and Company.

• Queiroz, A., Bailenson, J., Blair, K., Schwartz, D., Thille, C., & Wagner, A. (2024). Extended realities and the future of knowledge work: Opportunities and challenges. Proceedings of the 31st IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, 662-666. https://doi.org/10.1109/VRW62533.2024.00130

• Scoble, R., & Israel, S. (2016). The fourth transformation: How augmented reality and artificial intelligence change everything. Patrick Brewster Press.