Visual Cognitive Processing Reflected in Task-Evoked Pupillary Responses: Pupil-Size Dynamics as an Indicator of Cognitive Load and Mental Effort
Introduction
Pupil-size dilation dynamics have been extensively studied within the context of research on cognition, spanning from areas covering how the pupil-size is modulated in experiments involving attentional processing (working memory capacity, cognitive load, and task performance), arousal associated with different emotional states (surprise, exhaustion, sexual attraction), and overall emotional valence (pleasantness or unpleasantness of different stimuli) (Joshi & Gold, 2020). The size of the pupil can also be used as an indicator of certain psychological states. For example, larger pupils are often associated with increased arousal and interest, while smaller pupils are often associated with decreased arousal and disinterest (). Pupil size has also been found to be related to memory and attention, with some studies suggesting that larger pupils may be associated with improved memory and attention compared to smaller pupils (Kucewicz et al, 2018). Many attentional processing studies aim to uncover potential correlations, and overall associations, between pupil-size dynamics across a time-series as a subject completes a specific type of cognitive test, in relation to the difficulty of the task, coupled with the subject’s cognitive processing abilities, such as performance on the task and reaction times in response to various probe stimuli.
Pupil-Size Phenomena: Cognitive Load and Working Memory Capacity
The general phenomenon observed in various studies shows that pupil-size dilations covary with cognitive load and mental effort in working memory tasks—suggesting that pupil-size also reflects overall task difficulty in cognitive control tasks. Cognitive load refers to the mental effort required to process and understand information—in the context of psychophysical experiments, researchers have examined how different visual stimuli and varying task difficulty in cognitive tasks impact pupil-size, where task difficulty is often used as a measure of cognitive load, and referred homologously to the mental effort required of a subject to complete a particular task. A more difficult task corresponds to higher cognitive load, and requires a subject to use more of their mental resources required to complete the task. In essence, cognitive load quantifies the demands placed on an individual’s working memory, which is a short-term form of memory that temporarily stores and accesses information.
Studies have shown that pupil-size dilation increases in response to increased cognitive load, such as when a person is engaged in a challenging or complex task—often times, this increase is reflected in a pupil-size dilatory spike that occurs when a subject is required to make a decision or answer the task in response to a probe stimulus. Many studies involving working memory tasks link cognitive load and mental effort to attentional effort, linking increased or decreased performance on a task to improved or decreased attention to stimuli presented in working memory tasks (Kang et al., 2014)—pupil-size, thus, has also been associated with attentional effort reflected in the cognitive load of an experiment, which in turn impacts the subject’s performance on the task.
This relationship between pupil size and cognitive load has been observed in a variety of contexts, such as in reading, problem-solving, decision-making, visual search, and working memory tasks. When cognitive load is high, the brain is required to work harder and use more mental resources—research in accordance with this assumption has shown that pupil size increases, or dilates, in response to high cognitive load, which has been posited as a strong reflection of increased mental exertion on a subject’s working memory capacity (van der Wel, 2018). Many prior studies in the past have shown relations between cognitive load and pupil-size using the N-back task, where task difficulty can be manipulated by adjusting the value of N—larger values of N correspond to greater difficulty. This task requires subjects to indicate whether a letter, number, or stimulus presented currently had been presented N trials back. A collection of studies have shown that pupil-size dilation increases as N increases, ultimately when the cognitive load demanded by the task increases in difficulty. One study employed classification algorithms to predict high or cognitive load for a task using solely pupil-size metrics with up to 75% accuracy through a synthesized computational predictive approach (van der Wal, 2018). Whereas another study highlighted how increased pupil-size dilations in response to target stimuli were correlated with improved performance on an N-back task—lending credence to the notion that, as a potential index of cognitive load and mental effort, greater pupil-size dilations reflect greater mental effort and improved performance (van der Wal, 2018). In a visual search experiment, the authors observed that the accumulation of memory load over the course of a visual search corresponded to elevated pupil-size responses (Attar et al., 2016).
Thus, there is a potential nuance in the two-way relationship between both variables—on the one hand, when more mental effort is required, it may cause the pupil to dilate more through an adaptive mechanism to improve alertness or attention, triggered by activation of areas in the brain such as the locus coeruleus; on the other hand, perhaps more speculatively, pupil-size dilations may also in turn influence cognitive load in some way and enhance attention based on these observed correlations. While there are no definitive answers to this phenomenon, there is a clear positive correlation between cognitive load and pupil-size based on the extensive research that has been conducted, and is an area of speculation since the adaptive utility of pupil-size dynamics has not yet been grounded in fact.
Overview of Pupil-Size Anatomy and Neural Basis
Figure 1. Diagram of pupils within iris, showing the parasympathetic and sympathetic pathways that modulate pupil-size diameter, as well as the different dilator muscles surrounding the iris (Eckstein et al., 2017).
The pupil is the black circular opening in the center of the eye, and is primarily responsible for regulating the amount of light that enters the eye—light passes through the lens and into the retina, which is exposed by the pupil upon entrance. Pupil-size can vary from 2 to 8 millimeters in diameter in a healthy adult (Mathot, 2018).
The pupil is located in the colored part of the eye known as the iris—a muscular diaphragm that surrounds the pupil and modulates the size of the pupil depending on different lighting conditions. The iris consists of two sets of muscles: the sphincter pupillae and the dilator pupillae. The sphincter pupillae muscles constrict the pupil, while the dilator pupillae muscles dilate the pupil. Pupil-size is modulated primarily by the parasympathetic and sympathetic pathways, as shown in Figure 1. The parasympathetic pathway in the nervous system functions to maintain homeostasis by restoring the body to a calm state and slowing down movement in the nervous system—within the context of eye movements, this pathway constricts the pupil (Mathot, 2018). The sympathetic pathway conversely activates certain responses and areas in the nervous system, such as fight or flight, and is responsible for modulating heart rate, blood pressure, perspiration, and overall respiratory functions—within the context of eye movements, this pathway dilates the pupil (Matho, 2018). Importantly, from a neural perspective, the distinction between these pathways offers insight into why variational dynamics in pupil-size dilations, in particular, are potentially more reliable markers of cognition than the dynamics observed in pupil-size constrictions.
Pupil-size dilations, as opposed to constrictions, likely offer more insights into cognitive control, as the subcortical dilation pathway starts at the hypothalamus and the locus coeruleus (LC) , eventually connecting to the iris dilator muscle (Mathot, 2018)—the LC, particularly, is the site of norepinephrine (NE) transmissions, a key neurotransmitter involved in modulating arousal, alertness, and attention (Eckstein et al., 2016). Moreover, the hypothalamus reflects overall arousal and wakefulness—in tandem, both these structures are indicative of overall mental activation and arousal and connected directly to the dilatory pathways of the pupil. Conversely, the pupil constricts in response to activation of parasympathetic neurons in the Edinger-Westphal (EW) nucleus, which sends innervations to the iris sphincter muscle (Eckstein et al., 2016). And originating from the sympathetic pathway, a type of “activation” system that stimulates the nervous system—especially within the context of mental overload, exertion, or stress—pupil-size dilations inherently pose a potentially reliable way of modeling how increased cognitive load increases pupil-size.
Neural Basis for Pupil-size Effects on Cognitive Load
The LC, importantly, offers a potential neural basis for the positive correlations between pupil-size dilations and cognitive load—from a neurobiological perspective, pupil-size changes have been shown to indirectly affect noradrenergic activity in the brain, originating from the LC, which regulates physiological arousal—the LC-NE system releases norepinephrine, a neuromodulator involved in sensory processing, behavior, attention, and working memory (Eckstein et al., 2016).
The LC itself is a small nucleus in the brainstem that modulates physiological arousal and cognitive functioning. It projects out to the cerebral cortex, hippocampus, thalamus and midbrain—the parietal cortex, pulvinar nucleus, superior colliculus are also sites of particularly dense LC-NE projections—these areas have been also linked to attentional processing (Eckstein et al., 2016). Significantly, along with connections to the dilation pathway, the LC has inhibitory connections to the parasympathetic pathway, where the pupil’s constricting fibers originate—projections originating from the LC connect to the constricting and dilator muscles of the pupil. As such, the LC, when activated and promoting arousal and wakefulness, both evokes pupil-size dilations through the sympathetic pathway, while also simultaneously inhibiting constriction of the pupil.
Further evidence of this neural basis is illustrated in Figure 2, which plots the pupil-size dilations of a rhesus monkey fixating during a target detection alongside the firing rates of a single LC neuron collected over time (Rajikowski et al., 1993). Visually, there seems to be a strong correlation between both variables, as they follow the same trajectories across the time-series—Rajikowski et al.’s experiment here has been cited frequently as strong evidence for the connections between LC activation and pupil-size dilations.
Moreover, the sympathetic and parasympathetic pathways have been said to reflect overall activity of neurotransmitters such as catecholamine and acetylcholine, which play a further active role in particularly regulating mental effort—it has been proposed hat acetylcholine helps subjects maintain task performance under stressful conditions, with a neural basis originating from the basal forebrain, an area of the brain that regulates body temperature (van der Wel, 2018). Acetylcholine systems originate from an area of the brain known as the anterior cingulate cortex (ACC), which is largely responsible for behavior regulation—it has been suggested that this area monitors the environment for cognitive demands, which was furthered by an additional study concluding that the ACC helps in assessing costs and benefits of mental resource allocation in response to cognitive load, by communicating with other areas of the brain such as the insula, ventral prefrontal cortex, striatum, and midbrain—the ACC acts as a functional module that communicates with other areas of the brain in global workspace theory, and similar to functional specialization of the cortex where different areas of the brain are modulated for specific tasks and communicate with each other (van der Wel, 2018).
Significantly, the LC has been proposed to also communicate with the ACC regarding information on cognitive task demands and optimizing for cognitive control. The ACC, evidently, may play a role in a type of sensorimotor loop that continuously takes in sensory input and functions by assessing mental state and exertion, and in turn modulating the resources that can be devoted to cognitive control through communication with the LC-NE system. The ACC has yet to be further studied to verify this neural basis, but offers another potential perspective in communicating with the LC and in turn impacting pupil-size dilations to respond to the demands placed on the brain.
Figure 2. Time-series graph of pupil-size dilations alongside firing rates of a single LC neuron in a rhesus monkey during fixation in a target detection task (Rajkowski et al., 1993)
Pupillary Light Reflex
A key principle discussed in visual neuroscience is dark and light adaptation by the eye—in the context of pupil-size changes, pupil size is fundamentally regulated by the pupillary light reflex, which is a reflexive response to changes in light intensity, a mechanism that helps to regulate the amount of light that enters the eye. In darker environments, the pupil adapts by dilating the people to allow for more light to enter the eye and perceive the world, whereas in lighter environments, pupils constrict to moderate the amount of light that enters the eye and minimize overexposure to light. Within the context of cognitive load experiments, controlling for luminance intensity, both in the environment, and screen luminance, is crucial for canceling out the potential effects of the pupillary light reflex, or generally lighting on pupil-size.
Measuring Pupil-Size in Working Memory Experiments
Figure 3. Visuospatial working memory task administered by Wainstein et al. (2017) for ADHD positive and negative subjects. Consisted of low and high loads, varying in cognitive load.
In behavioral experiments, within the context of manipulating cognitive load and ascertaining mental effort exerted by a subject, there are a selection of different types of psychophysical working memory tasks that researchers employ to monitor pupil-size simultaneously: the most common psychophysical tasks include visuospatial working memory tasks (VSWM), N-back tasks, word recognition and recall tasks, digit span tasks, visual search tasks, as well as the Sternberg task (van der Wel, 2018). The Sternberg task tests for working memory by administering a string, or list, of letters or digits that are encoded in the first part of the trial, and after a few seconds, a subject is prompted with a probe stimulus and required to determine if the probe had been presented before. The digit span task is similar, but requires a subject to first encode a list of digits varying in length, and a few seconds later recall the digits in order. A visual search task consists of finding a visual stimulus among a group of distractors.
Each is a type of working memory task that requires a subject to keep track of stimuli that had been presented before—in cognitive load experiments, researchers vary the task difficulty with respect to the particular task administered: for example, in a visual search task, researchers vary the number and heterogeneity, or types, of distractors; in an N-back task, researchers vary N, the number of back-items a subject must recall; in the Sternberg or digit span task, researchers vary the content, length, and number of letters or digits required to be encoded.
Figure 3 illustrates an example of a visuospatial working memory task administered by Wainstein et al., who monitored pupil-size as ADHD positive and negative patients completed the task—importantly, their task consists of a low and high load variable, which modulates the task difficulty. Given a series of dot arrays shaped as 4x4 grids, consisting of either one dot (low load) or two dots (high load) at each time interval, subjects had to determine if a probe stimulus consisting of a singular dot in the grid had been presented in that location in any of the prior dot arrays (Wainstein et al., 2017). As will be discussed later, their paradigm for analyzing pupil-size variations within the context of a neurobehavioral disorder like ADHD, and recording responses dependent on cognitive load, signifies the clinical relevance of analyzing pupil-size in the context of cognitive load, as a potential mechanism for gauging other behaviors such as attention, memory, and sensory perception.
In working memory tasks, generally, there are often different levels of load administered (low, medium, high) that researchers vary in order to gauge its associations with pupil-size dilations. The experimental conditions, and variations, of the task are correlated with pupil-size measurements to observe potential relationships between dilation dynamics and task difficulty as a proxy for mental effort, for example. Common behavioral variables that are collected, alongside pupil-size changes over time, include task accuracy (overall performance), reaction times, and working memory capacity.
In concurrently tracking neural activity in the brain, which could potentially correlate with pupil-size changes, researchers use mainly EEG based measurements and fMRI studies to gauge brain activity and activation in different areas—Figure 2, for example, illustrates EEG data capturing the firing rates of single LC neurons; this data was correlated with pupil-size, as shown in the visualization, to uncover and highlight the relationships between pupil-size dilations and activity in the LC-NE system. fMRI paradigms are often used to model functional activity in different parts of the brain across time, and how these different parts are synthesized with each other and reflected in pupillary changes—Fietz et al., as will be discussed later, developed an experiment using fMRI bold responses and regression techniques to highlight the correlations between different parts of the brain and pupil-size changes in varying cognitive load conditions. Within the context of concurrently monitoring pupil-size and potential neuronal activity, EEG and fMRI based approaches are often employed to draw further connections between both pupil-size and neural activity in the brain.
Modeling Cognitive Load Using Pupil-Size Dilations and Microsaccades
In “Eye Tracking Cognitive Load Using Pupil Diameter and Microsaccades with Fixed Gaze,” Krejtz et al. describe a methodology for using microsaccadic eye movements as an alternative mechanism for gauging cognitive load than pupil-size dynamics—they measured microsaccades and pupil-size diameters as subjects completed mental calculation tasks, and compared these metrics with each other; they specifically compared microsaccadic rate, magnitude, and peak velocity and magnitude, with that of inter- and intra-trial baseline pupil-size dynamics data. Krejtz et al. suggest that microsaccades can be a potentially better marker of cognitive effort and load than pupil-size.
Methods
The authors replicated the methodology proposed by Siegenthaler et al. for analyzing microsaccadic movements while subjects completed mental arithmetic. The task consisted of three levels of difficulty: Difficult, Easy, and Control. For the Difficult and Easy tasks, subjects were asked and required to perform difficult or easy mental calculations, respectively. In the Control task, subjects were not required to perform any type of mental calculation. The authors employed a within-subjects design, in which all participants were required to complete all difficulty levels of the mental calculation task—the authors employed a 3x6 within-subjects design, in which each subject was required to complete 6 trials for each of the 3 difficulty levels, or loads.
Prior to completing the actual experiment, the authors required that the subjects complete a DSPAN (Digit Span) task to gauge each subject’s Working Memory Capacity (WMC), which was to serve as an additional independent variable in their statistical analyses. The DSPAN procedure consisted of 14 trials, where in each trial, subjects were presented with a random number on a computer screen for 1s—the starting number was a three digit number. The task was to recall the number in the same or reverse order, corresponding to a “Forward” or “Backward” assessment, respectively. If the subject’s response was correct, then the number of digits would increment by one for the following trials, and if incorrect, held constant. The presentation of Forward and Backward assessments was randomized for each trial.
The main task that each subject had to complete was 18 total trials, where trials were grouped into 6 blocks. Each block started with a Control trial, followed by the Easy, and then Difficult trials alternating in order per block of trials. During each trial, subjects had to center their gaze on a fixation point. For Difficult trials a number was randomly chosen from a list of {1375, 8489, 5901, 5321, 4819, 1817}, and in Easy Trials a number was randomly chosen from {363, 385, 143, 657, 935, 141}. In the Difficult trials, subjects were required to count backwards in steps of 17 as fast and accurately as possible, and in Easy trials, subjects were required to count forward in steps of 2, both starting from the randomly chosen number presented. Subjects had to enter their answers onto a computer. In Control trials, subjects did not have to complete any mental computation.
The experiment in total consisted of 13 participants: 7 males and 6 females from the ages of 20 to 40. Eye movements were recorded using an SR Research EyeLink 1000 with 500hz sampling rate, captured binocularly. Ambient lighting in the room was controlled for, and luminance on the computer screen was also controlled for at 120-130 lux.
Pupil-Size Metrics
Starting with the results obtained from the pupillometric data, the authors took a computational and statistical based approach, and calculated two metrics: CPD (intra-trial change in pupil diameter) and BCPD (inter-trial change in pupil diameter). The computations for CPD were setup as a complex mathematical equation applied to smoothed raw pupil size diameter data—in essence, CPD kept track of a running mean over a baseline time period within a single trial, and calculated the mean difference between the pupil diameter and the average over a baseline time period and an “extent” time period within that trial. Essentially, CPD was computed as a metric for assessing the direction and magnitude of pupil size dilations, on average, within each trial. The inter-trial change in pupil-size, BCDP, was calculated in the same way as CPD, where the baseline time period was obtained from another trial, and the extent time period subject to the trial at hand—in essence, the BCPD quantified the magnitude and direction of pupil-size changes, on average, across different trials, but more importantly, different task difficulties.
Microsaccade Metrics
The authors mainly examined microsaccadic peak and average velocity and magnitude in response to task difficulty, quantifying how sharp certain saccades were, and the extent to which saccades went in a certain direction.
Results
Figure 4. Left: graph of intra-trial changes in pupil-size against average time-on-task; Right: graph of intra-triangles against average time-on-task. (Krejtz et al., 2018)
Pupil-Size
For pupil-size metrics, the authors found that intra-trial changes in pupil-size remained significantly more dilated during the Difficult tasks than in Easy task, with the greatest difference between the Difficult and Easy tasks—this difference was statistically significant. However, they found that the difference between the Hard and Control tasks was not statistically significant, as well as between the Easy and Control tasks.
Moreover, the authors found that inter-trial changes in pupil-size were significantly greater in Difficult tasks compared to Easy tasks, which was verified statistically, and that the dilation, specifically, was greater in Difficult task than in the Easy task. They also found a statistically significant main effect of task on inter-trial changes, supporting the notion that pupil-sizes evidently were more elevated in tasks that had higher cognitive loads, and required more mental effort.
Microsaccades
The authors, through statistical analysis, found that there was a statistically significant increase in microsaccadic magnitude as compared to Easy and Control tasks, and they found no statistically significant interaction between peak velocity and magnitude on task difficulty, contrary to their original hypotheses.
Pupil-Size vs Microsaccades
When comparing pupillary and microsaccadic effect sizes, analyses of covariance showed that the magnitude of microsaccades, intra-trial change in pupil size, and inter-trial change in pupil-size significantly differed among the Difficult, Easy, and Control tasks. They also found that the task effect size for microsaccade magnitude was highest (.17), followed by BCPD (.16) and CPD (.07). The task effect size for CPD was noticeably lower than that of BCPD and microsaccadic magnitude.
Discussion
The methodology of this study is promising—and fascinating, as the first to compare pupillary responses with microsaccades as a potential index of cognitive load—particularly in the statistical analysis of pupil-size metrics, and computational techniques that the authors employ—however, there are clear flaws in the data collection, sample size, and discrepancies in the results that should be called into question.
First, the authors use a an EyeLink SR Research camera with 500 hz to measure microsaccades is likely enough to capture these micromovements, but practically, when measuring movements that already move quite rapidly on a much smaller scale, then optimizing for framerate to ensure that microsaccades can be captured as accurately as possible is pertinent. This may potentially speak to why the peak velocity and magnitude did not show a significant main effect of task difficulty, despite microsaccadic magnitude yielding statistically significant variations between task difficulties.
Moreover, the dataset of size 13 is very small and seems unideal for performing a statistical based project and analysis off of—the computational approach and metrics generated are promising methods of analyzing pupillometric variation within and between trials and different task difficulties; in fact, the pupil-size metrics and results obtained from the study seem more promising and reliable, despite the authors arguing in favor of microsaccadic magnitude as a more accurate marker of cognitive load. For recording pupil-size, high temporal resolution is not required as it is for capturing microsaccadic movements—hence, the pupil-size collection is sound and likely not subject to error as compared to microsaccadic movements. The results obtained from the pupil-size coincide with much of the literature and notions that pupil-size covaries with cognitive load. However, it also seems plausible to visually and statistically compare the actual pupil-size time-series obtained when completing the tasks, and compute a type of distance or dissimilarity metric between the standardized time series across different task difficulties to compare its impacts on cognitive load.
Moreover, capturing micro- and saccadic movements seem better suited for visual based tasks, such as visual searching tasks, whereas the task the authors employed in this study is based off of mental computations and entering in numerical information, as opposed to visually interacting with the computer to capture saccadic patterns—there does not seem to be a clear probe stimuli that would trigger a microsaccadic movement. Microsaccadic measurements seem better suited for visual-based tasks, as opposed to mental calculation based tasks, as conducted in this study.
Despite these potential shortcomings, the computational and statistical approach does seem promising for future studies, and the pupil-size metrics coincide with the general literature. In terms of whether microsaccades can be a more accurate marker of mental effort than pupil-size, more research likely has to be done with larger samples to potentially infer generality to a population, and eye-trackers with maximum temporal resolution to collect the highest quality data.
Pupillometry and fMRI Data Collection in Working Memory Tasks
In “Pupillometry Tracks Cognitive Load and Salience Network Activity in a Working Memory Functional Magnetic Resonance Imaging Task,” Fietz et al. describe a paradigm for assessing pupil-size metrics in conjunction with an fMRI-based approach for determining what parts of the brain may be associated with both pupil-size metrics or cognitive load. The study aims to answer what areas of the brain are associated with varying pupil-size changes that occur as a function of cognitive load in a working memory task, as well as how this varies with more dynamic pupil-size fluctuations across different cognitive load conditions. Importantly, the authors combine a psychophysical method with a neurological based method for measuring brain activity using fMRI—they use fMRI primarily to scan activity in an area of the brain that controls the “salience network,” which includes the dorsal anterior cingulate cortex (dACC) and bilateral insula, allowing for the detection of salient stimuli and distribution of neural resources in response. The authors aimed to test if pupil-size fluctuations reflected activity present in the salience network, potentially connecting correlations with cognitive load with its neural correlates.
Methods
The authors recorded the pupil-size of subjects placed inside an MRI scanner as they completed an N-back working memory task, while simultaneously measuring BOLD responses through fMRI. The study consisted of 52 healthy participants. The N-back task administered was a sequence of capital letters (B, C, D, G, P, T, W)—the authors administered eight blocks of trials with 16 stimuli each, and with different loads: 0-back, 1-back, 2-back, and fixation. In 0-back tasks, subjects were required to respond by pressing a button whenever a stimulus appeared on the screen. In the fixation condition, the letter X was shown repeatedly on the screen, and required no action from the subject.
Pupillometric data was collected using an EyeLink 1000 Plus with 250 hz within the MRI scanner, while a 3 Tesla MRI Scanner was used to acquire fMRI data. 40 slices corresponding to different orientations of the brain were collected using the scanner, resulting in 176 total image volumes.
Results
To calculate pupil-size changes across each trial, the authors calculated the first-order derivative across the time-series—the authors additionally calculated the local maxima obtained within specific windows after stimulus presentation. They found that there was strong evidence to suggest an effect of condition on pupil-size using a Bayesian one-way rmANOVA test, which was statistically significant—particularly, they found strong statistical evidence differentiating pupil-size between 0-back and 1-back, 1-back and 2-back, as well as the strongest evidence for 2-back and fixation.
For measuring pupil-size correlations with BOLD activity, the authors employed different statistical and regression techniques, correlating blood-oxygen levels with changes in pupil-size. Overall, they found that there was positive correlation between BOLD activity in the fronto-parietal network, consisting of the dorsolateral prefrontal cortex, as well as in the bilateral insula. They found negative correlations between pupil-size and activation in the ACC, posterior cingulate gyrus, and lateral parietal cortex.
Discussion
Figure 5. Difference in pupil-size across time-series for different cognitive load conditions. Noticeably, pupil-size is elevated as task difficulty increases.
As evident in Figure 4 and the results obtained, the authors found that pupil-size correlates with working memory load, while further pinpointing potential neural correlates in the brain that activate concurrently with higher cognitive loads. Their pupillometric results coincide with the general phenomena and theories regarding pupil-size relations with cognitive load—visually, as can be observed in Figure 4, pupil-sizes were elevated at higher loads, which was found to be statistically significant in their variation.
In pinpointing neural correlates of these pupil-size dilations, the authors verified through fMRI and regression based techniques on BOLD activity data specific areas in the brain associated with pupil-size changes and cognitive load increases. Particularly, they found that the bilateral fronto-parietal network and areas of the prefrontal cortex, correlated strongly with pupil-size differences between load conditions. The additional significance of this study lies in pinpointing a neural basis, particularly in the salience network, for pupil-size change in the context of cognitive load—their results link activation of the salience network with different areas of the brain to modulate cognitive responses with pupil-size changes. Particularly, their results show that the dACC plays a strong role in this network, especially in projecting information top-down to the LC, which elicits a response in pupil-size.
Their work, overall, pinpoints different neurological subprocesses that may be involved in modulating pupil-size responses. Given the ACCs’ previously proposed role in taking in information about mental exertion and assessing rewards of modulating mental resources, outlined by van der Wel & Steenbergen, there seems to be additional evidence to suggest that, through top-down communication with the LC, the ACC uses a type of sensorimotor loop to modulate allocation of mental resources, and potentially motor control, in response to the perception of visual stimuli.
Clinical Relevance of Pupil-size Responses and Cognitive Load
Within the context of cognitive load, assessing working memory and working memory capacity is a logical and inherent mechanism for extending to clinical populations, as both concepts are intrinsically related to cognitive load—particularly, observing how working memory is impacted by different cognitive loads, as reflected by pupil-size variation and attentional effort, forms a promising foundation for the diagnosis and treatment of different neurobehavioral and learning disorders, such as ADHD, dyslexia, and traumatic brain injuries. Particularly, many studies monitoring pupil-size and variables such as cognitive load and working memory have been done on neurobehavioral disorders such as ADHD, which is characterized by inattention, hyperactivity, and impulsivity—these are particularly promising for augmenting diagnostic and treatment regiments. Wainstein et al. conducted a study where they varied cognitive load in a visuospatial working memory task to assess pupil-size dynamics in control and ADHD positive groups. They found a statistically significant variation in pupil-size across different ADHD groups, and linked this with performance on cognitive tasks and overall cognitive load—proposing a paradigm for diagnosis that can be extended to other learning disorders. The basis of this paradigm hinges on intelligent computational techniques for statistical and time-series based analysis of pupil-size data over time, which can both help uncover insights into relationships with cognitive load in different clinical populations, and help develop predictive models for diagnosing disorders.
The utility of pupil-size and cognitive load studies lies in helping better quantify the objective difficulty of and mental exertion induced by various tasks, but also to observe how this mental exertion and attentional processing, reflected in task-evoked pupillary responses, manifest and vary in clinical populations as a function of varying cognitive loads—computational and machine learning based predictive algorithms, in particular, can help uncover additional patterns and methods for using pupil-size to diagnose disorders like ADHD. Many studies have incorporated machine learning based predictive models for analyzing pupil-size in working memory experiments. My own research on ADHD focused on developing predictive models for linking pupil-size dynamics with attentional processing in ADHD, using a time-series based analyses of pupil-size, and accounting for variation before and after the presentation of probe stimuli—along with complex mathematical analyses of the time-series, and different transformations to uncover hidden patterns (Das & Khanna, 2021).
Moreover, evaluating cognitive load and its impacts on pupil-size in tandem can help gauge how task difficulty impacts cognitive function and activity, using pupil-size as a proxy for assessing noradrenergic activity and task difficulty, and ultimately devise better methods for modulating cognitive tests and adjusting stimuli to help reduce cognitive load and promote better performance on tasks in different clinical populations—such as in different learning disorders and in an educational context for disorders like dyslexia.
The potential use of assessing pupil-size changes in response to changing cognitive load conditions lies in helping better diagnose different disorders that impact memory, attentional processing, and overall cognitive functioning. Moreover, this paradigm extends to treating different disorders like ADHD, as well, where performance on different tasks with varying cognitive loads can reflect if medication is improving task performance and attention—this can help doctors modulate stimulant medication and facilitate more effective treatment and monitoring. As such, the meshing of advanced computational techniques with the analysis of complex and noisy time-series pupil-size data, measured alongside metrics like cognitive load and working memory, can be used to engineer robust and powerful predictive models for diagnosing disorders like ADHD using pupil-size and psychophysical tasks, and also monitoring response to treatment over time.
Impact on Art and Technology
The impacts of understanding cognitive load through monitoring of pupil-size lies not only in theoretical understandings of the brain and in health, but also across different sectors of society, where learning is at the heart of creating interactive and effective user interfaces. For example, creating more effective educational systems and curriculum is a potential application, where methods of assessing relative task difficulty can be determined through monitoring of pupil-size, and be modified to help users respond more effectively to different educational material. These metrics can be used to determine, for example, if subjects are performing inadequately on a particular task, or struggling to comprehend specific learning material—the potential impacts, especially in virtual environments could help facilitate greater task engagement and pinpoint areas that mentally overload an individual, allowing for the proper modification of material to address those areas of overload. Apart from administering tasks, pupil-size metrics can be collected as patients are immersed in a learning environment and absorbing information, where the potential impacts of the educational material on cognitive load can be determined to gauge if a particular concept is confusing or mentally overloading an individual—in this paradigm, we would not ask for an elicit cognitive response as we would in a task, but continuously monitor pupil-size as a patient is learning material to gauge engagement, mental exertion, and overall mental effort, extending the paradigm of assessing task difficulty to that of assessing the difficulty of different learning concepts.
If there were no correlation between pupil-size and cognitive load, as well as overall mental effort, then task difficulty largely would be measured solely using performance accuracy on tests, which by in itself is not necessarily a solid measure of mental effort and exertion, and ultimately overall task difficulty—if we are, however, able to gather pupil-size data as a patient learns and gauge which concepts are particularly challenging as reflected in pupil-size activations over-time, we can not only determine the overall difficulty of a task, but pinpoint at which specific time intervals and what sections of the task caused the most mental exertion and created most difficulty. This can allow for the creation of more effective curricula and address challenging concepts to help subjects learn better and improve performance.
The most promising and potentially impactful incorporation of a pupil-size based system like this to monitor cognition lies in both virtual reality, and in our smartphones, which most of us use all the time. Taking away the difficulties of controlling for luminance in varying, decentralized settings, if we are able to reliably and accurately track pupil-size from the phone, then it easily unlocks an additional biometric that can be used to help everyone monitor brain activity, increasing accessibility—in fact, the new IPhones are equipped with infrared cameras, so the ease of incorporating this technology seems promising with respect to gathering pupil-size data. The smartphone medium provides an easy and accessible method of collecting potential data that can be linked to cognitive load and different neurological processes—the problem is whether we can effectively control lighting in different environments and account for different luminances and its potential impacts on pupil-size.
As such, pupil-size in VR could be more promising for monitoring cognitive load and mental load, as light control within an immersive and closed environment is much easier, allowing for effective gathering of pupil-size data—VR-based education, particularly, is a promising application for incorporating pupillometric models of inference in learning environments, allowing teachers to determine at what times students are struggling or mentally overloaded, as well as gauging task engagement throughout class. Many headsets, such as the Oculus Quest, are equipped with eye tracking capabilities, allowing for the incorporation of software that could track pupil-size and assess cognitive load. Accessibility of these VR headsets remains a potential problem right now, but if we are able to create a low-cost VR solution equipped with eye tracking, there are numerous applications in research and education, and for helping us unlock more information about brain health and mental overload.
Potential Adaptive Role of Pupil Size Dilations
Significantly, pupil-size changes pose fundamental and unanswered questions regarding its overall adaptive role and function in our daily lives—for example, pupil-size has been generally shown to dilate during emotional arousal, such as when a subject is presented with images varying in emotional valence, or when a subject is sexually attracted to a particular stimuli; the adaptive function of pupil-size dilations in the context of these emotional arousal studies is misunderstood, and potentially blurs the conclusions put forth by working memory based studies. Fundamentally, pupil-size changes are subject to the pupillary light-reflex and autonomic responses to changes in lighting—light entering the eye is a significant factor that affects pupil size. When the amount of light entering the eye increases, the pupil constricts to protect the retina from overexposure. This process, known as the pupillary light reflex, is automatic and occurs without conscious effort. The pupillary light reflex helps to optimize the amount of light entering the eye for visual tasks, which is important for proper vision. The interplay of this reflex, or inherent function, with modulating pupil-size in cognitive load tasks is still largely misunderstood.
In fundamentally modulating the amount of light that enters the retina, the utility of pupil-size dilations—particularly in low-light conditions—potentially lies in adaptively increasing alertness and visual sensitivity, as well as the visual saliency of stimuli, mainly used as a mechanism to increase attention and overall alertness in cognitive tasks. If pupil-sizes dilate in response to increases in cognitive load, it would follow that this response is an adaptive mechanism towards alleviating mental overload and exertion when completing challenging tasks, and focusing attention on different stimuli in pursuit of completing a task. The adaptive function would lie in improving visual perception, attention, and overall alertness—studies show that when more light enters the eye, subjects are more alert and aroused. The adaptive function of pupil-size dilations are largely misunderstood, but in general, it would follow that pupil-size dilations could potentially improve visual acuity and contrast sensitivity by allowing more light to enter the eye—Mathot seems to support this view, as well, in his review of pupillometry. When the pupil dilates, it allows more light to enter the eye, which can improve visual acuity and contrast sensitivity in low light environments—by increasing the amount of light entering the eye, pupil-size dilations can potentially help visually perceive finer details and increase visual sensitivity, as well, which allows for the completion of tasks and more focused attention on different stimuli in pursuit of that task.
In low light environments, dilated pupils may improve visual acuity and contrast sensitivity by allowing more light to enter the eye, allowing for the visual perception of finer details. However, while dilated pupils may improve visual acuity in low light environments, they may also decrease these abilities in bright light environments—dilated pupils allow more light to enter the eye, which can lead to overexposure and reduced contrast sensitivity (Mathot, 2018). Dilated pupils can also potentially cause blur and decrease visual acuity—when the pupil dilates, it allows more light to enter the eye, which can increase the amount of defocus (blur) in the image formed on the retina. Simultaneously, however, dilations can also improve visual acuity and visual sensitivity in low light environments, so there may exist a trade-off between increased blur and improved visual acuity depending on lighting conditions.
Conclusion
A a collection of research studies well-documented in literature highlights pupillometric variation within the context of different attentional and memory-based tasks when lighting is controlled for—for instance, some studies highlight how pupils dilate and contract at specific intervals when subjects complete a working memory task, and that these dynamics expose statistically significant variations amongst different experimental conditions. As of now, many studies have mainly exposed the correlations between pupil-size dynamics and different cognitive variables—most potently, working memory capacity, load, and the quality of memory representations that a subject retains when completing different working memory tasks. However, there has not been a definitive answer exposed within the context of the pupil’s role in attentional, or overall cognitive processing.
Two perspectives on the phenomenon correlating cognitive load and pupil-size dilations have been documented in literature, where some authors believe that pupil dilation reflects the demands or load by the task, while others have posited that pupil dilation actually reflects the effort exerted in response to these demands (van der Wel, 2018). While the latter viewpoint seems more plausible, these perspectives are not necessarily mutually exclusive, and a plausible synthesis of both would suggest that pupil-size reflects both through a sequential , starting from increased task demands, which exacerbate cognitive load and induces mental effort and exertion—the unanswered question is then why exactly does the pupil dilate in response to greater mental demands in the context of completing a cognitive task. Perhaps there is a reciprocal relationship between both associations, where one modulates the other and vice-versa—where increased task demands induce cognitive load, which requires mental effort and exertion, reflected in pupil-size. Pupil-size dilations in this situation would have to have some type of regulatory or parasympathetic role in calming this mental exertion, yet it originates from sympathetic activations. There is then the case that pupil-size dilations are not only a reflection of mental effort, but can simultaneously exacerbate mental exertion once activated as a product of sympathetic activation, which could in turn potentially heighten cognitive load, as more mental resources are devoted to completing a cognitive task.
It seems more plausible, however, that pupil-sizes instead are used as an adaptive mechanism for improving attention and are a reflection of the LC-NE system’s attempt to provide mental resources and stimulate attention and alertness—in this context, pupil-size dilations serve to potentially improve visual perception and attention through light modulations. This is a hypothesis, however—overall, additional research is needed to pinpoint this exact interplay, and further determine what the adaptive role of pupil-size dilations may be from a neurological perspective.
Bibliography
Attar N, Schneps MH, Pomplun M. Working memory load predicts visual search efficiency: Evidence from a novel pupillary response paradigm. Mem Cognit. 2016 Oct;44(7):1038-49. doi: 10.3758/s13421-016-0617-8. PMID: 27119363; PMCID: PMC5031546.
Binda, P., & Murray, S. O. (2015, February 1). Spatial attention increases the pupillary response to light changes. Journal of Vision. Retrieved October 17, 2022, from https://jov.arvojournals.org/article.aspx?articleid=2213250
Brink, R. L. van den, Murphy, P. R., & Nieuwenhuis, S. (n.d.). Pupil diameter tracks lapses of attention. PLOS ONE. Retrieved October 17, 2022, from https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0165274
Chen HC, Kao CH, Wang TH, Lai YT. Evaluation of children’s cognitive load in processing and storage of their spatial working memory. Front Psychol. 2022 Sep 8;13:918048. doi: 10.3389/fpsyg.2022.918048. PMID: 36160601; PMCID: PMC9493119.
Das, W., Khanna, S. A Robust Machine Learning Based Framework for the Automated Detection of ADHD Using Pupillometric Biomarkers and Time Series Analysis. Sci Rep 11, 16370 (2021). https://doi.org/10.1038/s41598-021-95673-5.
Eckstein, M. K., Guerra-Carrillo, B., Miller Singley, A. T., & Bunge, S. A. (2017, June). Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? Developmental cognitive neuroscience. Retrieved October 17, 2022, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6987826/
Joshi, S., & Gold, J. I. (2020, June). Pupil size as a window on neural substrates of cognition. Trends in cognitive sciences. Retrieved October 17, 2022, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7271902/#R39
Kang OE, Huffer KE, Wheatley TP (2014) Pupil Dilation Dynamics Track Attention to High-Level Information. PLOS ONE 9(8): e102463. https://doi.org/10.1371/journal.pone.0102463
Krejtz K, Duchowski AT, Niedzielska A, Biele C, Krejtz I (2018) Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze. PLOS ONE 13(9): e0203629. https://doi.org/10.1371/journal.pone.0203629
Kucewicz, M.T., Dolezal, J., Kremen, V. et al. Pupil size reflects successful encoding and recall of memory in humans. Sci Rep 8, 4949 (2018). https://doi.org/10.1038/s41598-018-23197-6
Mathot, S. (2018). Pupillometry: Psychology, Physiology, and Function. Journal of Cognition, 1(1), 16. DOI: http://doi.org/10.5334/joc.18
Murphy, P., Robertson, I., Balsters, J., & O’Connell, R. (1970, January 1). [PDF] pupillometry and P3 index the locus coeruleus-noradrenergic arousal function in humans.: Semantic scholar. undefined. Retrieved October 17, 2022, from https://www.semanticscholar.org/paper/Pupillometry-and-P3-index-the-locus-arousal-in-Murphy-Robertson/3b05760fdd94c56e480a40f858f7e80bacffeefb
Nobukawa S, Shirama A, Takahashi T, Takeda T, Ohta H, Kikuchi M, Iwanami A, Kato N, Toda S. Identification of attention-deficit hyperactivity disorder based on the complexity and symmetricity of pupil diameter. Sci Rep. 2021 Apr 19;11(1):8439. doi: 10.1038/s41598-021-88191-x. PMID: 33875772; PMCID: PMC8055872.
Rajkowski, J., Kubiak, P., Aston-Jones, G., 1993. Correlations between locus coeruleus (LC) neural activity, pupil diameter and behavior in monkey support a role of LC in attention. Society for Neuroscience Abstract. Presented at the Society for Neuroscience Conference (p. 974). Starc, M., Anticevic, A. and Repovš, G. (2017), Fine-grained versus categorical: Pupil size differentiates between strategies for spatial working memory performance. Psychophysiol, 54: 724-735. https://doi.org/10.1111/psyp.12828
Tapper A, Gonzalez D, Nouredanesh M, Niechwiej-Szwedo E. Pupillometry provides a psychophysiological index of arousal level and cognitive effort during the performance of a visual-auditory dual-task in individuals with a history of concussion. Vision Res. (2021) 184:43–51. doi: 10.1016/j.visres.2021.03.011
van der Wel, P., van Steenbergen, H. Pupil dilation as an index of effort in cognitive control tasks: A review. Psychon Bull Rev 25, 2005–2015 (2018). https://doi.org/10.3758/s13423-018-1432-y
Wainstein, G., Rojas-Líbano, D., Crossley, N. A., Carrasco, X., Aboitiz, F., & Ossandón, T. (2017, August 15). Pupil size tracks attentional performance in attention-deficit/hyperactivity disorder. Nature News. Retrieved October 17, 2022, from https://www.nature.com/articles/s41598-017-08246-w
Zokaei, Nahid, Alexander G. Board, Sanjay G. Manohar, and Anna C. Nobre. “Modulation of the pupillary response by the content of visual working memory.” Proceedings of the National Academy of Sciences 116, no. 45 (2019): 22802-22810.