Center of Excellence for Learning in Education, Science and Technology A National Science Foundation Science of Learning Center

Capstone Project 3: Learning, Attention, and Multisensory Binding

Led by CELEST Co-PI member Robert Sekuler

In the natural world, sensory input more often than not is multimodal, not unimodal. For example, many events give rise to correlated visual and auditory stimulation. Although researchers have traditionally focused on just one or another sense modality, the past few years have seen an explosion of interest in the ways that inputs from multiple senses complement or compete with one another, neurally and in behavioral outcomes. Correlation between multimodal sensory signals is known to sharpen and speed perceptual responses, but little has been done to understand how such correlations, whose importance is undeniable, might impact memory, learning, and other complex behaviors, and how attention might modulate such influences.

Research in the LAMB CP is guided by this research question:

How are learning and attention influenced by correlation-dependent binding of audio and visual inputs?

The group's efforts to date have focused on the design and implementation of a novel, rich test environment that LAMB’s researchers are optimizing for studies of multisensory interactions in human attention and learning. This interactive environment is being implemented as dynamic video displays accompanied by compelling stereo sound. The test environment deliberately departs from the traditional, trial-oriented design favored by experimentalists, and embeds a test subject in an audio-visual game-like environment. This test-bed design leverages a natural desire for competition in order to engage test subjects at a high level of controlled attention and varying task difficulty.

The test bed requires subjects to make a series of categorization judgments, rapidly but accurately partitioning sequentially-presented unimodal and multimodal moving/sound emitting objects into categories such as “dangerous” or “benign.” The extent of multisensory interaction within the interactive dynamic environment is indexed by benefits in speed and/or accuracy that follow experience with correlated and uncorrelated auditory and visual signals. This flexible test bed will be used by the CP’s multiple investigators to test a variety of hypotheses against behavioral, electroencephalographic (EEG), and functional magnetic resonance imaging (fMRI) data. Categorizations made under time pressure, as imposed by the dynamic interactive environment, lend themselves to the investigation of key issues defined in LAMB’s five projects. For example, in LAMB-1, Bohland, P. Miller, and Sekuler will use EEG in the interactive dynamic multisensory environment to examine resting state cortical networks, focusing on the time required to return to resting state after recruitment of auditory, visual, or audio-visual attention. In LAMB-2, Bullock will exploit behavioral and modeling techniques to examine the dynamics of rule changes and cognitive flexibility. E. Miller, a contributor to this subproject, will explore the possibility of using a simplified version of a categorization task to study multisensory influences on learning.