Catch a Brain Wave and Teach it to Communicate

Science Highlights
July 22, 2009

Three years ago, Scott Mackler’s journey with Lou Gehrig’s disease (amyotrophic lateral sclerosis, or ALS) could have ended much differently. He was locked in a body that no longer responded. A ventilator controlled his breathing. He communicated using an eyetracker device, and as the signals between his nerves and muscles stopped, the tool became unreliable. Mackler needed a new approach to maintain his quality of life and remain connected to his family, friends, and colleagues.

The right panel shows a person severely disabled by amyotrophic lateral sclerosis (ALS) using the Wadsworth brain-computer interface (BCI) system in his home. He wears an 8-channel electrode cap to record the brainwave (EEG) signals that control the BCI. The left panel shows a close-up of his computer screen. A flashing matrix of possible selections is located on the right of the screen, a text editor is located at the top left, and a predictive speller program is located at the bottom left. With this system, he can use any Windows-based program that can be operated with a keyboard. (Photo provided courtesy of The Wadsworth Center, New York State Department of Health.)

Eager to continue his work as a neuroscientist at the University of Pennsylvania, Mackler was put in touch with researchers at the Wadsworth Center, New York State Department of Health, Albany. With 20 years of experience developing brain-computer interface (BCI) systems to help disabled people communicate, the Wadsworth team thought they had a device to help Mackler.

Led by Dr. Jonathan R. Wolpaw, chief of the Laboratory of Neural Injury and Repair at the Wadsworth Center, New York State Department of Health, and a pioneer in BCI research, the Wadsworth team provided Mackler with a computer that could decipher his brain signals and allow him to select icons on a computer screen. He wore a specially designed cap lined with electrodes that picked up his brain waves much like a conventional electroencephalogram (EEG). As Mackler concentrated on a letter on the computer monitor, his brain generated signals that the computer analyzed and translated into useful device commands. After a few training sessions, Mackler was able to write e-mails, request food, and make known his needs and desires.

The fit between Mackler and the BCI system has been so successful that he has continued his research on the biochemistry of drug addiction and recently had his research grant renewed by the National Institute on Drug Abuse.

From Brain Waves to Actions

The system Mackler uses is based on BCI2000 software, a general-purpose software system for BCI research developed primarily by Gerwin Schalk, research scientist, Wadsworth Center and co-principal investigator on the BCI project. The software is currently used by more than 350 laboratories worldwide.

Feedback from Mackler and three other users has helped the Wadsworth researchers improve the system during the last three years. “Scott and people like him have given us a once-in-a-lifetime opportunity to get precious data,” says Theresa Vaughan, clinical director for the Center for Translational Neurological Research, a joint program of the Wadsworth Center and Helen Hayes Hospital. “The questions we’re now asking are a result of the information he’s given us.”

Because the BCI system relies on brain waves rather than muscles to produce actions, the system developers are entering a new area of understanding how the brain operates. “It’s like turning the nervous system on its head and asking the brain to do the job that muscles normally do,” says Wolpaw. “The brain can do it, but as yet only imperfectly.”

One of the principal challenges is to develop software that is flexible enough to adapt to each individual’s brain signals. “We can’t look into the brain and magically figure out what an individual’s own language symbols are,” says Schalk. “What we can do is measure the signals that have some physiological relevance to communication or movement.” Once identified, the BCI user’s brain signals could become the building blocks of a new language for the user. “We have to teach the brain to produce different signals in a certain order to produce a desired output,” he says.

As a user changes from day to day or even hour to hour, the system must continue to adapt to maintain effective performance. “We need to address these changes on a person-to-person basis,” explains Wolpaw. The result is a highly interactive communication tool tailored to each user’s abilities.

From Scalp to Cortex

While the current BCI system relies on EEG waves recorded from a user’s scalp, Schalk is exploring the use of another kind of brain wave, electrocorticographic (ECoG) activity. Electrodes are surgically implanted on the surface of the patient’s cortex, the outermost layer of the brain. These signals are clearer and stronger than EEG signals, and they experience less interference from the surrounding environment. These signals also come packed with information about movement. In a recent study with epilepsy patients, Schalk and colleagues were able to decode movement parameters such as finger speed and joint angle from brain signals relating to hand movements.

This work with epilepsy patients has also provided a new pre-surgical technique that hastens surgical planning and streamlines surgery. Before performing surgical procedures to reduce seizures in epilepsy patients, surgeons first need to identify the affected areas of the brain as well as those that are important in key functions such as motor and language skills. Traditionally, surgeons have done this using electrodes implanted on the brain’s surface to stimulate different areas and noting the impact on function. The process is time-consuming and can cause seizures.

The new technique still employs implanted electrodes, but Schalk uses signal-processing software to detect and translate the brain signals associated with a particular function such as speech or movement. The result is a map identifying the key functional areas in the brain. The entire process takes only a few minutes to complete, and no stimulation is needed.

The ECoG work could lead to a new BCI approach in which a small wireless device is implanted on the brain’s surface and relays brain signals to a computer for analysis and processing. This device may allow more precise control of movement-assistance hardware such as wheelchairs and limb prosthetics.

From Lab to Living Room

Clinical studies are integral to disseminating the technology, and the Wadsworth team is involved in several. One study is part of a collaboration with the Helen Hayes Rehabilitation Hospital, West Haverstraw, NY, to determine the needs of home users of the BCI system. This trial currently involves Dr. Mackler and three other users. As more users, including stroke survivors, come into the study, the team will assess new BCI approaches and new system functions. The Wadsworth team is working with Dr. Aiko Thompson of Helen Hayes, with additional support from the New York State Spinal Cord Injury Research Board, on a study to determine BCI’s role in rehabilitation for people with spinal cord injuries. In addition, a third multisite trial is about to begin with the Veterans’ Administration. In this study, 25 ALS patients at five VA centers will use the Wadsworth EEG-based BCI technology in their homes. The study will assess whether the system can remain viable with only a modest amount of technical support and whether it improves quality-of-life.

With the collaboration of faculty from Wharton School of Business at the University of Pennsylvania, the not-for-profit Brain Communication Foundation was recently established to provide BCI technology to more people who could benefit from it. “A non-profit endeavor is a new way to think about an orphan technology,” says Wolpaw. “We hope that this will be a way to support and maintain BCI systems for people who need them most.” In a decade, Wolpaw envisions highly reliable systems that will operate off-the-shelf with only routine technical support. “The basic expectation is that we’ll get to a point where users require very little, if any, ongoing assistance from laboratory personnel.”

Putting the technological advances in perspective, Schalk likens BCI research to computer science in the 1930s and 1940s. “The technology is still relatively crude, so we have limited applications. Once the capabilities improve, the potential for applications could explode. This new way of communicating could potentially improve the capabilities of healthy people.”

This work is supported in part by the National Institute of Biomedical Imaging and Bioengineering.


Schalk G, Miller KJ, Anderson NR, Wilson JA, Smyth MD, Ojemann JG, Moran DW, Wolpaw JR, Leuthardt EC. Two-dimensional movement control using electrocorticographic signals in humans. J. Neural Eng 2008;5:75-84.

Daly JJ, Wolpaw JR. Brain-computer interfaces in neurological rehabilitation. Lancet Neurol 2008;7:1032-43.

Schalk G, Kubanek J, Miller KJ, Anderson NR, Leuthardt EC, Ojemann JG, Limbrick D, Moran D, Gerhardt LA, Wolpaw JR. Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J. Neural Eng 2007;4:264-75.

Allison BZ, Wolpaw EW, Wolpaw JR. Brain-computer interface systems: progress and prospects. Expert Rev. Med. Devices 2007;4:463-74.