The Sound Of Progress: Reconstruction Of Music Using Brain-Machine Technology
(Posted on Friday, October 27, 2023)
This article was originally published on Forbes on 10/27/23.
This story is part of a series on the current progression in Regenerative Medicine. This piece discusses advances in brain-machine interfaces.
In 1999, I defined regenerative medicine as the collection of interventions that restore to normal function tissues and organs that have been damaged by disease, injured by trauma, or worn by time. I include a full spectrum of chemical, gene, and protein-based medicines, cell-based therapies, and biomechanical interventions that achieve that goal.
New brain-machine interface technologies can reconstruct music through the analysis of brain activity. During my ongoing series on regenerative medicine, I have discussed several brain-machine interface breakthroughs that promise to invigorate regenerative therapy in the coming years. Among these breakthroughs are devices that monitor brain activity and translate the brain’s electrical signals to written words, such as Dr. Jerry Tang and colleagues’ MRI scan speech decoder.
Dr. Ludovic Bellier and colleagues from the University of California have taken this concept a step further. What if our measured brain signals could produce not written but spoken words? Bellier and colleagues reconstructed a Pink Floyd song from patient brain signals using brain-machine decoding models. While an exciting accomplishment, the usefulness of reconstructed music from subject brain signals is limited. However, the derivative potential for reconstructing spoken words from the brain signals of nonverbal patients is extraordinary. Here, I will discuss Bellier’s brain-machine interface and speak to the implications it may have in the future.
The researchers used a cohort of 29 patients to obtain their intracranial electroencephalography dataset. Each patient was implanted with 2,668 electrodes directly on the surface of their brain surrounding the auditory cortex.
As the patients listened to Another Brick in the Wall, Part 1 by Pink Floyd, the electrode set recorded active brain signals and created an auditory spectrogram of the reconstructed song.
FIGURE 1: Auditory spectrograms of the original song vs the reconstructed song.
The decoding software separated brain signals into categories based on which musical element was being conveyed. Similar to how the band develops the score of a song by layering vocals, bass, guitar, and drums into harmonies, chord progressions, and so on, the researchers had to take the incoming signals, determine how the signals related to the individual elements of the score, and translate that into an auditory file.
While I cannot present the audio directly, it can be found in the supplementary material section of the original study published in PLOS Biology. The aggregated reconstruction from the 29 participants is recognizably Another Brick in the Wall, Part 1, but may be difficult for some to pinpoint. The individual recording from participant P29, who had a greater density of electrodes than others, sounds remarkably similar to the original, to the degree that you can hear the lyrics delivered by band member Roger Waters.
How can something like Bellier’s signal decoder be used in regenerative medicine? The greatest hope lies in communication aids for those with speech impediments.
Recall the speech program used by the late Stephen Hawking. He would input the words he wished to convey on a computer interface, which a speech synthesizer would process and create spoken words. With the Bellier signal decoder, the need to input onto a computer interface would be eliminated. One could think of the words they wish to convey, and the speech decoder could process and express said words.
Further, the greater understanding of auditory perception could also have some uses, including the rehabilitation of stroke patients or even commercial songwriting.
This story received much attention in the public press when the study appeared, with some assuming that implanted electrodes could read unspoken thoughts. This is an unfounded assumption, and I implore we approach these new technologies optimistically.
The work described here continues the new era of brain-machine interfaces with the power to transform lives, offering hope and a new voice to those who have struggled with communication barriers. The future is promising, with potential applications extending beyond regenerative medicine, paving the way for innovative and impactful advances in various fields.