New USC research has decoded how the brain sorts our visual memories into categories, similar to a filing cabinet. Image/Mankun Guo
Researchers at the University of Southern California have made a significant breakthrough in understanding how the human brain forms, stores and recalls visual memories. A new study, published in Advanced Science, harnesses human patient brain recordings and a powerful machine learning model to shed new light on the brain’s internal code that sorts memories of objects into categories — think of it like the brain’s filing cabinet of imagery.
The results demonstrated that the research team could essentially read subjects’ minds, by pinpointing the category of visual image being recalled, purely from the precise timing of the subject’s neural activity.
The work solves a fundamental neuroscience debate and offers exciting potential for future brain-computer interfaces, including memory prostheses to restore lost memory in patients with neurological disorders like dementia. It was led by Dong Song, associate professor in the Department of Neurological Surgery and the Alfred E. Mann Department of Biomedical Engineering. and Charles Liu, the USC Neurorestoration Center director at Keck School of Medicine of USC and professor of biomedical engineering at USC Viterbi School of Engineering. The first author, Xiwei She, is a former Ph.D. student from the Song Lab and is currently a postdoctoral researcher at Stanford University.
The hippocampus is a critical brain region, well-known for its role in creating new episodic memories – the “what, where, and when” of past events. While its function in encoding spatial (“where”) and temporal (“when”) information is relatively understood, how it manages to encode the vast and high-dimensional world of objects (“what”) has remained a mystery. It’s simply not feasible for the hippocampus to store every single object separately. Instead, scientists hypothesized that the brain might simplify this complexity by encoding objects into categories.
Associate professor Dong Song (L) and first author Xiwei She (R) discuss their machine learning model.
Song, who is Director of the USC Neural Modeling and Interface Laboratory, has been conducting pioneering work in the area of memory prostheses, creating devices that mimic and restore cognitive function, with potential clinical applications for patients with dementia and Alzheimer’s disease.
“We have tested our memory prostheses in a lot of human patients. We created the prostheses and have published several papers showing that it can enhance memory function,” Song said. “But I also wanted to take the opportunity to answer some fundamental neuroscience questions. And this is one of them.”
Song, Liu and their team’s latest work harnesses brain recording from 24 epilepsy patients with intracranial depth electrodes implanted in their brains for seizure localization. Recordings from these patients allowed the team to pinpoint how hippocampal neurons encode complex visual information, not by firing rate alone, but by the precise timing of their activity.
“Working with human patients suffering from memory dysfunction, it was exceptionally exciting to see the current studies reveal a model for the neural basis of memory formation,” Liu said.
The research team developed an innovative experimental-modeling approach to unravel this intricate process. The team recorded the electrical activity, specifically “spikes,” from hippocampal CA3 and CA1 neurons in the epilepsy patients. The recordings were gathered while patients performed a “delayed match-to-sample” (DMS) task — a popular neuroscience technique to test visual short-term memory.
“We let the patients see five categories of images: ‘animal,’ ‘plant,’ ‘building,’ ‘vehicle,’ and ‘small tools.’ Then we recorded the hippocampal signal,” Song said. “Then, based on the signal, we asked ourselves a question, using our machine learning technique. Can we decode what category image they are recalling purely based on their brain signal?”
The results confirmed the hypothesis that the human brain does indeed recall visual objects by sorting them into categories, and that these visual memory categories that the patients were thinking of were decodable based on their brain signals.
“It’s like reading your hippocampus to see what kind of memory you are trying to form,” Song said. “We found that we can actually do that. We can pretty accurately decode what kind of category of image the patient was trying to remember.”
Charles Liu, the USC Neurorestoration Center director at Keck School of Medicine of USC and professor of biomedical engineering.
The core of the discovery lies in the research team’s interpretable memory decoding model. Unlike previous methods that often rely on averaging neuronal activity over many trials or using pre-selected temporal resolutions, this advanced model analyzes the “spatio-temporal patterns” of spikes from an entire ensemble of neurons.
The study also provides evidence that the hippocampus uses a temporal code to represent visual memory categories. This means that the precise timing of individual neuron spikes, often at the millisecond scale, carries meaningful information. While previous studies often focused on individual neurons, this research revealed that hippocampal neuron ensembles encode memory categories in a distributed manner. This means that while a large proportion of neurons (70-80%) were involved in assigning a visual memory to a category, within each individual neuron, only brief, specific moments contributed to this encoding. This efficient strategy allows the brain to store diverse memories while minimizing energy consumption.
“With this knowledge, we can begin to develop clinical tools to restore memory loss and improve lives, including memory prostheses and other neurorestorative strategies,” Liu said. “While this result may be important to all patients who suffer memory disorders, it has profound relevance specifically to the epilepsy patients who participated in the studies, many of whom suffer from hippocampal dysfunction that manifests in both seizures as well as cognitive/memory disorders.”
These research team’s findings highlight the intricate ways the human brain organizes and stores information, bringing us closer to decoding the essence of human memory, and offering invaluable insights into future treatments for neurological disorders.
The work was supported by the DARPA RAM (Restoring Active Memory) program, in collaboration with institutions including Wake Forest University and Rancho Los Amigos National Rehabilitation Hospital, and the NIH BRAIN Initiative – Theories, Models and Methods program.
Published on July 21st, 2025
Last updated on July 22nd, 2025
