A Weizmann Institute of Science team has reported an AI brain scan image reconstruction system that can generate an approximation of an image a participant was viewing from functional MRI data. The work is a laboratory visual-decoding result, not a system for unrestricted access to private thoughts, dreams or whatever happens to be running through someone’s head.
According to MIT Technology Review, computer scientist Michal Irani and colleagues built the system using existing datasets in which volunteers looked at images while undergoing high-resolution fMRI scans. The team presented the finding at the Cognitive Computational Neuroscience conference in New York last month.
fMRI does not record individual neurons firing. It measures changes in oxygenated blood flow associated with brain activity, using a large magnetic scanner. Even the higher-resolution scans used in the reported work divide the brain into voxels covering roughly one cubic millimeter of tissue. That is a long way from casually pointing a consumer device at somebody and extracting a secret.
Can AI reconstruct images from brain scans?
In the reported demonstration, yes, with important constraints. The model starts with brain signals from people who were shown known images in an fMRI scanner, then produces a reconstructed image. MIT Technology Review says the results can preserve more of an image’s content and layout than earlier attempts, although the report supplies no quantitative benchmark or underlying paper for independent review.
The system uses two model branches. One estimates image structure, such as the arrangement of colors and forms. The other estimates content, such as whether the image contains food or a particular object. Those outputs guide a diffusion model, an image generator that progressively turns noise into an image.
The researchers also trained an encoder to predict the brain activity an image would produce. They could then feed an image through that encoder, pass the predicted scan through the decoder, and use the result to improve both models. Irani told MIT Technology Review that about 70% of the training data consisted of images that had not originally been paired with human fMRI scans.
How much scanning does the system need?
Irani said the system could be calibrated for a new participant with about one hour of fMRI data. The report contrasts that claim with roughly 40 hours that other image-decoding efforts have typically required for a new person. Less scanner time could make studies easier to run, but it does not remove the need for specialized imaging equipment and person-specific data.
The reconstructions also fail. MIT Technology Review described a cake that the system turned into three sandwiches, along with an incorrect reconstruction of a dog in a bathtub. Those examples are a useful corrective to the “mind-reading” label: the tool infers visual features from patterns learned in controlled experiments, and it can get them wrong.
Irani said she hopes the approach could help researchers study the brain and might eventually assist people who cannot communicate conventionally. She also mentioned reconstructing dream content as a possible future direction. None of those uses has been demonstrated by this system, according to the report.
Tommy Sprague, a University of California, Santa Barbara neuroscientist not involved in the work, told MIT Technology Review that the results appeared impressive while warning about the prospect of extracting information about a person’s thoughts without consent. That concern is real enough to discuss, but the reported experiment is not evidence that today’s fMRI decoder can secretly read arbitrary thoughts outside a lab.
This story draws on original reporting from MIT Technology Review.
