New AI can reconstruct what you’re looking at by reading brain activity.
Researchers presented Brain-IT, a method that reconstructs images people viewed from their fMRI brain recordings, and report it stays closer to the original images than current approaches do.
Researchers have described a new method called Brain-IT that reconstructs images a person was looking at from fMRI brain recordings. The authors report that the reconstructions are more faithful to the images people actually saw than those produced by existing methods.
Rebuilding a seen image from fMRI data offers a non-invasive window into the human brain. The paper notes that diffusion models have driven recent progress in this area. It also says that current methods often lack faithfulness to the actual seen images, which is the problem Brain-IT is designed to address.
The core of the approach is a Brain Interaction Transformer, or BIT. It allows clusters of brain voxels with similar functions to interact with one another. According to the authors, these functional clusters are shared by all subjects, so they act as building blocks for combining information both within a single brain and across different brains. Every component of the model is shared across all clusters and subjects, which the authors say allows efficient training with a limited amount of data.
To guide the reconstruction, BIT predicts two complementary kinds of localized, patch-level image features. The first are high-level semantic features, which steer the diffusion model toward the correct content of the image. The second are low-level structural features, which help start the diffusion process with the correct coarse layout. The authors say the design lets information flow directly from clusters of brain voxels to localized image features.
The researchers report that the method surpasses current state-of-the-art approaches, both in visual comparisons and on standard objective metrics. They also report that with only 1 hour of fMRI data from a new subject, Brain-IT produced results comparable to current methods trained on full 40-hour recordings. The page is a paper abstract and does not list the datasets, the number of participants or the metric scores behind these claims.
At a Glance
- The method is called Brain-IT and uses a Brain Interaction Transformer (BIT) to work with fMRI recordings.
- BIT predicts two kinds of patch-level image features, semantic and structural, to guide a diffusion model.
- With 1 hour of fMRI data from a new subject, results were comparable to methods trained on 40 hours.
- The authors say all model components are shared across clusters and subjects, allowing efficient training with limited data.
Sources
This brief was written by Brightside Report from the reporting linked above. Spot an error? Request a correction.