Yu Takagi, Shinji Nishimoto and Osaka University colleagues have published a study which demonstrates that AI can read brain scans and re-create largely realistic versions of images a person has seen. Future applications could include enabling communication of people with paralysis, recording dreams, and understanding animal perception, among others.
Additional training was used on the existing text-to-image generative AI Stable Diffusion system, linking text descriptions about thousands of photos to brain patterns elicited when those photos were observed. Stable Diffusion was able to get more out of less training for each participant by incorporating photo captions into the algorithm.
The algorithm uses information gathered from regions of the brain involved in image perception, such as the occipital and temporal lobes. The system interpreted information from fMRI scans, detecting changes in blood flow to active brain regions. When people look at a photo, the temporal lobes register information about image contents (people, objects, or scenery), and the occipital lobe predominantly registers information about layout and perspective, such as the scale and position. This is recorded by the fMRI as it captures peaks in brain activity, and these patterns can then be reconverted into an imitation image using AI.