AI reconstructs whatever you see just by reading a brain scan

quote:

---

AI can pluck images directly from a person’s brain. Given an fMRI scan of someone looking at a picture, an algorithm can reconstruct the original picture from the scan. Though the results aren’t yet perfect, they are still often recognisable and hint at what may be possible in the future.

---

quote:

---

To generate its captions, the artificial intelligence is given an fMRI brain scan image, taken while a person is looking at a picture. It then generates a written description of what they think the person was viewing. An illustration of the level of complexity it can offer is: “A dog is sitting on the floor in front of an open door” or “a group of people standing on the beach.” Both of those turn out to be absolutely accurate.

---

quote:

---

Imagine a reality where computers can visualize what you are thinking.

Sound far out? It's now closer to becoming a reality thanks to four scientists at Kyoto University in Kyoto, Japan. In late December, Guohua Shen, Tomoyasu Horikawa, Kei Majima and Yukiyasu Kamitani released the results of their recent research on using artificial intelligence to decode thoughts on the scientific platform, BioRxiv.

Machine learning has previously been used to study brain scans (MRIs, or magnetic resonance imaging) and generate visualizations of what a person is thinking when referring to simple, binary images like black and white letters or simple geographic shapes (as shown in Figure 2 here).

But the scientists from Kyoto developed new techniques of "decoding" thoughts using deep neural networks (artificial intelligence). The new technique allows the scientists to decode more sophisticated "hierarchical" images, which have multiple layers of color and structure, like a picture of a bird or a man wearing a cowboy hat, for example.

---

quote:

---

Quantitative modeling of human brain activity based on language representations has been actively studied in systems neuroscience. However, previous studies examined word-level representation, and little is known about whether we could recover structured sentences from brain activity. This study attempts to generate natural language descriptions of semantic contents from human brain activity evoked by visual stimuli. To effectively use a small amount of available brain activity data, our proposed method employs a pre-trained image-captioning network model using a deep learning framework. To apply brain activity to the image-captioning network, we train regression models that learn the relationship between brain activity and deep-layer image features. The results demonstrate that the proposed model can decode brain activity and generate descriptions using natural language sentences. We also conducted several experiments with data from different subsets of brain regions known to process visual stimuli. The results suggest that semantic information for sentence generations is widespread across the entire cortex.

---