The procedure is as follows:
[1] Record brain activity while the subject watches several hours of movie trailers.
[2] Build dictionaries (regression model; see below) to translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points in the brain at which brain activity was measured. (For experts: our success here in building a movie-to-brain activity encoding model that can predicts brain activity to arbitrary novel movie inputs was one of the keys of this study)
[3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions.
[4] Build a random library of ~18,000,000 seconds of video downloaded at random from YouTube (that have no overlap with the movies subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average those clips together. This is the reconstruction.
Does anyone has some insight/experience with this (ML for signal processing in general and brain imaging processing in particular)? Just be interested to hear someone riffing on this (there was also a very recent one where a neural net learned to recognize cat faces from youtube clips)
[link] [6 comments]