Recent technology has provided many tools and methods that allow researchers to look further into the blackbox of human minds. These include electroencephalogram (EEG), magnetoencephalography (MEG), positron emission tomography (PET), and magnetic resonance imaging (MRI), among which EEG and functional MRI (fMRI), two non-invasive techniques, dominate the field. These two major instruments have enabled researchers to identify latent neural processes and decode certain important cognitive states. Following the recent advances in neuroimaging technology that enable the concurrent recording of EEG and fMRI, much effort has been made to integrate these two modalities in order to leverage their complementary powers, or identify and characterize the underlying mechanism of neurovascular coupling. Numerous controlled experiments have been carefully designed and carried out, and large volumes of data have been recorded. Multiple statistical methods, including machine learning, have been applied to interpret these data, or to mine the corpus of information they create.
In this dissertation, we continue in this line of study, and propose methods built upon advanced statistical tools: HMM- and RNN-based methods for modeling and classifying cognitive states, and a temporal extension of canonical correlation analysis (CCA) for identifying temporal correlations between different modalities. We chose to tackle the complicated problem of decoding neural processes through two very different experimental data sets. In one study, fMRI data were collected when subjects were exposed to video/audio cues to induce craving of cigarettes; in the other study, both EEG and fMRI data were collected simultaneously when subjects were presented with visual stimuli in a spatial working memory task. Dynamic models that capture the temporal patterns in the data and exploratory methods that reveal the underlying relationship between modalities were designed and tested over the data sets. The findings and reflections of these studies are described here. Our work is one step closer to the goal of unlocking the secrets of human minds.