eduzhai > Applied Sciences > Engineering >

Interpretable multimodal fusion networks reveal mechanisms of brain cognition

  • Save

... pages left unread,continue reading

Document pages: 10 pages

Abstract: Multimodal fusion benefits disease diagnosis by providing a morecomprehensive perspective. Developing algorithms is challenging due to dataheterogeneity and the complex within- and between-modality associations.Deep-network-based data-fusion models have been developed to capture thecomplex associations and the performance in diagnosis has been improvedaccordingly. Moving beyond diagnosis prediction, evaluation of diseasemechanisms is critically important for biomedical research. Deep-network-baseddata-fusion models, however, are difficult to interpret, bringing aboutdifficulties for studying biological mechanisms. In this work, we develop aninterpretable multimodal fusion model, namely gCAM-CCL, which can performautomated diagnosis and result interpretation simultaneously. The gCAM-CCLmodel can generate interpretable activation maps, which quantify pixel-levelcontributions of the input features. This is achieved by combining intermediatefeature maps using gradient-based weights. Moreover, the estimated activationmaps are class-specific, and the captured cross-data associations areinterest label related, which further facilitates class-specific analysis andbiological mechanism analysis. We validate the gCAM-CCL model on a brainimaging-genetic study, and show gCAM-CCL s performed well for bothclassification and mechanism analysis. Mechanism analysis suggests that duringtask-fMRI scans, several object recognition related regions of interests (ROIs)are first activated and then several downstream encoding ROIs get involved.Results also suggest that the higher cognition performing group may havestronger neurotransmission signaling while the lower cognition performing groupmay have problem in brain neuron development, resulting from geneticvariations.

Please select stars to rate!

         

0 comments Sign in to leave a comment.

    Data loading, please wait...
×