BrainAR: automated brain tumor diagnosis with deep learning and 3D augmented reality visualization

Augmented Reality (AR) technology offers promising applications in healthcare by enabling interactive 3D visualization of anatomical structures. However, current AR implementations often lack patient-specific detail, limiting their effectiveness in clinical settings. In this paper, we present BrainAR, an innovative mobile AR-based application designed for the automatic segmentation, 3D visualization, localization, and interaction with brain tumors using multiparametric 3D Magnetic Resonance Imaging (MRI) data. Our method leverages a 3D Residual U-Net, trained on the BraTS2021 dataset, achieving a mean Dice score of 0.886 for accurate tumor segmentation. The segmentation outputs are integrated into a real-time 3D engine to enable precise and dynamic visualization of brain tumors. Key contributions of our work include: 1) a server-side deployment of the segmentation model for online, patient-specific inference; 2) seamless AR integration enabling interactive exploration through hand gestures and voice commands; and 3) a mobile-based platform aimed at enhancing accessibility and usability in clinical environments. The proposed solution facilitates early detection and diagnosis by providing clinicians with an intuitive, immersive, and patient-specific tool for enhanced medical imaging interaction.