A mobile AR computer-aided diagnosis: 6 DoF brain tumour pose estimator using a fine-tuned efficientpose-based model

Augmented reality (AR) technology is rapidly advancing, enabling interactive 3D virtual environments and fostering human interaction and participation. AR has made significant strides in medicine, particularly in medical imaging, leading to improved diagnostic capabilities. To contribute to this transformative landscape, we have developed an accessible and cost-effective AR application that can be used with affordable smartphones. Our application offers automatic brain tumour segmentation, 6 degrees of freedom (6 DoF) phantom head pose estimation, AR visualisation, and interaction, eliminating the need for expensive head-mounted devices (HMD). This paper introduces a novel approach that utilises transfer learning and fine-tuning techniques to train a deep learning model capable of accurately estimating the 6 DoF object pose using RGB images alone. We also present a brain tumour segmentation method based on a specific technique of Gradient Vector Flow (GVF)-based active contour models and demonstrate 3D reconstruction and printing of a phantom head. Furthermore, we propose a comprehensive mobile AR platform, ARBrain, specifically designed for localisation and visualisation of brain tumours. This platform incorporates the deployment of the 6 DoF pose estimator into the AR mobile platform, enabling 3D AR visualisation of the brain and tumour, as well as AR interaction through voice and hand-tactile gesture commands. To evaluate the effectiveness of our approach, we employ a layering procedure and compare it with other recent virtual overlay techniques. Through automatic tumour segmentation, precise pose estimation, and advanced AR interaction capabilities, our ARBrain platform offers an accessible and cost-effective solution for brain tumour localisation and visualisation, empowering medical professionals to make accurate diagnoses and treatment plans.