IoT-enabled edge architecture for real-time facial emotion recognition

Recognising emotion from facial expressions in realtime provides valuable insights, such as how an individual is feeling or how engaged they are in a specific task. Traditional approaches using RGB images present various challenges, including the identifiability of individuals and the introduction of latency when offloading processing tasks to cloud services. This paper presents a real-time emotion recognition system using thermal imaging integrated with an IoT edge architecture to optimise model latency and throughput. It also introduces a dataset containing posed thermal images from 12 participants. Our approach utilises a thermal camera interfaced with a Raspberry Pi and incorporates a CNN model. To further reduce model latency and improve processing efficiency, we propose the inclusion of an anomaly classification model, which serves as a gateway to the CNN. Testing the system with a video file containing 266 frames, the inclusion of the anomaly classifier improved model latency and throughput, enabling real-time performance in resource-constrained scenarios. Our key contributions include a novel IoT architecture for thermal emotion recognition, real-time processing capabilities on the Raspberry Pi, and a new dataset of posed thermal facial expressions. This work lays the foundation for real-time emotion recognition systems that can be deployed in resource-constrained environments, with applications in smart cities, smart campus, smart medical systems, and security.