Evaluating the effectiveness of the Peer Data Labelling System (PDLS)

The Peer Data Labelling System (PDLS) is a novel and extensible approach to generating labelled data suitable for training supervised machine learning (ML) algorithms for use in Child Computer Interaction (CCI) research and development. For a supervised ML model to make accurate predictions it requires accurate data on which to train. Poor quality input data to systems results in poor quality outputs often referred to as garbage in, garbage out (GIGO) systems.

PDLS is an alternative system to commonly employed approaches to facial and emotion recognition such as the Facial Action Coding System (FACS) or algorithmic approaches such as AFFDEX or FACET.

This paper presents the approaches taken to evaluate the effectiveness of PDLS. Algorithmic approaches did not produce consistent classifications and major amendments to the PDLS would be required if that validation route was pursued. The human review process found that the pupil observers and reviewers reached consensus in classifying most of the data as engaged. Recognising disengagement is more challenging, and further work is required to ensure that there is more consistency in what the participants recognise as engagement and disengagement.