TLS-aware anomaly detection for encrypted IoT traffic using a β-variational autoencoder with ANOVA–Mutual Information feature selection

The rapid growth of the Internet of Things (IoT) has increased dependency on Transport Layer Security (TLS) for securing device communications, enhancing confidentiality while reducing the visibility required by traditional intrusion detection systems. As payload inspection becomes impractical in encrypted environments, anomaly detection must instead rely on flow-level statistics and TLS metadata. This is challenging because IoT traffic is heterogeneous, non-stationary, and distributionally inconsistent across datasets, while many existing studies rely on single-dataset evaluation and therefore provide limited evidence of real-world generalisation. We introduce here a TLS-aware anomaly detector based on β-Variational Autoencoder (β-VAE) coupled with a hybrid ANOVA–Mutual Information (ANOVA–MI) feature-selection pipeline. The framework models benign encrypted IoT traffic using probabilistic latent representations and identifies anomalies through reconstruction-error-based scoring. Network flows from the BoT-IoT, IoT-23, and ToN-IoT datasets were processed using Zeek and CICFlowMeter to construct a unified metadata feature space incorporating flow statistics and TLS attributes such as JA3 and JA3S fingerprints. The model was trained on benign BoT-IoT traffic and evaluated in both in-dataset and cross-dataset scenarios. We present experimental evidence of both competitive in-dataset performance of it under domain shift (ROC-AUC ≈0.997; F1 ≈0.928) and the performance for it with respect to the domain shift (IoT-23: ROC-AUC ≈0.968, F1 ≈0.924; ToN-IoT: ROC-AUC ≈0.917, F1 ≈0.779). A comparative evaluation against deterministic autoencoders and classical baselines further indicates that the proposed β-VAE achieves stronger cross-dataset anomaly-ranking performance than the compared methods. These findings support the suitability of probabilistic latent modelling for privacy-preserving anomaly detection in encrypted IoT environments.