Evaluating AES Payload Encryption for Securing MQTT-based Smart Home Networks with Machine Learning-based Intrusion Detection

Abstract

The message queuing telemetry transport (MQTT) protocol is widely adopted in smart home IoT ecosystems despite its default configuration failing to offer adequate protection against eavesdropping or payload manipulation. This study addresses an important research gap and attempts to determine whether AES-128 payload encryption is capable of securing MQTT transmissions without degrading the effectiveness of machine learning-based intrusion detection systems (IDS). Three security configurations, namely TLS, payload encryption, and token-based authentication, deployed on the ESP32 microcontroller family, are compared and their impact on message latency is measured. Experimental results show that the AES-128 encryption overhead remains at below 25% of the message publication time on ESP32-S3. To evaluate the robustness of IDS under encryption, we apply a reproducible modification to the MQTTset benchmark dataset that replaces variable-length plaintext payloads with fixed-length ciphertext representations while simultaneously preserving feature semantics and labeling consistency. 5 out of 6 evaluated classifiers maintained their accuracy level at above 99% on the modified dataset, with tree-based and neural models showing no meaningful degradation. Only Naive Bayes proved unsuitable, with its accuracy dropping from 98.79% to 62.15% due to its independence assumptions being violated by cryptographic uniformity. These results confirm that AES-based MQTT payload encryption is a practical and efficient security measure for resource-constrained IoT environments, provided that appropriate classifiers are employed.