Scalable Anomaly Detection in Streaming Sensor Data Using Hybrid Statistical and Learning-Based Signal Models

Abstract

On-the-fly anomaly detection in high-rate streaming sensor data has been a high priority in current cyber-physical systems, industrial IoT, and smart infrastructure. Nevertheless, traditional methods relying exclusively on statistical modelling tend to overlook nonlinear behaviours, whereas purely learning-based methods are susceptible to enormous computational costs, and poor scalability in a streaming setting. In order to overcome these shortcomings, a scalable hybrid structure of anomaly detection is proposed in this paper and involves statistical signal modeling, as well as learning-based detection mechanisms. The statistical layer carries out preprocessing and residual generation in real time by employing probabilistic estimation algorithms, which allows one to effectively characterise the behaviour of the underlying system. A learning-based model, namely, a sequence-aware auto encoder is then applied to them to learn the temporal dependencies and detect subtle anomalies using reconstruction error analysis. To achieve better detection strength, the fusion approach is used to integrate statistical and learned anomaly scores. Large-scale testing of streaming sensor datasets show that the suggested solution can perform better, with the F1-score, a decrease in the number of false alarms, and a decrease in the detection time become much higher than when used only by individual methods. These findings substantiate the appropriateness of the hybrid design to achieve the compromise between accuracy and computational efficiency, and can be applied in scalable real-time anomaly detection in dynamic sensor-based settings.