Performance assessment of NOSTRADAMUS other machine learning-based telemetry monitoring systems on a spacecraft anomalies database

摘要

Spacecraft health monitoring on ground is commonly performed using two complementary methods: a short-term automatic Out-Of-Limits (OOL) verification after each new telemetry reception, and a long-term monitoring using statistical features (e.g. daily minimum, mean and maximum). In the past few years, various new monitoring methods based on machine learning have been suggested in literature, with a great interest for their new detection capabilities, already demonstrated for some use-cases or even for operational use. These methods differ not only by their mathematical core, but also by their data preprocessing and by the tuning of their control parameters, raising some issues for quantitative performance comparison. Indeed, benchmarks of machine-learning algorithms with classical datasets can be found in the literature, but it is no longer the case once they are embedded in a complete spacecraft monitoring system. This paper presents a methodology for such a comparison on an anomalies database constituted with housekeeping telemetry of CNES' operated satellites. The performance of NOSTRADAMUS system used at CNES, which principle and operational use are also described in this paper, is compared to the one of algorithms inspired from literature such as Novelty Detection (ESOC), Project Sybil, and ATHMoS (DLR).