A distance-based method to detect anomalous attributes in log files

摘要

Dealing with large volumes of logs is like the proverbial needle in the haystack problem. Finding relevant events that might be associated with an incident, or real time analysis of operational logs is extremely difficult when the underlying data volume is huge and when no explicit misuse model exists. While domain-specific knowledge and human expertise may be useful in analysing log data, automated approaches for detecting anomalies and track incidents are the only viable solutions when confronted with large volumes of data. In this paper we address the issue of automated log analysis and consider more specifically the case of ISP-provided firewall logs. We leverage approaches derived from statistical process control and information theory in order to track potential incidents and detect suspicious network activity.