Knowing when something has gone terribly wrong is incredibly crucial in a variety of applications. Whether it is monitoring power draw for machinery, financial transactions or server metrics, when things go weird — that means trouble.
An anomaly is an unexpected deviation from normal behaviour, and thus generally indicates some kind of problem.
Anomaly detection most commonly concerns time series data, because the incorporation of the temporal element allows for the establishment of some kind of “normal” behaviour. Therefore when things deviate from their “normal” behaviour, they are then by definition: anomalous.
In a follow up post, we will be utilizing the Oddity anomaly detection engine to help us detect anomalies in our time series data.
In most cases, and especially in practical real world applications, anomaly detection is not a simple problem with a simple solution.
Time series data is tricky to model, and in order to identify what is extraordinary, the “normal” behaviour of the series needs to be accurately captured and established. Essentially, some typical behaviour needs to be identified, and this generally begins by assessing both the trend and seasonality of a given time series.
Trend (shown in green): The trend of the time series is (as it’s name dictates) the general trend of the data.
Seasonality (shown in blue): The seasonality of the time series is any recurring pattern or behaviour that has a fixed periodicity. This is specially important for spotting behavioral anomalies.
The trend and seasonality of a time series are known as its components, and the process of breaking down a time series into its constituent components is known as decomposition. Both trend and seasonality are used to model a time series’ general behaviour, and thus knowing that, it becomes easier to distinguish something that doesn’t follow suit.
The example above is very crisp and clear cut. Reality is unfortunately not that simple, and thus the real challenge is in practical real world applications, where a given time series can have high levels of unpredictability and complexity. A time series dataset may be comprised of multiple changing seasonalities, patterns, nuances and layers.
Change Points: Change points are (as their name suggests) points where the time series changes. This can be anything from a shift in it’s level, to even a completely different seasonal pattern. A change point might be flagged as an anomaly at first, but may soon actually become the time series’ new “normal” behaviour.
Above is an example of a time series with a few change points. The first change (the increased shift in level) at the moment it occurs, can be considered as anomalous. However, we see that the time series then remains unchanged at that level, therefore becoming the new “normal” behaviour. The time series then shifts yet again (the decreased shift in level). Similarly to the previous shift, the exact moment of the change point itself can be considered anomalous. The time series however, then continues at this new level, consequently therefore changing the “normal” once again.
Local Seasonality: A time series may not have a simple constant seasonality across its entirety. Similarly to change points above, the seasonality and/or any recurring pattern in a time series (provided one exists) can also drastically change from context to context, as well as exist in multiple scopes.
Above is an example of different periodicities occuring in the same time series. At the beginning, we see a different local pattern than in the rest of the time series.
There can also be multiple seasonalities that encompass different scopes. For example, suppose a time series has a strong weekly seasonality. Within that weekly pattern might also exist a daily pattern from day to day.
Above is an example of how something like holidays can influence data points in a time series data set. Holidays for example, obviously occur every year, and a spike in page requests for an online store on black friday is not necessarily an anomaly, but rather in itself a predictable yearly pattern.
Irregular Time Series: Some time series are not so easily decomposable, and might not even have a seasonal or cyclical pattern.
Cyclicality: Cyclicality is a recurring pattern that does not follow a fixed periodicity. This is not to be confused with seasonality, which must follow a fixed period. Naturally, this lack of binding to a fixed period of time introduces a lot more difficulty in identifying cyclicality in data.
Above is an example of cyclicality in a given time series. Generally, the length of a cycle is larger than that of a seasonal pattern, such as in this case, occurring over a few years.
Anomalies are also not just necessarily extreme/outlier values in a time series, anomalies can also come in the form of perfectly normal values that simply just don’t follow an established pattern. They will be dubbed “value anomalies” and “behavioural anomalies” respectively.
Value Anomalies: Value anomalies are anomalies that are just simply extreme values in data. These are often easy to identify, and even static thresholds may suffice in detecting them.
Above can clearly be seen the anomalous points, with values far beyond anything else in the entirety of the time series.
Behavioural Anomalies: Behavioural anomalies are anomalous in a context relative to some behaviour. These are harder to spot, as their values can be perfectly normal within the scope of the data. Therefore a proper modelling and understanding of a given time series’ behaviour is required to identify such anomalies.
Above is an example of behavioural anomalies. Clearly, their values are nothing out of the ordinary, but rather it is the difference from expected behaviour that makes them anomalous.
A good anomaly detector must be capable of addressing the challenges that come with time series anomaly detection, as well as match some general key requirements:
- Accurate: An anomaly detector should no doubt be accurate enough to identify issues with a minimal level of false positives to avoid alert fatigue. The detector model should also be capable of identifying both value and behavioural anomalies correctly.
- Fast: Problems are problems, and it is important to identify and alert them quickly. Most common anomaly detection methodologies are not capable of doing true real-time online learning, and thus go with a sliding window approach. The calculations done per time window should be minimal, as to almost simulate near real-time learning.
- Smart: The anomaly detector model should be capable of learning seasonality and/or patterns in the time series well, including local and global patterns. The model should also be adaptive, and capable of adapting to change points in the data.
- Robust: The model should identify, but not easily be influenced by outliers. The model should be capable of working for most data, in most general scenarios without needing special tuning.
In the next post in the series, we will be exploring and describing some of the most common methods and algorithms used in detecting anomalies in time series data. We will see how we can use the Oddity anomaly detection engine to meet our requirements, and help discover anomalies in our own time series data.