Description
Time series anomaly detection: STL decomposition, dynamic MAD thresholds, multi-seasonality, alerting rules.
Most anomaly detection runs a static threshold ("alert if error rate >5%"). This fails on seasonal data: 5% errors at 3 AM is different from 5% at 3 PM during peak traffic. Seasonal decomposition separates the pattern from the anomaly. STEP 1 — SEASONAL DECOMPOSITION (STL) Break the time series into three components: Trend (long-term direction), Seasonal (predictable periodicity — daily, weekly, yearly), and Residual (everything else — the anomalies). The residual is what you want to alert on. ```python def stl_decompose(series, period=24): # period = seasonal window length (24 for hourly data with daily seasonality) trend = rolling_mean(series, window=period*2) detrended = series - trend seasonal = periodic_average(detrended, period) residual = detrended - seasonal return trend, seasonal, residual ``` STEP 2 — DYNAMIC THRESHOLDS Instead of static thresholds, compute the threshold from the residual's distribution over a rolling window: - Upper bound: median(residual) + 3 × MAD(residual) where MAD = median absolute deviation - Lower bound: median(residual) - 3 × MAD(residual) MAD is more robust than standard deviation (less sensitive to extreme values — the very anomalies we want to detect). STEP 3 — MULTI-SEASONALITY Many metrics have multiple seasonal patterns: daily (24h), weekly (168h), and yearly (8760h). Decompose with multiple periods in sequence. Example for web traffic: remove daily seasonality first, then weekly, then check the residual. What looks anomalous at the daily level may be normal weekly behavior. STEP 4 — ANOMALY SCORING Each residual point gets a score: |residual - median| / MAD. Score >3 = anomaly. >5 = critical. >8 = incident. This is the z-score equivalent but using robust statistics (MAD instead of std). STEP 5 — ALERTING RULES - Single point >3σ: WARN (monitor, no page). - 3 consecutive points >3σ: CRITICAL (page). - Single point >5σ: CRITICAL (page immediately). - Sustained shift >2σ for 2 hours: WARN (possible trend change, not anomaly). STEP 6 — HOLIDAY/EVENT EXCEPTIONS Known events (Black Friday, product launches) should be excluded from the baseline. Maintain a calendar of expected anomalies. These are not anomalies — they are expected deviations. OUTPUT: STL decomposition plots (trend, seasonal, residual), dynamic threshold bands, anomaly score table, alerting rules matrix, holiday exception calendar.
No comments yet. Be the first!