Exponentially weighted metrics for fleet telemetry

An exponentially weighted mean keeps one number per vehicle instead of a window of observations, which makes it the natural choice for a live fleet metric across tens of thousands of entities. Two details decide whether it is correct on mobility data: the decay has to be a function of elapsed time rather than of row count, and the early values need bias correction or every newly-seen vehicle appears to be doing nothing.

Why this happens

The standard exponentially weighted mean applies a fixed factor per observation. On regularly sampled data that is equivalent to a time-based decay; on telemetry it is not, because the interval between observations varies by orders of magnitude. A vehicle that reports every second while moving and every five minutes while parked gets its parked readings weighted as heavily as its moving ones, so the smoothed speed of a stationary vehicle decays far too slowly.

The second issue is initialisation. Starting the accumulator at zero means the first few readings are pulled toward zero, so a vehicle that has just appeared shows an artificially low mean speed for several minutes. On a dashboard that reads as an idle vehicle. Both problems have standard fixes, and both are usually missing. The window-based alternatives are covered in rolling statistics for mobility metrics.

Core pipeline

  1. Choose a half-life in seconds, which is interpretable, rather than an alpha, which is not.
  2. Decay by elapsed time between observations, so a sparse device loses weight properly.
  3. Bias-correct the early values, so a newly seen vehicle is not reported as idle.
  4. Keep one accumulator per entity, which is what makes this affordable at fleet scale.

Production-ready Python implementation

PYTHON
import math
from dataclasses import dataclass
from datetime import datetime


@dataclass
class TimeAwareEWM:
    """
    Time-aware exponentially weighted mean with bias correction.

    State is three floats per entity regardless of history length, which is
    what makes it usable for a live metric across a large fleet.

    Parameters
    ----------
    half_life_s : float
        Elapsed time after which an observation's weight halves. Prefer this
        to alpha: "the last five minutes dominate" is a statement anybody can
        check, and alpha is not.
    """
    half_life_s: float
    value: float = 0.0          # weighted sum
    weight: float = 0.0         # sum of weights, for bias correction
    last_t: datetime | None = None

    def __post_init__(self):
        if self.half_life_s <= 0:
            raise ValueError("half_life_s must be positive.")

    def update(self, t: datetime, x: float) -> float:
        """Ingest one observation and return the current corrected mean."""
        if t.tzinfo is None:
            raise ValueError("Timestamps must be timezone-aware (UTC).")
        if not math.isfinite(x):
            return self.current()          # skip, do not poison the accumulator

        if self.last_t is None:
            decay = 1.0
        else:
            dt = (t - self.last_t).total_seconds()
            if dt < 0:
                # A backwards timestamp means a clock step. Decaying by a
                # negative interval AMPLIFIES old state; reset instead.
                self.value, self.weight = 0.0, 0.0
                decay = 1.0
            else:
                decay = 0.5 ** (dt / self.half_life_s)

        self.value = self.value * decay + x
        self.weight = self.weight * decay + 1.0
        self.last_t = t
        return self.current()

    def current(self, at: datetime | None = None) -> float:
        """
        The bias-corrected mean, optionally decayed to a later time.

        Dividing by the accumulated WEIGHT rather than assuming it has
        converged to 1/(1-decay) is the bias correction: without it the first
        few observations are pulled toward the zero the accumulator started at.
        """
        if self.weight <= 0:
            return float("nan")
        if at is None or self.last_t is None:
            return self.value / self.weight
        dt = max((at - self.last_t).total_seconds(), 0.0)
        d = 0.5 ** (dt / self.half_life_s)
        return (self.value * d) / (self.weight * d)     # decay cancels; kept for clarity

    def staleness_s(self, now: datetime) -> float:
        """How old the newest observation is. A metric with no freshness
        signal cannot distinguish 'steady' from 'stopped reporting'."""
        return float("inf") if self.last_t is None else (now - self.last_t).total_seconds()

Validation block

PYTHON
def validate_ewm(half_life_s: float = 300.0) -> None:
    """Property tests that catch the three classic implementation errors."""
    from datetime import timedelta, timezone

    t0 = datetime(2026, 5, 14, 9, 0, tzinfo=timezone.utc)
    e = TimeAwareEWM(half_life_s)

    # 1. Bias correction: a single observation must return itself, not half
    #    of it. Failing this means the accumulator starts at zero uncorrected.
    assert abs(e.update(t0, 10.0) - 10.0) < 1e-9, "no bias correction on the first value"

    # 2. Constant input converges to that constant regardless of spacing.
    for i in range(1, 40):
        e.update(t0 + timedelta(seconds=i * 37), 10.0)
    assert abs(e.current() - 10.0) < 1e-6, "constant input does not converge"

    # 3. Time-awareness: a step change decays by half over exactly one
    #    half-life, whether that is one observation or fifty.
    a, b = TimeAwareEWM(half_life_s), TimeAwareEWM(half_life_s)
    a.update(t0, 0.0); b.update(t0, 0.0)
    a.update(t0 + timedelta(seconds=half_life_s), 1.0)
    for i in range(1, 51):
        b.update(t0 + timedelta(seconds=half_life_s * i / 50), 1.0)
    # Both saw the same elapsed time; the count-based bug makes them differ.
    assert abs(a.current() - 0.667) < 0.02, f"single-step value {a.current():.3f} unexpected"
    assert b.current() > a.current(), "denser sampling should weight recent values more"

    # 4. A backwards clock resets rather than amplifying.
    c = TimeAwareEWM(half_life_s)
    c.update(t0, 50.0)
    c.update(t0 - timedelta(seconds=60), 1.0)
    assert abs(c.current() - 1.0) < 1e-9, "backwards timestamp did not reset the state"
    print("OK — bias-corrected, time-aware, clock-safe")

Common mistakes and gotchas

  • Count-based decay on irregular telemetry. No alpha fixes it; the weighting is attached to observation index instead of elapsed time.

  • No bias correction. Newly seen vehicles report artificially low values for the first few half-lives, which on a dashboard reads as idle.

  • Ignoring staleness. An exponentially weighted mean holds its last value forever. A vehicle that stopped reporting looks steady, not absent, unless a freshness field is published alongside it.

  • Decaying by a negative interval. A clock step amplifies old state instead of decaying it, producing an implausible spike. Reset instead.

  • Comparing exponentially weighted values to windowed ones. They are different estimators; a five-minute half-life is not a five-minute window, and the numbers will not match.

  • Using it for anything that must reconcile. An exponentially weighted mean has no fixed denominator, so it cannot be summed or audited. Totals need tumbling windows — see computing tumbling and sliding windows over telemetry streams.

FAQ

Half-life or alpha?

Half-life, always, in the interface. Alpha is the implementation detail; a half-life is a sentence somebody can evaluate — “a change five minutes ago still counts half as much as one now”. They are related by alpha = 1 − 0.5^(Δt / half_life), and computing alpha per observation is exactly what makes the decay time-aware.

Can I get a variance too?

Yes: maintain a second accumulator for the weighted sum of squares and combine them the same way. The result is an exponentially weighted variance whose interpretation is the recent variability, which is often more useful for anomaly detection than the mean itself.

How does this compare to a Kalman filter?

An exponentially weighted mean is a Kalman filter with a fixed gain and no motion model. If the quantity has dynamics worth modelling — position, where velocity predicts the next value — the Kalman filter is strictly better. For a scalar operational metric such as mean speed or utilisation, the extra machinery buys very little and costs state.

Back to Rolling Statistics for Mobility Metrics