Handling late-arriving GPS events with watermarks

Allowed lateness is a measurement, not a configuration default. Collect arrival-minus-event-time over a representative period, look at the tail rather than the mean, set the watermark at a high percentile of it, and then decide explicitly what happens to the events beyond that percentile — because “dropped silently” is the decision you get if you make no decision at all.

Why this happens

Mobility devices do not deliver on a schedule; they deliver when they have power, signal and a reason to. A vehicle parked in an underground bay buffers for hours and then flushes; a phone in aggressive battery-saving mode batches an afternoon into one upload; a rural tracker waits for a cell it can reach. The result is a delivery-lag distribution with a small, tight body and a long, heavy tail — and the tail is not random. It correlates with underground parking, tunnels and rural coverage, which means dropping it biases the data geographically.

That is the whole argument for measuring rather than guessing. A default of zero allowed lateness silently deletes the events hardest to collect, and the resulting map shows less activity in exactly the places where collection was hard. The window mechanics this feeds are set out in streaming window processing; this page is about choosing the one number that mechanism needs.

The body and the tail come from different physical situations, which is why no single summary statistic captures both. The watermark is a choice about how much of the tail to wait for.

Core pipeline

Instrument both timestamps. Record event time and ingestion time on every record; without both, the lag is unmeasurable after the fact.
Build the lag distribution per population. Segment by device type or fleet, because their tails differ by orders of magnitude.
Set allowed lateness from a percentile, not a round number, and re-derive it on a schedule.
Route the residual. Send events beyond the watermark to a side output with a counter, and reconcile them in the batch path.

Production-ready Python implementation

PYTHON

import numpy as np
import pandas as pd


def measure_delivery_lag(
    df: pd.DataFrame,
    event_col: str = "event_time",
    arrival_col: str = "ingest_time",
    group_col: str | None = "device_class",
    percentiles: tuple = (50, 95, 99, 99.9),
) -> pd.DataFrame:
    """
    Summarise delivery lag per population, which is what a watermark is set from.

    Parameters
    ----------
    df : pd.DataFrame
        Must carry BOTH timestamps, tz-aware. If ingest_time was not recorded
        at ingestion, this cannot be reconstructed later — instrument first.
    group_col : str | None
        Population to segment by. A single watermark across mixed populations
        either drops the slow one's data or delays the fast one's results.

    Returns
    -------
    pd.DataFrame
        One row per population with the requested lag percentiles in seconds.

    Raises
    ------
    ValueError
        On missing columns, naive timestamps, or negative lag.
    """
    for c in (event_col, arrival_col):
        if c not in df.columns:
            raise ValueError(f"Missing required column: {c}")
        if df[c].dt.tz is None:
            raise ValueError(f"{c} must be timezone-aware; naive timestamps "
                             "silently mix zones and the lag becomes meaningless.")

    lag = (df[arrival_col] - df[event_col]).dt.total_seconds()
    if (lag < -1.0).any():
        # Arrival before the event means a clock problem on the device, not
        # a fast network. Fix the clock before trusting any of this.
        raise ValueError(
            f"{(lag < -1.0).sum()} events arrived before they happened — "
            "device clock drift, not delivery lag."
        )

    work = df.assign(_lag_s=lag.clip(lower=0.0))
    keys = [group_col] if group_col else []
    rows = []
    for name, g in (work.groupby(keys, sort=False) if keys else [("all", work)]):
        rec = {"population": name if keys else "all", "n": len(g)}
        for p in percentiles:
            rec[f"p{p}_s"] = float(np.percentile(g["_lag_s"], p))
        rec["mean_s"] = float(g["_lag_s"].mean())     # reported, not used
        rows.append(rec)
    return pd.DataFrame(rows)


def recommend_watermark(summary: pd.DataFrame, percentile: str = "p99_s",
                        floor_s: float = 5.0, ceiling_s: float = 21600.0) -> dict:
    """
    Turn a lag summary into an allowed-lateness setting per population.

    The ceiling is a business decision, not a statistical one: waiting six
    hours for the last 0.1% may be correct for a compliance report and absurd
    for a live dashboard. Clamping makes that choice explicit rather than
    letting one pathological device set the latency for everyone.
    """
    if percentile not in summary.columns:
        raise ValueError(f"{percentile} not in the summary; recompute with it included.")
    out = {}
    for _, r in summary.iterrows():
        chosen = float(np.clip(r[percentile], floor_s, ceiling_s))
        out[r["population"]] = {
            "allowed_lateness_s": chosen,
            "expected_drop_frac": 0.01 if chosen >= r[percentile] else float("nan"),
            "clamped": bool(chosen != r[percentile]),
        }
    return out

Validation block

PYTHON

def validate_watermark(events_dropped: int, events_total: int,
                       allowed_s: float, observed_p99_s: float) -> None:
    """Assert the watermark is doing what it was sized to do."""
    drop = events_dropped / max(events_total, 1)
    # 1. Drop rate must be near the design target. Much higher means the lag
    #    distribution moved; near zero means the watermark is over-generous
    #    and you are paying latency for nothing.
    assert drop < 0.03, f"dropping {drop:.2%} — the lag distribution has shifted"
    assert drop > 0.0001 or allowed_s < observed_p99_s * 3, (
        "dropping nothing and waiting far past p99 — reduce allowed lateness"
    )
    # 2. The setting should still be near the measured percentile.
    ratio = allowed_s / max(observed_p99_s, 1e-6)
    assert 0.5 < ratio < 4.0, (
        f"allowed lateness is {ratio:.1f}x the observed p99 — re-derive it"
    )
    print(f"OK — dropping {drop:.3%} at {allowed_s:.0f} s (p99 {observed_p99_s:.0f} s)")

The curve flattens fast, which is why "wait a bit longer to be safe" is rarely the right instinct. Past the knee, the batch reconciliation path is cheaper than the latency.

The correlation between delivery lag and place is the whole argument for measuring the distribution. A uniform 1% loss would be tolerable; a loss concentrated in car parks and the countryside is a finding waiting to be misread.

Common mistakes and gotchas

Not recording ingestion time. The lag is unrecoverable afterwards. Stamp it at the edge, before any queue, and keep it.
Setting the watermark from the mean. A heavy-tailed distribution has a mean nowhere near its body; the p99 is the only summary that maps onto “how much do I capture”.
One watermark for mixed populations. Scooters and HGVs differ by two orders of magnitude in tail lag. Partition, or accept that one of them is badly served.
Dropping late events without counting them. The drop rate is the only way to notice that the fleet, the network or the firmware changed. A silent drop shows up months later as an unexplained dip.
Ignoring negative lag. Arrival before the event means device clock drift, which time-series synchronization strategies covers; treating it as fast delivery corrupts the percentiles.
Never re-deriving the setting. A firmware update or a new device model shifts the distribution. Re-measure monthly and alert on the change, not just the level.

FAQ

Should late events update the emitted result or go to a correction path?

Both, at different scales. Events within allowed lateness should update the result in place, keyed by window and entity so the update is idempotent. Events beyond it belong in a batch reconciliation that recomputes the window from the archive later — which also gives you a free measurement of how much the watermark is missing.

What if a small number of devices dominate the tail?

Partition them out. A handful of vehicles that always park underground should not set the latency for a fleet of thousands. Giving them their own population with a longer allowed lateness, and letting their results finalise later, is almost always better than either dropping them or delaying everyone.

Does allowed lateness affect state size?

Yes, directly. State has to be retained until the window is sealed, so allowed lateness multiplies the number of open windows. Doubling it from eleven minutes to twenty-two roughly doubles retained window state — which is the practical ceiling on how generous the setting can be.

Streaming Window Processing — the parent reference and the window semantics this setting feeds.
Time-Series Synchronization Strategies — device clock drift, which contaminates lag measurement if uncorrected.
Computing Tumbling and Sliding Windows over Telemetry Streams — the state cost that allowed lateness multiplies.
Gap Filling in Sparse Trajectories — what to do about the intervals that never arrive at all.
Rolling Statistics for Mobility Metrics — the batch counterpart that reconciles the dropped tail.

Back to Streaming Window Processing