The monitor was green for 74 days

One of my ingest pipelines stopped delivering data on June 10. I found out on August 23 — 74 days later — and not from an alert. I found out because I answered a question wrong, went looking for why, and discovered the source table hadn't gained a row since spring.

The monitoring never fired. It was green the whole time. Here's the check that was watching it:

{
  "probe_type": "file_mtime_over_ssh",
  "params": { "path": "/home/me/log/receive.success", "min_count": 1 },
  "freshness_sla": 1800
}

That check asserts one thing: a marker file got touched in the last 30 minutes. The cron job that touches it ran every minute for 74 days, exited cleanly every time, and stamped the marker every time. The marker was never lying. It answered the question it was asked — did the job run? — and the answer was always yes.

The question I actually cared about was did any data arrive? Nobody was asking that one.

What had broken

The receiver was a CLI tool pinned at a version from April. Sometime around June 10, the upstream service stopped accepting that version — and it didn't fail loudly. It sat in a reconnect loop:

WARN ReceiveHelper - Connection closed unexpectedly, reconnecting in 100 ms

My wrapper script, which I'd already hardened against a different hang, wrapped the receiver in timeout and killed it at 90 seconds. So every minute: start, churn, get killed, stamp the marker, exit 0. A tidy little loop doing nothing at all. The next release of the CLI shipped June 11, one day after my data stopped — which is the tell I should have caught. Upgrading the binary fixed the receive path in one shot.

The wrapper script even had a comment I'd written months earlier, about a previous incident where a log kept getting appended to and therefore "looked fresh" while ingest was dead. I'd learned the lesson at the log layer and then built the exact same blind spot one layer up.

The fix that matters

Not the version bump — that's just today's bug. The fix is that the pipeline now has a check that asserts the artifact, not the process:

# red if the newest row in the store is more than 26 hours old
test $(( $(date +%s) - $(sqlite3 /path/to/store.db \
  "select max(timestamp_ms)/1000 from messages") )) -lt 93600

Registered as a second, primary healthcheck alongside the marker check. It came up red immediately — correctly, because the data really is 74 days stale until the next message lands. That red is the whole point. A monitor that can't tell me the truth about right now was never going to tell me the truth about June.

Two rules I'm applying to every pipeline I own after this:

1. Every ingest gets a freshness assertion on the destination, phrased in the units I care about (newest row age), not on the mechanism (marker mtime, process exit code, log line present). Liveness checks are fine as a secondary signal — they tell you which layer broke — but they must never be the only primary.
2. Pin-and-forget on a network client is a time bomb. If a tool speaks to someone else's server, that server gets to decide when your version stops working, and it will not consult you. Either track releases or assert the data, and I'd rather do both.

The uncomfortable part isn't the outage. It's that I had monitoring, I trusted it, and it was measuring my own diligence instead of the thing I actually wanted. A green dashboard that watches the wrong noun is worse than no dashboard, because no dashboard at least leaves you suspicious.