The alert that fired because the system was working

I have a little daemon that keeps OAuth tokens alive. It runs once a day, reads how much runway each token has left, and renews anything getting close to expiry. It also knows one annoying fact about the CLI it renews for: a process started before the renewal will write its stale in-memory credentials back over the fresh ones when it exits. So if there's a live session running, the daemon politely defers — better to try again tomorrow than to renew into a session that's going to clobber it on the way out.

That deferral is the whole point. It's the correct behavior.

It also paged me every single day.

Two numbers that were the same number

The monitor's probe was the same script, run read-only:

bad = [r for r in rows if r["state"] != "ok"]
return 1 if bad else 0

Exit non-zero if any target isn't healthy. Sensible. And warn — the state that means "start thinking about renewing" — begins at seven days of runway.

So: token drops under seven days. Daemon says "warn, but there's a live session and plenty of runway, I'll defer." Monitor says "state != ok, exit 1, page." Both are right. Every day, for as many days as the deferral lasts, the monitor reports a broken automation while the automation does exactly what it was designed to do.

The bug isn't in the daemon and it isn't in the probe. It's that warn was being used for two different jobs: it was the daemon's cue to act, and it was the monitor's definition of failure. One threshold, two meanings, and the meanings don't overlap at all. The daemon's "I should think about this" window is precisely the window in which the monitor has nothing useful to say.

Give the alert its own line

The fix is to stop asking "is anything not-ok" and start asking "is anything past the point where the daemon should already have fixed it."

def failing(rows, fail_below):
    """Rows that should make this process exit non-zero."""
    if fail_below is None:
        return [r for r in rows if r["state"] != "ok"]
    bad = []
    for r in rows:
        if r["state"] == "ok":
            continue
        if r["state"] != "warn":
            bad.append(r)      # expired/unreadable always fail
            continue
        days = r.get("days_left")
        if days is None or float(days) < fail_below:
            bad.append(r)
    return bad

warn with more runway than the line prints but passes — the daemon owns that window, and the output still shows it so a human running the script by hand sees the whole picture. Anything actually broken still fails immediately: an expired token, an unreadable credential file.

Then the two numbers get separated deliberately. The daemon's deferral floor moved from 2 days to 4; the monitor's alert line sits at 2.5. The daemon runs daily, so worst case it acts with ~3 days left, and the monitor stays quiet unless a renewal actually failed — at which point there are still 2.5 days to do something about it.

One more detail that mattered: the status artifact the probe reads keeps recording the strict grade. ok still means everything is genuinely healthy. Only the exit code relaxed. If you fudge the stored state to silence the alert, you've deleted the signal instead of routing it.

The general shape

Any automation with a "wait and try again later" window has this trap in it. Retry backoff, maintenance windows, rate-limit deferrals, batch jobs that skip a run because the upstream isn't ready — all of them spend time in a state that is neither healthy nor broken, and a monitor that grades on != healthy will page for the entire duration.

The rule I'd write on the wall:

Your alert threshold must sit strictly below the floor of any window in which the system is deliberately doing nothing.

If they touch, you can't tell "working as designed" from "broken," and after a week of that you're not reading the alerts anyway — which is the actual failure, and it's a much more expensive one than a stale token.