A forecast turned off every automation I own

I have about eighteen background jobs that spend from a shared weekly quota, and a gate in front of all of them. Before a job runs it asks: given where the week is, where is it projected to land? Over your tier's ceiling, skip this cycle. The whole point is that a scheduled job must never drain the budget I need for myself.

Monday morning, a monitor paged: one of those jobs was 2.8 hours behind. The job wasn't broken. It hadn't run in five and a half hours, because the gate kept telling it to skip.

Here's what the gate was looking at:

weekly usage      15%   (17h into a 168h week)
projected peak   120%   [empirical, median of 6 weeks]
  linear est.    148%

Fifteen percent, six days to go, and the gate had turned off the entire tier.

The estimator was fine. The decision rule wasn't.

Naive linear extrapolation is terrible here — usage is front-loaded, so dividing a few hours of burn by a few hours and multiplying out projects >100% on essentially any Monday. I already knew that; there's a guard for it, added after linear-from-start silently killed five consecutive nights of a job.

So the primary estimator is empirical. For the current elapsed-hours mark, look at every past week, take that week's reading at the same mark and its eventual peak, and the ratio is "how much more was still to come from here." Median those, multiply by today's reading.

Reasonable. Now look at the actual pool at hour 17:

w/e 2026-07-26   14% ->  77%  (x5.5)
w/e 2026-08-09    6% ->  89%  (x14.8)
w/e 2026-08-16    8% ->  80%  (x10.0)
w/e 2026-08-23   12% ->  94%  (x7.8)
w/e 2026-08-30   10% ->  82%  (x8.2)
w/e 2026-09-06   13% ->  54%  (x4.2)

The multiplier ranges from 4.2 to 14.8. Multiply a 15% reading by anything in that range and you get somewhere between 62% and 222%. That is not a forecast. That's a shrug with a decimal point on it.

And the punchline is in the right-hand column: every one of those six weeks actually finished between 54% and 94%. Not one went over. The model confidently predicted an event that had never happened, because this week's hour-17 reading happened to be one point higher than any comparable week's, and a multiplicative model amplifies exactly that.

The fix: make the expensive verdict carry the burden of proof

The two outcomes are not symmetric. Running when it turns out the week was hot costs one percent of a budget with a hard stop behind it. Skipping costs the work, permanently — nobody replays Monday.

So the skip decision now uses the 25th-percentile ratio instead of the median. The median is still what gets reported; it just doesn't get to make the call.

if len(ratios) >= 4:
    opt_ratio = statistics.quantiles(sorted(ratios), n=4)[0]
elif ratios:
    opt_ratio = min(ratios)

# ... in decide()
opt = proj.get("projected_optimistic_pct")
if over and opt is not None and opt <= threshold:
    return True, (
        f"median projection {proj['projected_pct']:.0f}% is over the "
        f"{threshold:.0f}% ceiling but a light week lands at {opt:.0f}% "
        f"— too wide to skip on, allowing"
    )

Read out loud: skip only if even a light week lands over the ceiling.

Monday's numbers, after:

projected peak   128%   [empirical, median of 6 weeks]
  linear est.    158%
  light-week end  83%   (x5.16) — a skip needs THIS over the ceiling

free-time   ceiling 75%   SKIP
day-to-day  ceiling 90%   RUN

Free-time jobs still skip, which is correct — 83% is over their 75% ceiling, and those are the ones that are genuinely fine to miss a night.

The part I like: I didn't have to add a time threshold. My first instinct was another MIN_ELAPSED_HOURS constant, a second magic number sitting next to the first one. But ratio dispersion collapses on its own as a week fills in. At hour 17 the spread is 3.5x; by Friday every comparable week's multiplier is around 1.05 and p25 is the median. The rule self-tightens. The uncertainty was already in the data — I just wasn't reading it.

The two hard stops are untouched: if actual usage is already over the ceiling, skip, no projection involved. Same if the short-window rate limit is pegged, because then the call fails anyway.

What I'd tell past me

The original bug report was "this job is behind." The actual bug was three layers up, in a forecast that was working exactly as designed. I'd already patched this failure mode once, on the other estimator, with a threshold — and then built a second estimator with the same flaw and no guard. A patch that names one code path is a patch you'll write again.

And the general one, which I keep relearning: if a prediction drives an irreversible action, the prediction needs an error bar and the action needs to respect it. A point estimate handed to an if statement throws away the only information that mattered, which was how badly the estimate could be wrong.