Two ways a CloudWatch alarm quietly stops meaning anything

I spent this morning on a monitoring alert and ended up fixing the monitoring instead of the thing it was monitoring. Twice, for two different reasons, both of which I'd call design bugs rather than mistakes.

A page-only alarm rots into permanent red

Here's an alarm I found that had been in ALARM for 92 consecutive days:

StateValue:    ALARM
StateReason:   Threshold Crossed: 1 datapoint [1.0 (04/06/26 10:01:00)]
               was greater than or equal to the threshold (1.0).
AlarmActions:  ["arn:aws:sns:...:alerts"]
OKActions:     []

Look at the last two lines. It can page you when the job breaks. It has no way whatsoever to tell you it recovered.

That asymmetry is worse than it sounds. The first time it fires you get an email, you glance at it, maybe you're busy. From then on the alarm is just... red. There's no event that ever un-reds it in your inbox, so the only way to learn it recovered is to go look at the console, which is exactly the thing the alarm existed to save you from. After a week it's furniture. After three months nobody remembers it's supposed to be green.

The fix is one flag, and you should treat it as mandatory:

aws cloudwatch put-metric-alarm \
  --alarm-name nightly-sync-failing \
  ... \
  --alarm-actions arn:aws:sns:us-east-1:ACCOUNT:alerts \
  --ok-actions    arn:aws:sns:us-east-1:ACCOUNT:alerts

If you have an alarm worth paging on, you have an alarm worth un-paging on. Go audit yours for empty OKActions — I'd bet money you have at least one.

A 24-hour Period is a bucket, not a window

Second one, and this is the subtler of the two.

You want "this nightly job hasn't succeeded in 24 hours." The obvious encoding is a log metric filter that emits 1 on the job's success line, and an alarm like:

Period:            86400
EvaluationPeriods: 1
Threshold:         1
ComparisonOperator: LessThanThreshold
TreatMissingData:  breaching

Reads perfectly in English. It is also wrong, because a CloudWatch Period of 86400 is a fixed bucket aligned to UTC midnight, not a trailing 24 hours.

So if your job runs at 10:00 UTC, then every day between 00:00 and 10:07 the current day's bucket is legitimately empty. Missing, treated as breaching, page. The job then runs, the bucket gets its datapoint, and the alarm clears itself. That's a ten-hour false-alarm window, every single day, on an alarm that is technically working exactly as configured.

And a daily false page is how you train yourself into problem one. You learn the alarm is noise, you stop reading it, and then it's furniture again.

What you actually want is a rolling window, which you build out of an hourly period and 24 evaluation periods:

aws cloudwatch put-metric-alarm \
  --alarm-name nightly-sync-not-completing \
  --namespace MyApp/Sync --metric-name SyncCompleted --statistic Sum \
  --period 3600 \
  --evaluation-periods 24 \
  --datapoints-to-alarm 24 \
  --threshold 1 --comparison-operator LessThanThreshold \
  --treat-missing-data breaching

Now the semantics are "all 24 of the trailing one-hour buckets contained no success," which is what you meant in the first place. It can't be tripped by midnight alignment, and it still catches the real case.

The general rule: for "X hasn't happened in N hours," use Period 3600 / EvaluationPeriods N / DatapointsToAlarm N. Never Period N*3600 / EvaluationPeriods 1.

The same bug, but with arithmetic

There's a nastier version of the bucket problem if you're using metric math. I had a second alarm computing a gap between two series:

inmap  = MAX(TenantsFound)     # emitted early in the run
synced = MAX(TenantsSynced)    # emitted at the end of the run
alarm on: inmap - synced >= 1

With a 24-hour period and one of those series missing from the window, the subtraction didn't return "no data." It returned inmap. The alarm cheerfully paged with a gap of 140 when the real answer was 13.

Dropping to Period 3600 fixed it too, and for a reason worth internalising: both metric filters fire during the same run, so an hourly bucket guarantees both operands land in the same bucket and the subtraction is between two real numbers. An hourly period isn't just less noisy here — it's what makes the arithmetic meaningful at all.

What I'd do differently

Metric filters don't backfill. If you create one and an alarm over it in the same breath, with TreatMissingData: breaching, the alarm evaluates against an empty namespace and pages you within the minute. That's what started this whole morning — a monitor paging about itself, 78 seconds after being born.

So: create the filter, trigger the source once to seed a real datapoint, then arm the alarm. And before you walk away from any new alarm, ask it two questions — can you tell me when this recovers, and is your window the window I actually meant?