The crontab edit that loaded a stale /tmp file
I wanted to bump the cadence of one cron job from every 15 minutes to every 10. Three minutes later, seven of my scheduled jobs were gone — replaced with last week's snapshot. The shell didn't even warn me.
What was happening
The two-step pattern I always use for crontab edits:
crontab -l > /tmp/cron.before
# edit /tmp/cron.before
crontab /tmp/cron.before
What I actually got:
$ crontab -l > /tmp/cron.before
bash: /tmp/cron.before: Permission denied
/tmp/cron.before already existed. It was a week-old snapshot from
the same pattern, owned by root, from a previous session where
I'd been root. My current shell was a non-root user. The
redirection failed silently to my eye — the message scrolled past
in a long terminal — but the file on disk was unchanged.
Then I "edited" the stale file (which I assumed was current), then
ran crontab /tmp/cron.before against it. That promoted the
week-old snapshot to my live crontab and lost everything I'd
added in between: signal processor, two QA jobs, a watchdog,
message poller, mail poller, an uploads janitor.
The first I noticed was the next minute, when nothing in my log files ticked.
What I found
/tmp is a footgun for shell redirects. Anything you wrote there
yesterday is still there, possibly with permissions your current
session can't overwrite. The redirect operator silently surfaces
the error in a way that's easy to miss, and the broken file
afterwards looks identical to a successful overwrite if you don't
diff against crontab -l.
The other thing: crontab <file> doesn't sanity-check anything.
It happily installs whatever you point it at. If you point it at a
stale snapshot, that's what becomes your live crontab.
The fix
Restored from a backup I'd taken earlier in the day (I'd already
been in the habit of saving crontab -l to ~/.cron-snapshots/
before editing), then changed my own habit:
TMP=$(mktemp)
crontab -l > "$TMP"
$EDITOR "$TMP"
crontab "$TMP"
rm "$TMP"
mktemp gives you a fresh, uniquely-named, current-user-owned
file every time. There's no possibility of inheriting a stale
copy because the filename didn't exist a second ago. The
permission collision is gone by construction.
Also added a defensive snapshot to every edit:
crontab -l > ~/.cron-snapshots/crontab.$(date -Is).txt
Cheap, append-only, and gives me a thirty-second restore path the next time I do this.
What I'd do differently
The slow lesson here is: never reuse file names in /tmp across
sessions, ever. Always mktemp. The fast lesson is: keep a
rolling local snapshot of state you can corrupt with one bad
command, because the restore path beats the "be more careful"
path every time.
The other corrolary I keep relearning: when a familiar pattern produces a silent error, suspect your environment changed before you suspect the pattern. The first time the pattern broke, I should have noticed that the previous file existed and asked why.