My kiosk monitor was green for five hours while the TV showed a permission dialog

A TV in my house runs a dashboard kiosk: an Android box in kiosk-browser mode, pointed at a page I serve, babysat by a cron watchdog that re-launches the browser if anything steals focus. It has been solid for months.

This morning it wasn't. The TV was showing Android's "Allow USB debugging from this computer?" dialog. No dashboard. Every monitor I had on it: green.

The log was fine. That was the problem.

The watchdog had been failing every two minutes since about 02:20:

[02:20:11] WARN living-room cube not found on the LAN (miss 1)
...
[07:20:11] WARN living-room cube not found on the LAN (miss 150)

150 consecutive failures. Five hours. And the healthcheck on that automation was a file-mtime probe: does the watchdog's log file keep getting written?

It did. Enthusiastically. Once every two minutes, with a failure message in it.

That's the whole bug. The check asserted the job ran, not that the job worked. A failing loop is still a running loop, and a log that fills with errors is the freshest log on the box.

The actual fault was in my client, not the device

"Not found on the LAN" was a lie the resolver told me. The device was there:

$ timeout 2 bash -c 'echo >/dev/tcp/$TV/5555' && echo OPEN
OPEN
$ adb devices
List of devices attached
<tv>:5555   unauthorized

Port open, adb handshake dead. The resolver identifies the box by getprop ro.product.name, and on an unauthorized device that returns an empty string, so "is this the TV I want?" answered no — forever.

The fix took one command:

$ adb kill-server && adb start-server && adb connect $TV:5555
$ adb devices
<tv>:5555   device

The long-lived adb server on the host had wedged. Not the TV, not its authorization, not DHCP (which has been the culprit before). A daemon on my side that had quietly gone stupid and had no way to notice.

So step one was to teach the resolver to suspect itself: if the port is open but the handshake fails, bounce the local adb server once and retry before declaring the device missing.

probe() {  # probe <ip> -> 0 if it's the device we want
  local ip="$1" name
  timeout 1.5 bash -c "echo >/dev/tcp/$ip/5555" 2>/dev/null || return 1
  name=$(handshake "$ip")
  if [ "$name" != "$WANT" ] && adb_bounce; then
    adb disconnect "$ip:5555" >/dev/null 2>&1
    name=$(handshake "$ip")
  fi
  [ "$name" = "$WANT" ]
}

Port closed means the box is genuinely gone. Port open and no handshake means somebody is broken, and the cheap thing to rule out first is me.

Probe the artifact

The real repair was replacing the healthcheck. Not "did the cron run", but "is the thing the cron exists to produce actually true":

TV=$(resolve_tv) || { echo "KIOSK_OFF"; exit 0; }   # TV is off. Not a fault.

name=$(adb -s "$TV" shell getprop ro.product.name | tr -d '\r\n')
[ "$name" = "$WANT" ] || { echo "KIOSK_FAIL adb handshake"; exit 1; }

adb -s "$TV" shell pidof "$KIOSK_PKG" >/dev/null \
  || { echo "KIOSK_FAIL kiosk browser not running"; exit 1; }

echo "KIOSK_OK"

Three decisions in there that matter more than the code:

Silence when there's nothing to assert. The box drops off the network when the TV is powered off. That's not an outage, that's a television. Port closed everywhere → exit 0, say so, don't page anyone. A check that cries during normal behaviour gets muted, and a muted check is a deleted check with extra steps.

Don't assert more than you mean. My first draft required the kiosk browser to be foreground. That would fire every time somebody watched a movie. Foreground is the watchdog's job, with its own idle rules. The health probe only asserts the browser is alive — the weakest condition that still catches every real failure I've had (wedged adb; kiosk app not auto-starting after a reboot).

Distrust anything downstream of your own code. The file-mtime probe was measuring my watchdog's diligence. The new one measures the TV.

What I'd generalize

Go look at your monitors and ask, for each one, what does green actually prove? Mine proved a cron was scheduled correctly. It could not have told the difference between a working dashboard and a black screen — and for five hours, it didn't.

If the assertion can be satisfied by a failing job, it isn't a healthcheck. It's a heartbeat with good PR.