The keepalive nobody could see

The dashboard on my wall said a job had been running for 11 hours and 33 minutes. The server it streams from said there were no jobs at all. Both were telling the truth.

The panel gets its live job tiles over Server-Sent Events. The TV runs a kiosk browser, the TV's wifi goes to sleep, and the socket went half-open: the client's TCP stack still has a connection object, the server's is long gone, and nobody sends anything to find out. Classic. The part that made it stay broken for eleven hours was more interesting.

Three bugs stacked up

1. The client could not detect it. EventSource never fired onerror — nothing errored, the socket just stopped speaking — and readyState stayed OPEN forever, because readyState reflects what the browser believes, not what the network is doing. So every reconnect path I had was gated behind a condition that would never be true.

2. The heartbeat was invisible. The server sent a keepalive every 15 seconds, which I had been treating as the liveness signal:

yield b": keepalive\n\n"

That is an SSE comment. Its entire purpose is to push bytes down the wire so proxies don't reap an idle connection — and it works fine for that. But comments are consumed by the EventSource parser and never surfaced to JavaScript. No onmessage, no event of any kind. From the page's point of view, a server pinging politely every 15 seconds is indistinguishable from a server that died. I had a heartbeat with no stethoscope.

The fix is one character-for-character swap to a real frame:

yield b'data: {"type":"ping"}\n\n'

Now the client can time the stream out, because now there is something to time:

// Stamp on EVERY message, ping or real payload.
CL.lastRx = Date.now();

// ...and in the tick that runs regardless of what's on screen:
if (CL.es && Date.now() - CL.lastRx > 50000) {
  CL.es.close();          // don't ask readyState, it will lie
  connectLiveFeed();      // redial
}

Two details there cost me more time than the diagnosis. The watchdog has to run even when no live tile is visible — mine was inside the render path for the live panel, so a stream that died while the dashboard was rotating through other pages could never be noticed, and therefore never came back at all. And the stale-tile check has to age tiles by local receipt time, not the started_at the server sent, because the TV's clock drifts and a drifted clock will happily decide a stale tile is from the future.

3. The server was lying too, a little. A worker killed mid-flight — host reboot, OOM, watchdog — never runs its "job finished" path, so its tile stayed running in the registry with nothing left alive to ever close it. Tiles now get reaped when nothing has reported on them for a while, which is the boring lesson: any state a client can enter and never leave needs a server-side expiry, not just a client-side one.

What I'd do differently

Assume every long-lived stream will eventually go silent without erroring, and design the client around a timer instead of a connection state. A half-open socket is the normal case on consumer wifi, not an edge case.

And the transferable bit, the one I'll actually remember: an SSE comment keeps proxies happy, not clients. If your client needs to notice silence, send it something it can see.