The Fourth Action: When the Right Answer Is 'I'll Call You Back'

My after-hours triage agent had three terminal actions: GATHER (get more information), RESOLVE (handle it now), and ESCALATE (page the on-call engineer). Last week I added a fourth: CALLBACK.

Here's why it was missing and why it mattered.

The problem with three actions

The original design modeled after-hours calls as a binary: either the system can handle it without waking anyone up (RESOLVE), or it can't and someone gets paged (ESCALATE). GATHER is the middle state while the agent is figuring out which one applies.

That's the right model for most calls. But there's a class of call it handles badly: the non-urgent request where the caller explicitly doesn't want immediate help.

"My internet is slow but it's not critical — can someone just call me back in the morning?" or "I locked myself out of the VPN but I'm done for the day anyway, is there a ticket I can follow up on tomorrow?"

Under the old model, these calls get ESCALATED. The agent can't resolve them (no one on the system can reset a VPN password at 11pm without human involvement), and the only other option is to tell the caller to wait. But waiting without a ticket feels like abandonment. So the agent pages the on-call engineer.

An engineer gets woken up. They call back. The caller says "oh, it's not urgent, I just wanted to leave a note." The engineer is annoyed. The caller is embarrassed. Nobody wins.

The right answer was obvious in retrospect: the agent should be able to say "I'll create a ticket and have someone call you in the morning" and then actually do that.

What CALLBACK does

When the agent classifies a call as CALLBACK:

1. It asks the caller when they'd like to be reached. The agent normalises this — "tomorrow morning", "before lunch", "whenever", "Monday afternoon" all get mapped to a human-readable label.

1. It creates a ConnectWise ticket prefixed with [CALLBACK REQUESTED] and sets the status to "Callback - Scheduled". The ticket body includes the preferred callback time, a brief summary of the issue, and a transcript snippet.

1. It gives the caller a warm closing speech: "I've created a ticket for you. The team will reach out tomorrow morning. If anything changes in the meantime, the ticket number is SR-4892."

The on-call engineer is never paged. The ticket sits in the "Callback - Scheduled" queue. In the morning debrief — the Slack message that goes out at 6am to summarize overnight activity — there's a line: "3 callbacks scheduled 📞 — SR-4892, SR-4893, SR-4895."

The morning team opens their queue and sees exactly who to call and why.

The NL time normalisation piece

"I'll call you back" is simple. "When would you like us to call?" is where it gets interesting.

Callers say things like:

• "ASAP" — which means as early as you can
• "First thing in the morning" — business hours, first slot
• "Whenever is convenient" — no preference
• "Monday" — specific day, default to business hours
• "Before my 10 o'clock" — relative to a time they have, which I can't know

I implemented normalise_callback_time() as a regex-based classifier that maps natural language to one of a small set of display labels:

_PATTERNS = [
    (r"\b(asap|immediately|urgent|right now|soon)\b", "ASAP"),
    (r"\btomorrow\s+morning\b",                         "Tomorrow morning"),
    (r"\btomorrow\b",                                   "Tomorrow (business hours)"),
    (r"\b(monday|tuesday|wednesday|thursday|friday)\b", lambda m: f"{m.group().title()} (business hours)"),
    (r"\b(morning)\b",                                  "Tomorrow morning"),
    (r"\b(afternoon)\b",                                "Tomorrow afternoon"),
]

For anything that doesn't match, the label is "Unspecified — first available." The ticket body always includes the raw caller transcript snippet too, so the morning team can read exactly what the caller said rather than relying on the normalised label.

I deliberately didn't use an LLM for this step. The normalisation is simple enough that regex is reliable and fast, and I didn't want to introduce a failure mode where a flaky LLM call corrupts the callback time field.

The morning debrief integration

The morning debrief already counted ESCALATE calls, RESOLVE outcomes, and issues that fell through to GATHER. Adding CALLBACK was straightforward — the calls table has an action column, so:

callbacks = [c for c in calls if c.get("action") == "CALLBACK"]
n_cb = len(callbacks)
if n_cb:
    summary_parts.append(f"{n_cb} callback{'s' if n_cb != 1 else ''} scheduled 📞")
    ticket_ids = [c["ticket_id"] for c in callbacks if c.get("ticket_id")]
    if ticket_ids:
        fields.append(f"Callbacks to make today: {', '.join(f'SR-{t}' for t in ticket_ids[:5])}")

The "📞" is the only emoji in the debrief. It earns its keep because it makes the callback summary visually distinct from the escalation count and the resolve count when you're reading on a phone over coffee.

Why this matters for the pilot pitch

Before CALLBACK, the pitch was: "NightDesk handles calls without waking your on-call engineer." The implicit assumption was that every call either gets resolved or gets escalated.

With CALLBACK, the pitch becomes more accurate: "NightDesk handles calls appropriately — resolves what it can, escalates what it must, and schedules callbacks for everything in between."

That middle bucket is probably 20–30% of after-hours calls for a typical MSP. Clients calling about something non-urgent at 9pm don't want to wake a technician. They want to know someone heard them and will follow up. CALLBACK gives them that without creating an unnecessary page.

The CW ticket is the paper trail. The morning team works through the callback queue the same way they work through any other queue. No new workflow, no new tooling — just a status value they already had ("Callback - Scheduled") and a naming convention ([CALLBACK REQUESTED] prefix) that makes them easy to filter.

The agent now has four terminal actions. Three of them existed before. The fourth one existed as a capability — ConnectWise has always had ticket statuses and the morning team has always made callbacks — but the agent had no way to invoke it. Now it does.

Sometimes adding a feature is just surfacing something that was already there.