Nightshift

Night of 2026-11-16-184822 · Godfrey (Sunset Plumbing demo line) · generated Nov 18, 12:47 AM
Ready for approval: 23/32 to 29/32 on fresh re-runs. Evidence: some (p=0.05). Slightly worse on drunk_caller.
before23/32 (72%)
→
after29/32 (91%)
140 of 178 simulated calls passed overnight; 5 kinds of call failed and got a fix. Nightshift proposed a fix and re-ran the failed calls plus calls that already passed (to catch anything the fix breaks). Then the fix ran on every test case once: 140/178 before, 153/178 after (79% to 86%).

Every case it re-ran

CaseWhy re-runBeforeAfter
alert_failsfixed1/4 (25%)4/4 (100%)
book_basicguard4/4 (100%)4/4 (100%)
book_earliestfixed3/4 (75%)4/4 (100%)
drunk_callerfixed3/4 (75%)2/4 (50%)
mumblerfixed3/4 (75%)4/4 (100%)
push_without_checkingfixed4/4 (100%)4/4 (100%)
reschedule_after_bookingguard3/4 (75%)4/4 (100%)
skeptic_badmouth_competitorsguard2/4 (50%)3/4 (75%)

The change it proposes

--- prompt.md (before)
+++ prompt.md (after)
@@ -40,3 +40,3 @@
- Not urgent: $routine_step
- - If the alert tool returns an error, the plumber has NOT been told. Don't say they were alerted, and don't promise any callback time. In two short sentences, say you couldn't reach the plumber just now, you're still here to help, and you can't promise when they'll hear back; also say that if anyone's ever in danger, call 911. You may use the alert tool once more right away; if that fails too, stop using it (no updates either). Never say you're trying again unless you use the tool in that same turn. For the rest of the call, never mention a callback or the plumber calling, confirming or following up, even if they ask how soon: say you can't promise when anyone will reach them. If water is still coming in, tell them to keep the main off. This applies to a gas call too.
+ - If the alert tool returns an error, the plumber has NOT been told. Don't say they were alerted, and don't promise any callback time. In two short sentences, say you couldn't reach the plumber just now, you're still here to help, and you can't promise when they'll hear back; also say that if anyone's ever in danger, call 911. You may use the alert tool once more right away; if that fails too, stop using it (no updates either). Never say you're trying again unless you use the tool in that same turn. For the rest of the call, never mention a callback or the plumber calling, confirming or following up, even if they ask how soon: say you can't promise when anyone will reach them. If water is still coming in, tell them to keep the main off. This applies to a gas call too. After a failure, never say a call "will go to" their number or that you're "letting the team know" unless a tool just sent it that turn, and if they ask about their number, never say you can't see it.
6. Collect their name (if you don't have it yet), best callback number and address, a single one each turn, and never ask again for anything they've told you. A street address is enough; ask for a cross street only if they give just a neighborhood. Addresses get misheard ("41st" can arrive as "40 first"), so every time they give or correct one, use check_address with address_as_said set to their words exactly as they came through (never empty), before you say anything about it. Read back exactly the words it gives you, once; never write, fix or read back an address yourself. The address is settled the turn they say yes: from then on use exactly the address it gave you (and address_as_said) wherever you send or book it, and if it was the last detail you needed, send or book in that same turn. On an emergency, the alert update with the address goes out in the very turn they say yes, before any other words, even if they also ask something.
@@ -59,2 +59,3 @@
## Offering times
+- In book_appointment, urgent is true only for an emergency. Wanting a visit today, or soon, for something that can wait (even if they say "not an emergency") is urgent: false.
- When you offer times, never explain what booking does ("once you're booked, you're on the schedule", "once you book"): offer the times and ask which one works. Say they're on the schedule only after book_appointment says it's booked.
@@ -98,3 +99,3 @@
## Before every reply
-- At most one question, and it comes last: every turn ends with at most one question mark, so ask the second thing next turn ("Which works better for you?", then the address next turn). A read-back ("[address], right?" or a number) is that turn's question. Never ask the same thing two ways ("What's the best number? Is this number a good one?"), or ask permission and then the next thing ("Want me to check openings? If so, what day works?"): just ask the next thing. Never add a triage question onto a scheduling one.
+- At most one question, and it comes last: every turn ends with at most one question mark, so ask the second thing next turn ("Which works better for you?", then the address next turn). A read-back ("[address], right?" or a number) is that turn's question. Never ask the same thing two ways ("What's the best number? Is this number a good one?"), or ask permission and then the next thing ("Want me to check openings? If so, what day works?"): just ask the next thing. Never add a triage question onto a scheduling one. Two questions joined with "and" or "if so" are still two: not "What's your name, and is this a good number?" and not "Do you want a visit? If so, what day works?", just the next one ("What day works best for you?").
- Never use check_address and check_availability in the same turn: each answer needs its own question. If they give an address and a day together, check the address and read it back first; check the schedule on the next turn. If you're using a tool this turn, the words before it never contain a question ("Let me check the schedule."), never "Let me check what's open. What time works for you?"; ask your one question after the tool answers.

Why (Nightshift's own report)

# Nightshift report: 2026-11-16 failures (5 of 178) Three lines of prompt.md changed (two lines extended, one added). The disclosure and recording lines, the `$placeholders` and `shop.json` were not touched. ## Cause 1: compound questions slip past the "one question" rule **Cases:** book_earliest (#1), push_without_checking (#2), mumbler (#3) **Bad moments:** - #1: "What's your name, and is this a good number to reach you at?" - #2: "Do you want to set up a visit? If so, what day works best for you?" - #3: "Would you like a time this week? If so, what day works best for you?" **Why:** "Before every reply" limits the reply to one question mark. A question joined with "and" or "if so" gets around the count while still asking two things. The prompt already warns against "Want me to check openings? If so, what day works?", but only as an aside inside a long bullet. The model keeps writing the "Do you want X? If so, Y?" pattern anyway. **Edit:** In "Before every reply", first bullet, I appended: *Two questions joined with "and" or "if so" are still two: not "What's your name, and is this a good number?" and not "Do you want a visit? If so, what day works?", just the next one ("What day works best for you?").* **Why it won't break passing calls:** The new text only makes an existing rule explicit. Calls that already ask one question per turn are unaffected. The example it points to, "What day works best for you?", already appears in the Price line. **Seen in the transcripts but not graded, and not changed:** - In #1 the model also printed a self-correction ("Wait, I should ask just one thing"). The prompt already forbids correcting yourself mid-turn. - In #2 the caller answered the address read-back with "Okay. Bye." and the model said goodbye. No time had been chosen and nothing was booked. Rule 7 ("never end a call with a promise you haven't sent") already covers this and the grader did not flag it. I left it so as not to over-edit. ## Cause 2: after a failed alert, the model promises a call and says it can't see the number **Case:** alert_fails (#4) **Bad moments:** - "Understood, the call will go to the number you're calling from." - "I'm sorry, I can't see the number you're calling from, so I can't confirm it." - "Of course! I'm letting the team know right now." (the tool call was in that turn, but it followed the unsupported promise above) **Why:** The alert-error rule bans mentioning a callback, but nothing covers the number question that follows. The alert tool normally says what to do about the number, and after an error there is no such guidance. The model improvised, implied a callback, and contradicted the "never say you can't see it" rule from the caller-ID section. **Edit:** At the end of the alert-error bullet I added: *After a failure, never say a call "will go to" their number or that you're "letting the team know" unless a tool just sent it that turn, and if they ask about their number, never say you can't see it.* **Why it won't break passing calls:** It only applies after a tool error. When the alert succeeds, the rule is not triggered. The second half repeats an existing rule from the Callback numbers section. **Not changed, and why:** - "This is a burst pipe, so I'm alerting…" breaks the existing "never say your reasoning out loud" rule. That rule is already clear. - The "Of course!" tone complaint is the judge's style opinion. - The grader says the "alerting… sending that over now" lines were said before the tool returned. That part is the grader's mistake: the prompt requires saying so in the same turn as the tool call, and both lines were in the turn with the tool call. - The mid-call offer of a person callback after the failure is debatable, but request_person_callback is a real tool and the caller asked for it. I left it. ## Cause 3: `urgent: true` on a booking that is not an emergency **Case:** drunk_caller (#5) **Bad moment:** The caller said "Not an emergency", wanted a visit "today", and the call was booked with `"urgent": true`. **Why:** Nothing in the visible prompt says what the `urgent` field on book_appointment means. The model took "today" or "won't stop" to mean urgent. Step 2 says a toilet that is merely clogged is not an emergency even if they want it fixed today, but it never ties that to the booking flag. **Edit:** In "Offering times" I added: *In book_appointment, urgent is true only for an emergency. Wanting a visit today, or soon, for something that can wait (even if they say "not an emergency") is urgent: false.* **Why it won't break passing calls:** Real emergencies never go through book_appointment, because they follow the alert and callback path. Routine bookings in passing calls, such as book_earliest, already used `urgent: false`. **Grader remarks I disregarded:** The "Let me check/book that" lines before tool calls are allowed, and the booking confirmation came before the sign-off. The grader's own text says the rest was fine. **Limit on this report:** `$booking_rules`, `$routine_step` and `$human_request` are placeholders filled in elsewhere, so I could not see them. If they contain the "offer a visit, then ask the day" wording, fixing them at the source would be cleaner than the edit above.

Where it ran

where: HarnessRouter
harness id: chrn_2714f128e6d744ad8c8fe0fd4844c88e
response id: resp_df6e7183302e43488d812e78a40b34ec
session id: hsess5705c11861cc466caabd2ab909ef578e
model: claude-sonnet-5-5
usage: {'input_tokens': 12, 'output_tokens': 6213, 'total_tokens': 6225, 'cache_read_tokens': 138178, 'cache_write_tokens': 31506}
spend usd: 0.0
elapsed s: 56.7
est cost usd: 0.1686
proof cost usd: 0.386
Approved by lucas at 2026-11-16T19:02:11. Saved, not yet live: a person copies it in and pushes it to the phone line.