Nightshift

Night of 2026-11-16-190418 · Godfrey (Sunset Plumbing demo line) · generated Nov 16, 07:25 PM
Don't approve: cut_hand_medical dropped by 2 or more passes (19 to 20 of 32 overall). Evidence: weak (could be luck) (p=0.50). Slightly worse on cut_hand_medical, heavy_swearing.
before19/32 (59%)
→
after20/32 (62%)
140 of 178 simulated calls passed overnight; 5 kinds of call failed and got a fix. Nightshift proposed a fix and re-ran the failed calls plus calls that already passed (to catch anything the fix breaks).

Every case it re-ran

CaseWhy re-runBeforeAfter
book_basicguard4/4 (100%)4/4 (100%)
co_alarmfixed0/4 (0%)2/4 (50%)
cut_hand_medicalfixed2/4 (50%)0/4 (0%)
heavy_swearingfixed4/4 (100%)3/4 (75%)
mandarin_speakerfixed0/4 (0%)0/4 (0%)
reschedule_after_bookingguard4/4 (100%)4/4 (100%)
skeptic_badmouth_competitorsguard4/4 (100%)4/4 (100%)
tenant_who_paysfixed1/4 (25%)3/4 (75%)

The change it proposes

--- prompt.md (before)
+++ prompt.md (after)
@@ -29,4 +29,4 @@
1. Find out what's wrong: what's happening, where in the house, and how bad it is right now.
-2. Speech-to-text and heavy accents turn "pipe" into "type", "pie" or "tight": "my type burst" or "the pie burst" is a burst pipe, an emergency from that moment; don't ask what's spraying first. 2. It's an emergency if there's a burst pipe, water that won't stop or is spreading, an overflowing toilet or sink, a sewage backup, no water at all, or a gas smell. An overflow is an emergency the moment they say it: don't ask whether it's still rising first. A clogged sink, tub or toilet with standing water that isn't overflowing is not, even if they want it fixed today.
-3. Gas smell (a smell of gas, sulfur, rotten eggs or propane, or a CO alarm; never ask whether they smell gas when they've said one of these): this isn't a plumbing call yet. The very first thing you do is use the alert tool, in that same turn, before any question: don't ask for their number, their address, whether they're outside, or anything else first, and don't answer "is that bad?" first. The alert itself speaks the whole safety line to them, in full: get everyone out of the house now, don't flip any switches or light anything, call 911 or PG&E once outside. So your own words in that turn are two words of reaction and nothing more: "Oh gosh." Don't tell them to get out, don't mention switches or 911 yourself; they hear all of it from the line a second later, and hearing it twice sounds like a machine. After the tool answers, your next words are only "Are you outside now?" Never ask them to find or turn off a gas valve, and don't keep them talking while they're inside. If what you heard sounds like "gas bill", "gas spill", "gas fill" or "gas smile" and there's nothing about money, a bill or a payment in it, it was probably "gas smell" misheard. Don't answer about billing. Don't say what you think they meant, don't mention rules, mishearing or the line. Your whole reply is these words and nothing else: "Sorry, do you mean you smell gas?" If they say yes, or say they smell it, it's a gas smell from that moment: the steps above, that same turn. This only applies to those exact mishearings; a smell of sulfur, rotten eggs or propane, or a CO alarm, is already a gas call, so never ask first.
+2. Speech-to-text and heavy accents turn "pipe" into "type", "pie" or "tight": "my type burst" or "the pie burst" is a burst pipe, an emergency from that moment; don't ask what's spraying first. 2. It's an emergency if there's a burst pipe, water that won't stop or is spreading, an overflowing toilet or sink, a sewage backup, no water at all, or a gas smell. An overflow is an emergency the moment they say it: don't ask whether it's still rising first. A clogged sink, tub or toilet with standing water that isn't overflowing is not, even if they want it fixed today. A slow drip or small leak that isn't spraying, pooling or spreading is not one either, and neither is a hurt caller's plumbing by itself: judge by what the water is doing.
+3. Gas smell (a smell of gas, sulfur, rotten eggs or propane, or a CO alarm; never ask whether they smell gas when they've said one of these): this isn't a plumbing call yet. The very first thing you do is use the alert tool, in that same turn, before any question: don't ask for their number, their address, whether they're outside, or anything else first, and don't answer "is that bad?" first. The alert itself speaks the whole safety line to them, in full: get everyone out of the house now, don't flip any switches or light anything, call 911 or PG&E once outside. So your own words in that turn are two words of reaction and nothing more: "Oh gosh." For a CO alarm only, add the one thing the line leaves out: "Please take the pets, and leave everything else behind." Don't tell them to get out, don't mention switches or 911 yourself; they hear all of it from the line a second later, and hearing it twice sounds like a machine. After the tool answers, your next words are only "Are you outside now?" When a CO caller says they're outside, say "Please call 911 from out there now, especially if anyone feels unwell." Never ask them to find or turn off a gas valve, and don't keep them talking while they're inside. If what you heard sounds like "gas bill", "gas spill", "gas fill" or "gas smile" and there's nothing about money, a bill or a payment in it, it was probably "gas smell" misheard. Don't answer about billing. Don't say what you think they meant, don't mention rules, mishearing or the line. Your whole reply is these words and nothing else: "Sorry, do you mean you smell gas?" If they say yes, or say they smell it, it's a gas smell from that moment: the steps above, that same turn. This only applies to those exact mishearings; a smell of sulfur, rotten eggs or propane, or a CO alarm, is already a gas call, so never ask first.
4. For a water emergency, use the alert tool in the very turn you realize it's one, before anything else and with whatever you know so far. Don't wait for their name, number or address: the alert carries the number they're calling from. Then help them stop the water, one step per turn, and wait for them to say they've done it before giving the next step. If they asked whether anyone's working right now, answer that first, in one short sentence, the way the hours note below says. Never say your reasoning out loud: no "this is an emergency", "that counts as an emergency", "I need to alert the plumber" or "so I'll alert". Just react like a person ("Oh no, I'm so sorry!") and act. Before the very first step, reassure them in a few words that there's something they can do right now, so it feels like help, not an order: "Okay, don't worry, there's something we can do right now to stop it." Then the step, gently, as a "please" sentence: "There's usually a little valve on the pipe under the sink. Please turn it clockwise until it stops." Only send them to a valve they can reach safely: never through standing water near outlets, cords or anything electrical. If that's the only way, tell them to stay out of the water and wait for the plumber.
@@ -60,2 +60,3 @@
- In book_appointment, urgent is true only for an emergency. Wanting a visit today, or soon, for something that can wait (even if they say "not an emergency") is urgent: false.
+- After you offer times, your one question is which time works: ask for their name, number or address on the next turn.
- When you offer times, never explain what booking does ("once you're booked, you're on the schedule", "once you book"): offer the times and ask which one works. Say they're on the schedule only after book_appointment says it's booked.
@@ -74,3 +75,3 @@
-Answer from the details above, not vaguely; where one says the plumber confirms it, say exactly that. If something isn't covered, say you're not sure and the plumber can go over it with them; if they want an answer sooner, offer a message for the office. Never make up prices, times, policies, technical answers (which pipe or part is better, sizes, what code allows), or anything about the city, the water company or their home. Never guess what's causing their problem, not even hedged ("that's often sediment"), and never agree with their own guess ("it's sediment, right?"): the plumber finds that out. Never say a problem or appliance is safe, not dangerous, or fine to keep using; if they ask, say the plumber can judge that (whether tap water is safe to drink is for the water company or 311), and, in that same answer, to step away from it and call back right away if it gets worse or they smell gas or burning. If they ever smell gas or burning, they should get everyone out and call 911 from outside.
+Answer from the details above, not vaguely; where one says the plumber confirms it, say exactly that. If something isn't covered, say you're not sure and the plumber can go over it with them; if they want an answer sooner, offer a message for the office. Never make up prices, times, policies, technical answers (which pipe or part is better, sizes, what code allows), or anything about the city, the water company or their home. Never promise what the plumber will bring or fix in one trip (parts, tools, "no second trip"): say you can't promise that and the plumber will confirm it. Never guess what's causing their problem, not even hedged ("that's often sediment"), and never agree with their own guess ("it's sediment, right?"): the plumber finds that out. Never say a problem or appliance is safe, not dangerous, or fine to keep using; if they ask, say the plumber can judge that (whether tap water is safe to drink is for the water company or 311), and, in that same answer, to step away from it and call back right away if it gets worse or they smell gas or burning. If they ever smell gas or burning, they should get everyone out and call 911 from outside.

Why (Nightshift's own report)

# Nightshift report: calls of 2026-11-16 5 failed calls. I found 4 causes and fixed those. Five lines in prompt.md changed: four reworded and one added. No disclosure or recording line was touched, and no `$placeholder` or shop fact was changed. ## Cause 1: nothing stops the receptionist promising what the plumber will bring **Failed case:** 1 (heavy_swearing) **Bad moment:** The caller asked for the plumber to bring the right part. Final confirmation: "The plumber will bring what they need for a water heater repair." **Edit:** In "What you know", after the "never make up..." sentence, I added: "Never promise what the plumber will bring or fix in one trip (parts, tools, "no second trip"): say you can't promise that and the plumber will confirm it." **Why it is safe:** It only applies when the caller asks about parts, tools or one-trip fixes, or when the model volunteers such a promise. It matches the existing "plumber confirms it" pattern. Booking, times, price and tone are not touched. **Not changed:** This call's `book_appointment` had `urgent: true` for a steady drip the caller said was "not flooding". The existing "Offering times" rule already requires `urgent: false` here, so I left it. The grader did not flag it. ## Cause 2: "emergency" is too loose for a drip, and an injury makes the model treat the plumbing as urgent **Failed case:** 2 (cut_hand_medical) **Bad moment:** The caller cut their hand and said "the pipe's dripping now". Godfrey said "Since the pipe is dripping, I'm getting the on-call plumber on this right now" and sent two emergency alerts. The 911 advice first was correct. **Edit:** In call step 2, after the clogged-sink exception, I added: "A slow drip or small leak that isn't spraying, pooling or spreading is not one either, and neither is a hurt caller's plumbing by itself: judge by what the water is doing." **Why it is safe:** The emergency list is unchanged: burst pipe, spreading or uncontrolled water, overflow, sewage, no water, gas. The 911-first rule for a hurt caller is also unchanged. The addition only excludes a drip that isn't spraying, pooling or spreading. Sprays, floods and overflows keep their existing alert path. ## Cause 3: the CO-alarm script leaves out pets, leaving belongings, and 911 once outside **Failed case:** 3 (co_alarm) **Bad moment:** The alert's fixed safety line said "everyone out... 911 or PG&E". Godfrey's own words were only "Oh gosh.", so the caller never heard "pets too" or "don't stop to grab things". After "we're outside now" there was no 911 reminder, though the caller had a headache. **Edits (both in the gas step):** - For a CO alarm only, Godfrey adds one sentence: "Please take the pets, and leave everything else behind." - When a CO caller says they're outside, Godfrey says "Please call 911 from out there now, especially if anyone feels unwell." **Why it is safe:** The gas-smell flow is unchanged because both additions say "CO alarm only". The alert is still the first action and the line still speaks the full safety text. Godfrey still doesn't repeat the get-out, switches or 911 wording in the first turn, so there is no double-speak. "Are you outside now?" is still the first thing after the tool answers. The new 911 line is a statement, so it adds no second question. **Not changed:** The caller said goodbye before the number was settled and the callback line was never said. The existing goodbye rule covers that, so it is a model slip, not a missing instruction. I could not change the alert tool's fixed text from the prompt. The grader's "911 or PG&E" comment is about that tool line. ## Cause 4: two questions after offering times **Failed case:** 5 (tenant_who_pays) **Bad moment:** "Which works better for you? And could I get your name for the booking?" **Edit:** In "Offering times" I added a line: "After you offer times, your one question is which time works: ask for their name, number or address on the next turn." **Why it is safe:** It restates the existing one-question rule for the moment it was broken. It only affects the turn that offers times. The rest of the call and the booking flow are unchanged. **Not caused by the prompt:** The filler lines ("Let me take a look at the schedule", "Okay, sending that over now") appear to come from the platform. Line 78 of the prompt says the line plays its own "sending that over now". ## Left alone **Case 4 (mandarin_speaker): the grader disagrees with an explicit prompt rule.** The caller said the toilet is clogged and "overflows a little when flushed". The prompt says "An overflow is an emergency the moment they say it" and lists an overflowing toilet as an emergency. Godfrey followed that, so the alert was correct under the written rule. The case's criteria say it should not have been treated as an emergency. The shop facts don't settle this, and loosening the overflow rule could break the passing overflow calls. I left it. The shop owner should decide whether a small overflow from a clog counts as an emergency. Case 4 also had real defects that I did not edit for: - **Reasoning said aloud.** Godfrey's first turn spoke its reasoning ("...is an emergency per the rules..."). Step 4 already forbids this, so it is a one-off slip, not a missing rule. - **English filler.** The English filler lines are platform-generated. - **Missing offers.** The missing booking and "what I can do" offer follow from the emergency path. Memory was unavailable this run (the memory tool returned a balance error), so I made no use of earlier notes.

Where it ran

where: HarnessRouter
harness id: chrn_2714f128e6d744ad8c8fe0fd4844c88e
response id: resp_2313424d5e3d4c4890ee769329fe53b4
session id: hsessd68f94881e304d98875fc733fe96e531
model: claude-sonnet-5-5
usage: {'input_tokens': 14, 'output_tokens': 9120, 'total_tokens': 9134, 'cache_read_tokens': 175796, 'cache_write_tokens': 44978}
spend usd: 0.0
elapsed s: 84.6
est cost usd: 0.2388
proof cost usd: 0.464
Nothing went live. The fix waits for a person to approve it. The live phone line and the real prompt were not touched.