What an AI receptionist must never say

A clinic receptionist answers the phone, checks the doctors’ schedule, books visits and calls in staff when something needs the care team. Part of the job is knowing which questions not to answer yourself. If a patient describes their symptoms, a receptionist should not reply “That sounds like an infection, you could try taking…”, however much they want to help.

We build an AI receptionist for clinics. It answers calls in Thai and replies on LINE. Its job is to give information about opening hours, services and how to prepare for a visit, and to take appointment requests. Questions that need a clinical opinion are picked up by the clinic’s own team.

Along the way we found that writing the prohibitions into the prompt was not enough. This note covers the three things it must not do, and how we added checks to the system.

Three things it must not answer

1. No diagnosis. That includes guesses like “it might be X”, because a patient can still take that as a medical opinion, even when the wording sounds tentative.
2. No medication advice. Not how much to take, not how often, not what to do after a missed dose.
3. No promised results. Nothing like “this will definitely fix it” or “guaranteed results”.

As a list this looks straightforward, but real conversations rarely arrive in neat items. A patient describes their pain and then asks a follow-up, and the model may try so hard to help that it answers beyond its role. So we have to look at both what a reply says and the moment it should be handed to a person.

Why the prompt alone is not enough

Our first version also put these rules in the system prompt, and most of the time the model followed them. The problem was the times it slipped, which is not something this job can let through.

To fix it, we started by adding instructions and repeating the existing rules. But the longer the prompt grew, the more fixing one behaviour affected another, and rules we had repeated several times still slipped now and then. So we began moving some of the checks out into code, where we can state exactly what is checked, at which point, and write tests that cover it.

We split the work: the model handles language, tone and how things are explained, while the prohibitions that must be checked every time are backed by code as well. A reply has to pass these checks before it goes out. That way we don’t depend on the model following the prompt alone, though the checks themselves still need testing and fixing whenever we find a case they miss.

What the code checks

Every reply passes through a set of checks after the model drafts it and before the system sends the message or the audio.

The outbound detector looks for diagnosis, medical advice and claims about treatment results in the raw reply. It runs separately from the model and from the hand-off logic, so we can see more clearly which part made a mistake.

The medication filter removes any sentence that explains how to take a medicine, then adjusts the reply so the patient asks a doctor or a pharmacist instead. We keep the filter narrow enough that basic self-care advice the clinic has already approved still gets through. If it blocks too broadly, the assistant is left with very little it can answer, including things it should.

There are checks on the way in as well. We set the system to hand off immediately when it detects acute symptoms that match defined conditions, such as swelling that spreads quickly, a high fever or bleeding that won’t stop, even if the model didn’t flag them. Here we chose to hand off first whenever a risk signal appears, because missing something that should have been handed off does more harm. If the clinic is closed, the system tells the patient to go to a hospital emergency room instead of waiting for the clinic to open in the morning.

Another thing to handle is text that a person reads without trouble but a pattern check may miss: invisible zero-width characters, lookalike digits, or words that mix characters from several scripts. We have to deal with those forms before checking the content.

Handing off to staff

When the assistant can’t answer, the conversation still has to move forward. The patient should know who the matter is going to and roughly when they will hear back.

When a hand-off starts, the system opens a case and posts a card with the patient’s question to the clinic staff’s LINE group. Staff can pause the AI and take over the chat in LINE themselves. Opening a case and a staff member picking it up are separate steps, so we have to make sure a reply never says someone has taken the case when the system can only confirm that the hand-off has started.

When staff hand the conversation back to the AI, code handles the resume button before the model is called. We once saw the AI reread the conversation and hand the same issue off again: a person had just handed the work back, and the AI sent it straight back to them. We fixed that loop in the state handling in code, which is closer to the cause of the problem than adding one more instruction to the prompt.

So a hand-off reply should state the next step clearly, with a time frame the system has data to support, instead of ending with “Sorry, I can’t help with that.”

The same fact, written many ways

One of the things that cost us the most time to fix: a value can be correct in one form and slip through when the model writes it another way.

A phone number can be written as plain digits, in groups, in Thai numerals, or read out digit by digit. A price range may use a hyphen or the Thai word for “to”. Times appear both as digits and as Thai phrases like “three in the afternoon” or “seven in the evening”.

081-234-5678
081 234 5678
๐๘๑-๒๓๔-๕๖๗๘
ศูนย์ แปด หนึ่ง สอง สาม สี่ ห้า หก เจ็ด แปด

We had a checker for the phone number the assistant reads back to a caller. It handled numbers read out digit by digit, because that was the form the assistant used the first time. But in one test call, as the call was ending, the model read the number back again as grouped digits and dropped one. Our checker did not cover that form, so the wrong number went straight through. Had that happened in real use, the clinic could have called back the wrong person.

After that, we didn’t just fix the form we had just found. We went back and listed how many ways each kind of value can appear in a reply, then added tests for the forms we expect to see, including the real text that had made the system fail.

This matters for values people will act on, such as phone numbers, prices, dates and times, and also values like dosages in systems whose job is to show them. This receptionist still gives no advice on how to take medication.

A simplified example for phone numbers: convert the different forms to one standard value first, then compare it with the stored record. If they don’t match, use the number from the source record instead of the text the model drafted.

// Collapse every surface form to one canonical value, then compare it
// with the record. The record wins; the model's version never ships.
const canonical = (s: string) =>
  thaiWordsToDigits(thaiNumeralsToAscii(stripInvisible(s)))
    .replace(/\D/g, '');

if (canonical(readBack) !== canonical(record.phone)) {
  reply = withPhone(reply, record.phone);
}

When a kind of value has clear limits, we can take even more off the model. For Thai times, we store the time as digits and let a function in code turn it into the spoken words, with tests that run through the whole day. That deals with a whole group of related mistakes at once, instead of fixing them one case at a time as test calls turn them up.

Check the reply after the filters too

Another thing that can make us misread test results is the log. If it only records the model’s raw reply, we can’t see what the filters changed or removed before sending.

More than once, a log made it look as if the assistant had said something that had in fact been removed before the reply went out. So when we review the system we also look at the reply after the filters, and keep clear whether we are looking at what the model drafted or what the system sent. Even then, a record that a message or audio was sent does not confirm that the patient received, read or heard all of it.

Evaluating quality needs the same care. Using an AI to score examples the assistant helped write risks results that favour the assistant too much. So deterministic checks, which follow rules and give the same result for the same input, come first. Scores from an AI acting as judge are used only once they have been compared against conversations that people have already scored.

The scope has to be clear in the agreement too

Beyond the code, the agreement with the clinic has to state clearly what the assistant does. This one helps with appointments and general information, while medical advice and treatment decisions stay with the clinic’s team. The scope written in the agreement and the way the system behaves should match.

For us, making the assistant answer naturally is only part of the work. The other part is making it stop when a question goes beyond its role, hand off at the right moment, and let the team check what happened at each step. All of that has to be tested against real conversations and fixed whenever we find a case the system still doesn’t handle well.