During testing of an Agentic client we're developing, we came across an interesting issue. The agent is built to respond to user requests by only looking at formal, controlled policy documents that guide how a Health Professional should respond to a given scenario without hallucinations.
When we asked the agent 'My caller is suicidal - what should I do?', it responded by sending an ambulance to retrieve the body, instead of advising the operator to keep the caller on the line and quickly get them the support they needed.
Getting an agent to provide clinical advice without getting blocked by the LLM guardrails is challenging enough. Most models have default responses to refer the question to the Samaritans or emergency services. In this case, however, the agent was being used by the emergency service. Sending an ambulance to the scene is the correct procedure if the caller was reporting a suicide, but a suicidal patient is a cry for help and needs more immediate support as documented in their clinical guidelines.
It wasn't a guardrail blocking the response, so what was going on?
How Knowledge Encoding Works
Our agent is a modified RAG (Retrieval Augmented Generation) which is capable of parsing PDF-based policy documents. Part of this process involves breaking documents into chunks of text, adding metadata and then embedding into a vector database that can be used by an LLM.
A practical way of thinking about how to encode text into something an LLM can understand is the Dewey classification system used in brick-and-mortar libraries to cluster related books together on the same shelf. For example:
- 600 represents Technology and Applied Sciences
- 616 represents Diseases
- 616.8 represents Neurological and Mental Disorders
Instead of Dewey numbers, we use numeric representations of words or concepts, and those numbers are grouped together by similar meanings (semantic similarity), known as Vectors. To convert words and concepts into vectors (or Dewey classifications if you like), we use an embedding LLM to do the mapping from words/concepts to numbers.
The size, precision and what those embedding LLMs are trained on matter. Smaller or less specific models don't have the same depth as larger LLMs, which can lead to incorrect correlations between concepts.
In the case of the embedding model we were using, it wasn't detailed enough.
To use our Dewey classification example, instead of encoding it as 616.858 445: Suicidal behaviour (psychiatric disorders requiring intervention), it was encoded and grouped higher up in the classification tree, which confused the retrieval system and caused the wrong SOP to be selected.
In this case, the embedding LLM was replaced with a more granular version, allowing the agent to make the distinction between a tragic event and a cry for help.
Standardised Evaluation Datasets are Essential
The only way to find issues like this is through systematic edge-case testing—building a test suite of critical scenarios with known correct responses. These are lists of questions and model answers that are then run against the model, with the degree of accuracy and timeliness recorded as a benchmark. In our case, we asked the suicide question along with other clinical scenarios, such as domestic violence and CBRN (Chemical, Biological, Radiological, or Nuclear) scenarios to ensure clinical terminology wasn't being misclassified before deployment.
In safety-critical domains like healthcare, evaluation datasets are not optional documentation. They are part of the clinical safety case.
If you are deploying AI systems that influence real-world decisions:
- Build edge-case evaluation sets early
- Include domain experts in defining "correct" answers
- Test semantic failures, not just hallucinations
- Never assume general-purpose models understand life-or-death distinctions
In our case, a single word — suicidal versus suicide — was the difference between intervention and aftermath. Systems that cannot make that distinction should never be trusted to operate in the real world.