project writeup · october 2026 · 6 min read
Poscere: AI patients where code, not the model, decides what’s true
A telehealth virtual-patient simulator for health-professions students. They interview an AI patient while a chart fills in beside the call, then get rubric feedback where every score is tied to a quote from the transcript. The interesting part is everything we built to keep the language model from making things up.
It covers 63 learner roles in 10 groups, from medicine and nursing to dietetics and pharmacy. All 81 seed patients are defined as data with no patient-specific code, and drafting a new one from a paragraph takes about 80 seconds. Care Journeys scores 19 TeamSTEPPS-aligned teamwork behaviors.
Why practice is the bottleneck
Health-professions students get few chances to practice a patient conversation and get specific feedback on it. Standardized-patient time is expensive (one UK school’s 15-station final-year OSCE cost £355 per student) and skill comes from exactly what’s rationed: repetition with immediate feedback. Students almost never get to practice handing a patient to another profession, even though handoff failures are a well-documented patient-safety problem.
LLM patients are the obvious fix, and the published studies show the obvious failure: in one, where the script didn’t cover a question, 56.4% of answers used invented information. A chatbot with the whole case in its system prompt can reveal anything at any time, the chart has to be reconstructed by asking a model what was said, and feedback becomes a model’s impression of a transcript.
The design: split “what’s true” from “how it sounds”
A patient is a JSON document: persona, facts, exams and orders with fixed results, documents, objectives, an answer key and a rubric. Program code sits between the model and everything that gets recorded or graded.
- Code decides disclosure. Each fact has a tier: volunteered, when asked, needs trust, or guarded. The tier, the rapport the student has built, and the difficulty level decide when it’s released. The model writes the patient’s sentences from released facts only. It never writes the chart or an order result.
- Every reply is audited. After each reply a second, faster model compares it with every fact in the case. A released fact the patient didn’t actually say goes back to locked. A clinical claim outside the case is shown to the student as “Outside the case record” and is never filed.
- Feedback is a record, not an impression. Findings elicited and missed come from the disclosure ledger. The scoring model must quote the transcript for each rubric domain, and code checks that each quote exists. A separate running score per turn is plain arithmetic with no model call.
- Orders can’t be invented. Ask for a brain MRI the case doesn’t contain and you get a refusal, not a plausible-looking result.
Models are picked by job: Claude Haiku 4.5 for the per-turn gatekeeper and audit (it runs every turn, so latency matters), Sonnet 4.5 for the patient’s voice, and Sonnet 5 for drafting, consistency review, scoring and synthesis. Structured output comes from forcing a single tool call whose schema is the JSON we want, and a response cut off by the token limit is an error, not something to parse.
Faculty write patients in a paragraph
A staged generator drafts a patient in about 80 seconds: one call fixes the clinical story and every number, four calls write the persona, the two halves of the history and the chart in parallel, and a last call links the learning objectives. Rule checks plus a model review then list contradictions with proposed fixes, and nothing is applied until faculty accept it. Faculty can also plant deliberate mismatches between what the patient says and what the record shows, and whether the student notices is scored.
Care Journeys: one patient, several professions
A student takes one encounter (say, the hospitalist) and AI colleagues play the others, interviewing the same patient through the same disclosure rules. Everyone writes into a shared patient state: append-only entries, each storing who recorded it, in which role and encounter, at what simulated time, with what status and the quoted evidence. Handoffs are written as SBAR and judged against 19 teamwork behaviors, each credited only when the quoted words are found. A huddle works through whatever is still disputed or unanswered, and the care plan drops any action that cites no real entry.
What it isn’t yet
This is a hackathon prototype, and none of its educational claims has been tested:
- No clinician has reviewed the patients. All 81 are stored as unreviewed. The consistency check finds contradictions inside a case; it doesn’t establish that the medicine is right.
- Scores aren’t validated. Published work shows LLM graders agree with experts well on yes/no items and much worse on multi-level communication scales, which is what this rubric is. Until they’re compared with faculty raters, Poscere is a practice tool, not an assessment tool.
- The patient can still say something outside the case. The audit flags it after the reply has been shown and spoken, and the checker can miss a claim. It never reaches the chart or the feedback findings.
- It hasn’t been used by learners, and it runs on one server with SQLite on a temporary hackathon AWS account.
What would settle the scoring question is a validation study: faculty raters, blind to the AI score, rating the same transcripts, with a per-domain intraclass correlation target set in advance.
The architecture docs are in the repo.