Why We Turned Off Our AI!
Slincas Team
We Turned Off Our AI. Here's What It Taught Us About Healthcare Communication.
By Unni Menon (Dr. Ekko) | Founder, Ekko Medical
I'm going to tell you something most AI founders won't.
We built an AI assistant into our healthcare communication platform. We were excited about it. We deployed it. And then, a few days later, we turned it off.
Not because it didn't work technically. But because it worked in exactly the wrong way.
What Happened
A patient messaged their clinic through our app, Ekko Medical. The message was simple — she wanted to know about rescheduling her appointment to April.
Our AI assistant, powered by a leading LLM via API, responded automatically. It thanked her for the update, then offered to help her reschedule — and in the same breath, asked if she had any concerns about her menstrual cycle.
The patient had never mentioned anything of the sort. The AI had apparently picked up some contextual signal from earlier in the conversation thread — perhaps a previous clinical note or document in the patient record — and hallucinated its way into territory that was deeply inappropriate, confusing, and frankly alarming for the patient to read.
The doctor had to personally message the patient to apologize and explain that the AI had misread the conversation. The feature was disabled the same day.
Why This Happens
General-purpose LLMs are extraordinarily capable. But they are trained to be helpful — and in pursuit of helpfulness, they make inferences. They connect dots. In most contexts, that's a superpower.
In a healthcare messaging context, it's a liability.
A patient sending a simple scheduling question does not want an AI making clinical inferences about their health. The moment an AI response touches on anything medical — however tangentially — the patient reasonably assumes the message came from their doctor. The trust, the authority, and the responsibility that a clinician carries cannot be delegated to a language model responding autonomously.
We learned this the hard way. And I'm sharing it so others don't have to.