Compliance 7 min read
How to Run an AI Pilot in Your Practice Without Losing Control
Most AI pilots fail not because the technology doesn't work, but because the pilot wasn't structured to produce a decision. Here's a 90-day framework.
DS
Dr. Sajad Zalzala
2026-04-10
The worst outcome of an AI pilot isn't that it fails. It's that it runs forever without producing a clear yes-or-no decision. Here's how to structure a pilot that actually tells you something.
The 90-Day Framework
- •Days 1-14: Setup
- •Define exactly what you're measuring (time saved per encounter, note quality score, claim denial rate)
- •Baseline the metrics with 2 weeks of pre-pilot data
- •Train 2-3 champion users — not the whole staff
- •Set a kill criteria: what result would make you stop the pilot early?
- •Days 15-60: Run
- •Champions use the tool on every eligible encounter
- •Weekly 15-minute check-ins: what's working, what's friction
- •Collect both quantitative data (time, accuracy) and qualitative feedback (ease of use, trust)
- •No changes to the tool configuration during this period — let it run clean
- •Days 61-75: Measure
- •Compare pilot metrics to baseline
- •Calculate actual cost per encounter (software + review time + integration overhead)
- •Survey all users: would you keep using this tool if given the choice?
- •Check compliance: any incidents, near-misses, or documentation concerns?
- •Days 76-90: Decide
- •Present results to decision makers with a clear recommendation
- •Three possible outcomes: expand, modify and re-pilot, or stop
- •If expanding: create the implementation plan, training schedule, and vendor contract terms
- •If stopping: document why, so you don't re-evaluate the same tool in 6 months
The Two Mistakes That Kill Pilots
1. No baseline. If you don't measure before, you can't prove improvement after. "It feels faster" is not data.
2. Scope creep. The pilot was supposed to test AI scribes for 3 providers. By week 4, leadership wants to add coding assistance, inbox triage, and patient-facing chatbots. Each addition invalidates your measurement framework.