Background
In late August 2023, I participated in a a 3-day internal hackathon at Flatiron health, an oncology data company. Though I worked on the Real World Evidence side of the businesses I signed up to contribute to a hack pitch on the company’s EMR side of the business. The pitch was focused on utilizing large language models (LLMs) to automatically generate a visit note from passively collected audio from the visit i.e. using the natural audio from the visit and not using a dictation pattern.
The Problem
Clinicians face a tedious process in documenting patient encounters through visit notes in the company’s EHR system, OncoEMR. This manual documentation takes time and attention away from direct patient care. Clinicians often delay completing notes until later, sometimes days after seeing patients.
My role
I worked on a cross-functional team as the sole product designer and collaborated with Clinical, Engineering and Data Science on this hack. Additionally, since the team was lacking a Product Manager, I took responsibility for team ceremonies – bringing the team together to start collaborating in a hybrid environment.
The Solution
Our team prototype an AI-powered solution to:
- Passively transcribe audio from patient-provider conversations
- Using LLMs like GPT-3 to generate a draft visit note summarizing the encounter for easy review and editing
- Format notes based on preferences (e.g. SOAP format)
The Approach
- To overcome our cold start-problem, I partnered with clinical to simulate clinical scenarios and recorded conversations to create test transcripts. Using a messy transcript with natural speech was critical since it would help pressure test the technology!
- We generated 3 sample scenarios and experimented with various LLM models (GPT-3, Claude 2, etc.) to generate visit note drafts
- Data Science partnered with Clinical to evaluated output quality manually. From a UX perspective, I prototyped a clinician review workflows to verify and edit AI-generated notes
Key Learnings
- The quality of training data impacts the performance of language models
- Prompt engineering is crucial for shaping tone, content and accuracy
- Ongoing user feedback and metrics are needed to enhance system performance over time
- Regulatory considerations are required around clinician review and sign-off of AI-generated content
Impact
The prototype demonstrated the potential for LLMs to reduce documentation burden for clinicians, enabling them to spend more time delivering patient care. Using a noisy transcript with simulated interruptions further helped build confidence in this area. Our hack, won two major categories at the Hackthon: Patient Advocacy’ and ‘Powered by AI Showcase’.
Based on findings from the hackathon, the business was able to prioritize continued investment in this area including strategic partnerships with medical LLM providers.