Rae Woods (00:10):
From Advisory Board, we are bringing you a radio advisory, your weekly download on how to untangle healthcare's most pressing challenges. My name is Rachel Woods. You can call me Rae. Look, no health system is debating whether AI is worthwhile anymore. We're long past that. But let me tell you the real frustration that I hear. We have pilots, we have promising tools, but we're stuck. We can't scale. We're just doing pilot after pilot. And scaling AI beyond one specialty, one site, one workflow, that's where most organizations get stuck.

(00:47):
Today's guest has cracked that problem. Dr. Veena Jones is chief medical informatics officer at Sutter Health, an organization that has worked to scale AI across the clinical journey, before the visit, during the visit, and after, all while maintaining safety, clinician trust, and adoption. She's also a practicing pediatric hospitalist, so she understands firsthand what it means when technology improves the clinician and the patient experience. At Sutter, that means reducing clinician documentation time by up to 30%, driving high adoption across multiple specialties and doing that with safety and scalability in mind. So here to tell the story of Sutter Health is Dr. Veena Jones. Veena, welcome to Radio Advisory.

Dr. Veena Jones (01:37):
Hi, thank you so much for having me.

Rae Woods (01:40):
So I want to get into the story of Sutter Health. And before we get into how you adopted and scaled AI, I want to start with the why. And I want to warn you, I'm not going to be satisfied with an answer of, well, everyone is focused on AI and that's why. What specific pressures were happening at Sutter that made it clear that focusing on isolated specific AI pilots just wasn't going to cut it for the clinicians or the patients at Sutter?

Dr. Veena Jones (02:11):
I don't think we at Sutter choose to do things because everyone else is doing it. We really do try to focus on what are the issues that we are trying to solve? What are the system level strains, experiences that our clinicians are sharing with us? What do our patients need? I would say those are really the pressures that drive us to change what we're doing or to think innovatively.

Rae Woods (02:38):
And by the way, you are already sharing one of the principles that is really central to advisory board's take on AI, which is a problem first approach, not chasing the shiny object. And it sounds like some of the problems were on the patient facing side and on the provider facing side.

Dr. Veena Jones (02:53):
Yeah, healthcare is not perfect. I think there are opportunities across the board. And so the problem first approach, I don't think it should be unique to AI. I would say that is how we've approached digital strategy as an organization for a long, long time and AI as well.

Rae Woods (03:14):
And I'll tell you some of the problems that I hear in the market. It's a lot on administrative burden, clinician burnout. Is that part of the top of the list?

Dr. Veena Jones (03:24):
So that I would say is some of the lowest hanging fruit. When we look at the strains that our system is experiencing, I think there's a lot of administrative burden, whether it's documentation, rising inbox volumes, message volumes that our clinicians are experiencing, just the vast number of inputs that on the clinician side, one has to process in order to make a decision. You've maybe heard the concept of pajama time. So that's the work that our clinicians are doing after hours, after their patient facing time in the office or in the hospital. It's that documentation time being super, super burdensome.

Rae Woods (04:05):
It's one of my least favorite phrases in healthcare, the concept of pajama time. We already ask so much of our nurses, our doctors, our clinicians. I'll name another buzzword though, and that's low hanging fruit. I think most of our listeners would agree that that operational efficiency, that administrative burden is the low hanging fruit. But here's where I think Sutter's story starts to differentiate itself from some of the other stories I hear about AI for the example of administrative burden. You all seem to accept very early on that even successful point solutions wasn't going to be enough. My question is, what convinced you that system-wide scale, not just experimentation, not dabbling with AI, had to be the goal?

Dr. Veena Jones (04:56):
I think it's the pressures that we're talking about. Coming up with a point solution or going all in on something that was only going to help a small number of people in a system as large of ours just quite frankly doesn't feel worth it. I think it's super important to think big, and I think that is how we always were thinking when we approached this. Here's another buzzword, pilotitis. I think we've lived that. I had a conversation earlier today about, "Hey, we've got to start somewhere. What's the pilot plan?" But the bigger conversation is, "Okay, yeah, but what's our scaling plan to bring this into the hands of as many as we can?"

Rae Woods (05:37):
I think people talk about it. They talk about scaling beyond pilots, but actually operationalizing it, that's an entirely different beast. And I will say it's especially a different beast for this generation of generative AI because the tech itself is changing so fast. And it's reasonable for me to hear leaders out there that say, "We're starting with pilots because we're picking the right vendor, because the tech is changing so quickly." And again, that speaks to how bold it is for Sutter to have said it's not just about finding the right tools, it's about building the operating model that would make scale possible for any tool. My question for you is, what is that operating model? What does it look like in practice and where did Sutter start in building that operating model?

Dr. Veena Jones (06:26):
When it comes specifically to AI, I would say we are somewhat unique in how our digital team is structured. In our core infrastructure, we have a center for applied AI. So we have a chief AI officer, Dr. Ashley Beecy, that oversees a team of people who really focus on topics like AI governance, initial evaluation, validation, risk classification. In my role as CMIO, I oversee our team of clinical informaticists and we partner really closely with the Center for Applied AI on helping with tool evaluation and validation, but then also taking it that step further on how do we integrate it into actual clinical workflows? How are we actually going to scale? How are we going to make it useful? How do we train to that? How do we garner adoption? We also have a team called our Digital Academy solely focused on at the elbow support, training, optimization for our clinicians. And I think it's really that partnership across these teams that set us up to scale effectively.

Rae Woods (07:40):
Yeah. I'm hearing you have really clear executive support and sponsorship.

Dr. Veena Jones (07:46):
If you're talking about executive sponsorship, I think that starts way at the top. Our CEO is bought in. A couple of years ago we had organization-wide leadership team sessions talking about, okay, what is this thing called AI that everybody is talking about and how do we want to position ourselves? We have really open dialogue about it and we have for a long time.

(08:09):
So that executive level sponsorship starts way at the top. And then within our digital division, our chief digital officer, Dr. Beecy, our chief AI officer, our VP of applications, support from our chief physician executive, it takes support at every level. It takes sponsorship definitely at the executive level, but then ultimately when you're trying to drive scale, you need everybody to buy in.

Rae Woods (08:35):
So I'm hearing this rich executive sponsorship and support, and then to make sure that you're hitting every level, you also have the coordination through the Center for Applied AI, and then you have this at the elbow support through the Digital Academy. Why was it important to have these things be separate hubs, and then how do the hubs interact and work together?

Dr. Veena Jones (08:58):
When we think about all of the digital work that happens in our organization, we're keeping an EHR running, we have so much digital infrastructure that runs our entire system. Digital Academy is at the elbow training new clinicians, new physicians when they join the organization need to learn how to use the EHR. But then when we think about AI, I think there's some added layers that we didn't really have a foundation for, and that's where the Center for Applied AI was really born and why Dr. Beecy joined us is to set up the frameworks around governance specifically when it comes to introducing different AI solutions.

(09:40):
How do we validate them? How do we evaluate them? How do we classify their risk and therefore drive the way that we adopt them? I think that's been what's unique to AI implementation. And I would say the rest, what a clinical informaticist does, what a digital academy analyst does, how we work on change management, workflow integration, training, adoption, those skills are not unique. I think it's about how do you apply them to these different use cases and these different types of technologies.

Rae Woods (10:14):
There's a couple of things that I'm hearing from you. One is that scale and safety actually have to be hand in hand, and if you are going to get to scale and do it in a way that is safe and effective, you actually have to focus on the infrastructure. But I'm imagining the pushback that our listeners are having right now, and that's that in their mind at some point, the solutions do matter. They matter to, let's say the doctors who are now increasingly bombarded with different tools and different solutions that could maybe help them in their individual practice. So my question is, when you evaluate individual solutions, how do you operationalize against the question, does this solve one of those problems that Sutter cares about, not that this individual doctor cares about?

Dr. Veena Jones (11:06):
I would say that's probably one of the most significant parts of the job is constantly being that intake. I hear from clinicians all the time about their pain points and the possible company they talk to or the person they talk to that's using some solution that they want to bring in. And so how do we filter through that noise? Sometimes it's, well, how many people are coming to me with that problem? Is it widespread enough? Am I hearing about it enough at scale that it needs to be on our top five problem list as an organization? What is the impact? So if I hear about something from one clinician, but we dig in and we realize that that pain point has a really broad impact across our clinicians or across our patients, I think that would be a reason that that problem would come to the surface.

(11:59):
I think another factor that we look at really early on is scalability. We talked about how we go into a relationship with the company or with an AI solution with the thought of scale at the top of our minds. Pretty early on, if someone comes to me with a problem, I'm thinking, "Huh, I need to find a solution that could achieve scale. If it only helps one or two individuals, is it worth our time and resources?" The answer is often probably not, which isn't a very satisfying answer to the one or two clinicians that it could help, but I think that is how we have to think in a system this large.

Rae Woods (12:41):
How do you say no to the clinician when they come to your door with a solution that could work, but it's just not going to have enough impact or enough practicality or enough scale?

Dr. Veena Jones (12:51):
I mean, I think it's an honest conversation sharing the very things you just said. I think when it's a vendor, it's clear cut. It's not meeting our current needs. It's not solving for a problem that we feel we need to be solving right now. With our clinicians, I don't think it's a hard no, but I think it is a probably not now or not in the framework that you're thinking about it right now. That doesn't eliminate it forever. Often by keeping that running list of, "Hey, what are these pain points I'm hearing?" In the future, there are often ways that I can connect the dots and be like, "You know what? That point solution that they thought could solve that problem wasn't the right direction to go, but there's this other solution that maybe could solve for that and those five other things that could allow us to achieve scale."

Rae Woods (13:40):
I already mentioned that one of the big things that stood out to me when I was learning about Sutter Health's approach is this emphasis on scale early on. Another thing that stood out to me is that you've mapped artificial intelligence very deliberately across the clinical journey, thinking about it before the visit, during the visit, after the visit, so that every tool can also reinforce the other. That supports your central purpose of reducing clinician burden throughout the entire care journey, getting to scalability, et cetera. So to use your own framing, what was the problem you cared about optimizing before the patient visit?

Dr. Veena Jones (14:20):
Before the visit, clinicians spend a tremendous amount of time and energy reviewing charts, synthesizing a patient history, just understanding what it is they're walking into. And back to that term pajama time, the only time clinicians have to do that often is outside of their patient care time. So when we think about what could we do to make that easier to reduce that burden, there are tools out there that can help. And so that felt like a pretty easily scalable solution that could have significant impact. And so that's where one of the areas we decided to focus pretty early on was an AI solution that could help summarize a patient's chart and provide insights to the clinician quickly.

Rae Woods (15:12):
I think there's two unlocks there. One unlock is just speed, time, efficiency. That clearly matters. But the other unlock that you're describing is actually being able to capture a holistic and data rich picture of the patient so that you can be effective in the moment. And I find that often with AI, there's a lot of conversation about speed. How do we get as productive as possible? How do we get as efficient as possible? But what you're describing is how do we make use of the data that most clinicians would love to have in order to have a more holistic picture of the patient before they actually walk into the exam room?

Dr. Veena Jones (15:51):
Yeah, and I think the hard part is that the data is all there. So how do you distill that down into the most useful pieces of information that a clinician can quickly glance at to know what they need to know, as opposed to, let me spend an hour pre-charting for every patient that I see to make sure that I have a sense of what happened to them in the last year since I last saw them. And so AI in that space I think has been just incredible. The unlock there when we implemented this outpatient insights tool has meant that all of a sudden I can open up a patient chart and immediately there's this beautiful summary that tells me what's happened to my patient in whatever parameters I want to know them. I can actually ask questions or ask it to summarize a certain piece of the chart and it's right there.

Rae Woods (16:43):
So you've got the AI-driven chart summary. What told you that this program was ready to scale? And maybe just as importantly, what would've told you it wasn't ready to scale?

Dr. Veena Jones (16:57):
We had a select group of clinicians that we tested it out with. We started small and were really focused on understanding what the degree of adoption by those clinicians was, what their feedback was, how they actually incorporated or didn't into their workflows, and then how did they perceive it?

(17:24):
It started in the outpatient setting, the ambulatory setting, across specialties. We felt that was important to try to pick a representative sampling of where you would scale. We didn't just pick five primary care clinicians. We picked as diverse a group of clinicians as we could, but in the ambulatory setting. And then we had a pretty refined process for bringing those clinicians together, hearing their feedback. We actually created a survey where we asked them to give us some objective feedback on the quality of the summary that was generated and how useful was this to you and do you have any issues about the safety of the information that you're being served or the safety of the care that you could deliver based off of the summary?

(18:14):
There were a few options that we had when it came to how to configure the chart summary and how we present it to the clinicians. Do we present it to them categorized by patient problem or in a narrative form? We used this group of representative clinicians to give us that feedback to then ultimately drive the decisions that we made at scale.

(18:36):
You also asked me what would make us decide not to scale it. I think kind of the reverse of that. So if our clinicians were like, "Honestly, this isn't working for me, it's not adding any value," that to me would be a reason to not scale. But when what we were hearing was, "When can I tell my colleagues they're going to get it too?" That in itself is a driver to try to give it to everyone.

Rae Woods (19:01):
Absolutely.

Dr. Veena Jones (19:03):
And then I would say safety or quality concerns is another big one. We have an easy way that clinicians can tell us, is this working or is it not working? And if the majority of them are saying it's working and they really want others to get it and it's working within their workflows of how they use the EHR, those are all check marks to tell us that we should be able to scale.

Rae Woods (19:28):
I love that as you're in this wave one, you are testing with a diverse group of clinicians and you're starting to refine. And this is where I think many organizations can get stuck in that pilot purgatory is that they just keep refining because when you ask for feedback, especially from doctors, you are going to get it, but you did not stop there. Where did you decide to scale this pre-visit AI piece next?

Dr. Veena Jones (19:57):
We are not aiming to achieve perfection. We are aiming to achieve good enough to scale. And I think those are two different things. I have not yet seen or really been able to achieve perfection in a single tool for every person that uses it. I don't necessarily think that that's possible. Some of it is flipping that narrative. When I have these conversations with our pilot group, I mean, I ask them, "What do you all think? Is this ready? Is it good enough to bring to others or do we need to make some refinements or changes to get there?"

(20:39):
With the chart summarization, we spent a couple of months having the initial user group try it out, give their feedback. Based on their feedback, we made some changes, asked them to try it out for another couple weeks, regrouped. But I will say the fact that along that journey we kept hearing, when are my colleagues going to get this? Despite the fact that they don't think it's perfect, despite the fact that they're telling us they want us to refine it, others want it, that's a huge signal.

Rae Woods (21:09):
And others want it not just in the outpatient setting, they want it in the inpatient setting too.

Dr. Veena Jones (21:14):
So really good point. When we first scaled, we scaled across outpatient, across all ambulatory clinicians. Very quickly, we started hearing that everyone else wanted it too, and it wasn't designed for them. It was not a tool that was really meant to be used by our ED docs or our hospitalists, but they thought there could be value too. And we said, "Well, great. Let's give it to them and see what they can teach us." That curiosity and that willingness to try something out, even though it's not how the tool is designed, has also allowed us to go pretty far in how we've experienced the technology.

Rae Woods (22:49):
Scale is the goal, and because of that, you're not just thinking across outpatient for the pre-visit planning piece, you're also thinking about how you can use AI during the visit. And this is where the conversation naturally turns to ambient listening, which is what a lot of our listeners are really, really excited about. The typical approach is start narrow, go specialty by specialty, roll out ambient listening and documentation tools one by one. You didn't do that. You launched ambient listening with a phased approach, but one with a unusually broad scope. How big are we talking here and how did you do it?

Dr. Veena Jones (23:26):
Sutter Health was one of the earlier adopters of ambient listening technology. We started over two and a half years ago now, and I said this multiple times today already, but the thought was always scale. And so when we chose our initial group of clinicians, I will say the ambient listening technology at the time was nowhere near as mature as it is now. It was only available to our outpatient docs. But when we chose our initial group of ambulatory users, we chose representatives from as many specialties as we could. We really wanted a diverse and representative population so that we could learn from those experiences and so that it could then drive how we approached scaling.

Rae Woods (24:17):
And there's kind of no way that you can have a diverse group of clinicians without scale, without going big.

Dr. Veena Jones (24:24):
Yes. Our pilot, so to speak, was a hundred clinicians.

Rae Woods (24:28):
Wow.

Dr. Veena Jones (24:28):
I know in talking with other organizations, that's large for a pilot, but for an organization as large as us with as large of a geographic footprint as we had, when I was selecting these initial users, I was actually looking at our geography because I wanted to be able to say, "This works for that primary care doc that works in that rural setting, and it works for the specialist that works in the heart of San Francisco." I needed to gain all of that experience to then drive how I approached scaling it.

Rae Woods (25:02):
So before the visit, we've got AI supporting patient histories. During the visit, you've got ambient listening, and I think you've also added some tools to drive some faster clinician decision making during the visit. I'm not sure you answered my question, which was how big are we talking about knowing that scale was your goal?

Dr. Veena Jones (25:22):
I'll give you some numbers from just last month. Over 6,000 of our clinicians have access to our ambient listening technology today.

Rae Woods (25:32):
Wow.

Dr. Veena Jones (25:33):
As of about a month ago, we had generated over five million notes.

Rae Woods (25:37):
Wow.

Dr. Veena Jones (25:39):
And what's crazy is if you look at the graph of number of notes over time, it looks like it's rising exponentially.

Rae Woods (25:47):
And those numbers, that's just for the ambient listening?

Dr. Veena Jones (25:50):
Yeah.

Rae Woods (25:50):
Which takes me to the last piece of the patient journey. Where are you focusing your AI efforts after the patient visit?

Dr. Veena Jones (25:57):
The problem we were trying to solve after the visit, our clinicians are incredibly burdened by in-basket messages, by patient medical advice requests. In that technology, I would say we were one of the earliest adopters of leveraging AI to help our clinicians draft responses to patient medical advice requests.

Rae Woods (26:23):
Veena, we've admittedly focused most of this conversation on how Sutter is using AI to address the central challenge that you set out to solve, which is clinician burden, and also to achieve your central goal, which is scale. But at the end of the day, these tools ultimately touch the patient. You spent all this time trying to focus on the clinicians and how do you test and scale these tools across as diverse of a group of physicians as possible. You also involved patients and families as advisors. How did that work and how did that feedback change the way that you use these AI tools?

Dr. Veena Jones (27:01):
One of the best examples of that would be how we involved our patient family advisors in refining our AI for in-basket message responses. Very, very early on when we started evaluating and validating that technology, we included our patient and family advisors and actually had them review the messages that were being generated, and we had them review those against certain criteria. So what is the tone of the message? How clear is it? Is it actually answering the question? Is it appropriate and sensitive in the ways that a patient needs them to be? And based on our patient family advisor feedback, we actually refined the LLM prompt and the outputs of the model.

Rae Woods (27:51):
Oh, wow.

Dr. Veena Jones (27:52):
We made specialty specific adjustments. We made adjustments based on tone and clarity. And what we found in doing that was as we were asking questions about trust, accuracy, those metrics improved, and that's what helped us make a decision, back to the question of scaling, that's what helped us decide, "Hey, this is actually in a good place and we think we can bring it into the hands of more of our clinicians." And therefore more of our patients since they're the ones actually receiving those draft responses.

Rae Woods (28:27):
I think there are going to be a lot of listeners that are, I'm going to use the word jealous of how much that Sutter has been able to achieve, because for many of them, they're probably still feeling stuck in that pilot purgatory. So I want to give you a moment to speak directly to our listeners. What's the very first move that they should make before just buying or building another AI tool? How do we help our listeners get to the kind of scale that Sutter Health has achieved?

Dr. Veena Jones (28:56):
I want to take us back to the very beginning of the conversation we had and urge everyone out there to really look inside and try to understand the problems that you need to solve within your organization. Staying focused on the simple question, where could AI or any technology you're looking at, meaningfully reduce friction for your clinicians, improve the patient experience, reduce administrative burden? I think it's really, really important to hone in on the problem you're trying to solve. And if you can do that and then layer on the technology and the solutions with that goal in mind, I think that's what will help you be successful.

Rae Woods (29:42):
Well said. Well, Dr. Jones, thank you so much for coming on Radio Advisory.

Dr. Veena Jones (29:46):
Thank you so much for having me. This was a great conversation.

Rae Woods (29:54):
Here's what makes Sutter's story different. It's not just the tools they chose. It's the infrastructure that they built behind them. And there's something subtle that Veena said that I want to make sure that you don't miss. The infrastructure isn't just for AI tools, it's for any tool. That's why we at Advisory Board spend so much time talking about governance and operations and taking a problem first approach to tech adoption, whether it's artificial intelligence or otherwise. There's a lot more to Sutter's story, and so we put some more details in the show notes because remember, as always, we're here to help.

(31:02):
New episodes drop every Tuesday. If you like Radio Advisory, please share it with your networks. Subscribe wherever you get your podcasts and leave a rating and a review. Radio Advisory is a production of Advisory Board. This episode was produced by me, Rae Woods, as well as Abby Burns, Chloe Bakst, and Atticus Raasch. The episode was edited by Katy Anderson with technical support provided by Dan Tayag, Chris Phelps, and Joe Shrum. Additional support was provided by Dominique Delgadio. Special thanks to Elysia Culver. We'll see you next week.