Aren’t you still using a cloud service, though? "Athena has been a cloud-based company from day one. We built most of our infrastructure on private clouds — data centers we build out — but increasingly we’ve used third-party clouds. We have a substantial footprint in AWS and a non-negligible footprint in Microsoft Azure. Azure is the gateway to OpenAI in a data private way that we required.
"So, the AI machinery is running in the Azure cloud, we’re accessing it through an API in the Azure cloud, and then it interoperates with the rest of our machinery, which is in private data centers and AWS."
Were there significant hurdles? Even in a pilot of AI, what were the most difficult challenges? "There’s always hurdles whenever you roll out any new feature, much less new technology. We already talked about ensuring enough data privacy. Then we had contractual safety and right to use features of this type in our products that we were going to team with early adopters who were interested in using it.
"Then on the technical side, there are a couple of additional hurdles that are unusual in the software development space — or at least unfamiliar to many software organizations. That’s because AI capabilities don’t behave the same way that a lot of software developers are used to. You have to go through extra work to ensure it’s behaving the way you want it to and to ensure it’s behaving safely.
"To give a concrete example, if you purchase a database from Oracle or whoever, you go pick up Postgres or rent RDS from Amazon, whatever. You know how it’s going to behave. You know if a particular query will give you a specific response. You know in very deterministic ways how it’s going to behave. When the database doesn’t behave that way, it’s a bug, and you report a bug.
"In the AI world, part of the power of it is that it can generate novel and unexpected responses. That means that you now have this odd problem of how do you ensure yourself that the responses the AI is generating are the ‘correct’ responses?
We have to be very careful with how we protect our data and how we use our data. We had infosec involved and legal involved and procurement involved – all these people were involved in evaluating the contracts we had with Microsoft and OpenAI.
"For example, if we’re going to take a patient record that we imported from another EHR and reduce it to a thumbnail summary for another provider, how are we sure it was summarized correctly? How do we ensure that [genAI] extracted the information it should have and it didn’t extract some irrelevant information or it didn’t hallucinate something altogether. These are well-known dangers of large language models, and so you have to test for them.
"So we had to put testing processes into place and perform human review of a large number of cases and we had to have patient safety reviews of the outputs of these large language models. We had to do bias evaluation to see if they were producing biases or hate speech and things like that. Those are processes that are unfamiliar to many development teams."
Do you expect to see any ROI any time soon? How would you calculate that based on what you’ve deployed? "I think that’s one of the brave new world questions we’re all struggling with. Many people in the industry are asking that question at the moment. The capabilities of generative AI are quite impressive. However, how those things turn into real value, I think, is what we’re all discovering. I think we’ll be asking those questions as we’re rolling out our early preview features to our alpha partners, and they can give us direct feedback on how valuable it is for them to have these new capabilities. We will be asking questions in our strategy sessions and planning sessions."
Who are your customers, primarily? "Broadly, Athenahealth serves mostly the ambulatory physicians in the US, their nurses and their clinical staff. We have a small hospital footprint and a significant footprint in primary care and a number of other ambulatory specialities like orthopedics and OB-GYN."
"The customers who are getting the early preview are the ones interested in these capabilities. When you think about all the providers and all the networks of the providers in the United States, some of them are more hungry for technological advancements than others. Some of them desire stability much more. Our customer support managers who work with customers daily have a very good sense of who wants what and they were able to help us locate customers who were most excited about technological innovations and who’d be willing to partner with us to explore these new capabilities."