Healthcare
UC San Diego Health, Harvard Medical School and Anthropic build oversight system for patient-facing AI
UC San Diego Health, Harvard Medical School and Anthropic are building an open-source system to monitor the safety of recommendations made by patient-facing AI.
UC San Diego Health, Harvard Medical School and Anthropic are developing an open-source oversight system for patient-facing AI. The project is part of the Advanced Research Projects Agency for Health’s ADVOCATE program and focuses on AI-supported care for patients with heart failure.
The proposed system is intended to check the safety of medical recommendations made by patient-facing AI after deployment. One example cited in the report is reviewing a recommendation to increase a patient’s medication dose.
The broader program aims to develop an FDA-authorized agentic AI system for cardiovascular care. The oversight layer is designed to assess recommendations rather than simply assume that an AI tool remains safe after it goes live.
The work matters because patient-facing systems can influence treatment decisions outside traditional clinician-to-clinician workflows. Independent or open-source monitoring could provide a mechanism for identifying unsafe recommendations and supporting regulatory review as agentic healthcare AI becomes more capable.
Original source: Becker’s Hospital Review