Figure 4: IVP AI Assistant - April to August 2025 initial testing results

• The system achieved 97% accuracy across 500,000+ data points, and there was consistent performance across document types including structured forms, unstructured emails and call center transcripts.
• Processing 2,753 total records with structured as well as unstructured data proved to be flexible in handling diverse sources of pharmacovigilance data like case report forms, spontaneous adverse event reports, literature cases and other types of documents.
• 150,000 individual case safety reporting fields were extracted with confidence scores that provided rich performance information, revealing patterns that inform optimization and appropriate confidence cutoffs.
• An 81% decrease in the need for manual reviews of call center transcripts is a big operational gain. The AI properly identifies and extracts relevant safety information from conversational text highly accurately.
• 99% accurate at identifying adverse events with only 6% false positives demonstrates precision in primary areas of pharmacovigilance. Low false positive rate indicates excellent balance of sensitivity-specificity attained.

Key Learnings: What the Data Told Us

Boiling down all the data from early testing, here are three key takeaways:

1. Care Must Be Taken When Interpreting Confidence Scores

• Confidence scores produced by AI provide valuable guidance, but care must be taken in interpreting them. Findings reported that scores were unreliable about 3% of the time—small numbers count at scale.
• More concerning: when confidence scores were inaccurate, the model was very confident in making the wrong predictions. This only made it more necessary to implement multi-layered validation approaches rather than simply accepting AI-generated confidence measures at face value.

2. Traditional Evaluation Metrics Fall Short

• Traditional AI evaluation metrics from other disciplines were not sufficient for generative AI in the healthcare field. Traditional measures like precision, recall and F1 scores fell short of capturing pharmacovigilance's nuanced requirements.
• IQVIA found that measurements for retrieval accuracy, response relevance and consistency were more valuable. Domain-specific measures required the collaboration of AI professionals, pharmacovigilance experts and regulatory peers.

3. Trust Must Be Established In, Not Bolted On

• Bolting on governance controls after the fact will not result in trust. Successful implementations blend governance into system design and user experience from their earliest stages.
• This was mirrored in confidence-driven review by humans, administratively configurable risk thresholds, simple-to-use interfaces presenting extraction coordinates, and audit trails as core capabilities rather than overlays.

The Regulatory Landscape: Navigating Evolving Mandates

The global regulatory landscape for healthcare AI continues to evolve quickly. The EU AI Act brings with it certain healthcare specifications and January 2025 draft FDA guidelines emphasize validation in settings of intended use and lifecycle monitoring. IQVIA's forward-looking regulatory engagement included the contribution of 17 subject matter experts to the development of FDA guidance, showing value from industry engagement in shaping frameworks rather than responding after publication.

Working AI Governance: The Human Factor

AI governance is more of a people challenge in the end. Success is about meticulous alignment between business, compliance, and technical teams. Business teams must prepare organizations through redesigned workflows rather than automating current ones, having clear use cases and success criteria, and pushing change management. Compliance teams define acceptable levels of risk, enable regulatory harmonization geographically, monitor changing requirements and set effective escalation processes. Technical teams implement secure architecture, enable guardrails and traceability, maintain model performance and security, and construct effective feedback loops with business processes.

Looking Forward: Questions for Implementation

Organizations that are exploring similar implementations must address critical questions:

• Will you be ready to rethink existing workflows and not simply automate current processes?
• What are the key decisions your AI system needs to help with?
• Do your solutions build stakeholder trust by having sufficient transparency?
• What is the appropriate level of human oversight for your specific applications?
• How should validation approaches differ for generative AI versus normal models?

The Path Forward

IQVIA continues to lead and prove that successful AI governance within regulated domains requires more than technical expertise checkboxes or technical eminence. It requires deep domain expertise, constant multi-disciplinary partnerships, and unwavering commitment to the formation of trust through transparency and proven performance.

Pharma's AI adoption will be measured, not by algorithmic sophistication or rate of uptake, but by the robustness of governance systems that ensure these technologies deliver on their full potential: better patient outcomes while maintaining the utmost safety standards. Firms committing to strong governance systems today, borrowing from the experiences of IQVIA but being context-specific, will be the ones to finally realize AI's transformative potential. It is an investment that requires waiting, but long-term dividends are worth the effort. As regulatory landscapes evolve and artificial intelligence technologies continue to advance, the principles illustrated – security by design, traceability for transparency, human oversight integration and continuous monitoring – will remain valid as regulating technologies and regulations evolve.

The destiny of AI in drug safety does not depend on an urge for technological progress, but rather on industry foresight and caution employing governance frameworks that win and maintain the trust of patients, regulators and healthcare professionals globally.

In closing, I leave you with an overview of what enterprises must demand from AI providers to ensure their tools meet compliance and safety standards:

1. Risk-based approach – should be performed for each AI system and should form the basis for a risk-proportionate approach applied throughout the AI system’s lifecycle from development to routine use.
2. Human Oversight Enabled – The requirement is firmly embedded across all regulatory, government and industry bodies
3. Validity/Robustness - Evaluation should cover a wide enough range of relevant examples to interrogate the model’s objective and be statistically significant (to be demonstrated in product and project validation)
4. Transparency – Stakeholders need to have a general understanding of the AI system, it is use, risks and limitations
5. Privacy and Security – Assurances that your data is used responsibly with Generative AI technologies
6. Fairness and Equity – Review AI solution design, parameters, and feature selection for bias – in design and deployment
7. Governance and Accountability – It remains the responsibility of the MAH to validate, monitor and document model performance and include AI/ML operations in the pharmacovigilance system, to mitigate risks related to all algorithms and models used (EMA). AI providers need to support the documentation of the AI model (Annex D of the PSMF)