DORA Observability for AI Systems: What Your Monitoring Must Cover

The Digital Operational Resilience Act (DORA) mandates that financial entities maintain comprehensive ICT risk management, including continuous monitoring of ICT systems. When those systems include AI — and increasingly they do — the observability requirements expand into territory that traditional APM tools don't cover.

This isn't about checking a compliance box. DORA observability for AI systems is about knowing, in real time, whether your AI is performing correctly, securely, and within the parameters your risk framework allows.

What DORA Requires

DORA's monitoring requirements are spread across several articles:

The AI Observability Stack for DORA

Traditional infrastructure monitoring (uptime, CPU, memory) is necessary but insufficient. AI systems require additional observability layers:

Layer 1: Infrastructure Metrics (Standard)

The baseline you already have (or should have):

• GPU utilisation and memory across model serving pods
• Inference latency at p50, p95, p99
• Request throughput and queue depth
• Error rates and HTTP status code distribution
• Container and pod health, restart counts
• Network throughput between model serving and data layers

Layer 2: Model Performance Metrics (AI-Specific)

This is where most organisations have gaps:

• Output quality scores — automated evaluation of model output quality using reference-based or reference-free metrics. Track over time to detect degradation.
• Confidence distributions — are model confidence scores shifting? A gradual decrease in average confidence may indicate data drift before output quality visibly degrades.
• Input distribution monitoring — are the inputs the model receives in production similar to what it was trained/evaluated on? Distribution shifts are the leading indicator of performance problems.
• Hallucination rate — for generative AI, what percentage of outputs contain fabricated information? This requires automated checking against source documents.
• Retrieval quality — for RAG systems, are the retrieved documents relevant? Track retrieval precision and recall with automated and sampled human evaluation.

Layer 3: Security Monitoring

DORA Article 9 requires security monitoring. For AI systems:

• Prompt injection detection — real-time scanning of inputs for injection patterns. Log and alert on suspected attempts.
• Data exfiltration detection — monitor model outputs for patterns that suggest training data or sensitive information leakage.
• Access anomalies — unusual query patterns, volume spikes, or access from unexpected sources.
• Model integrity — verify that deployed model weights haven't been tampered with. Checksum validation on deployment and periodically.

Layer 4: Compliance Monitoring

The layer that ties everything back to DORA requirements:

• SLA compliance — is the AI system meeting its availability and performance SLAs? Track and report continuously.
• Audit log completeness — are all required events being logged? Missing logs are themselves a compliance failure.
• Data residency verification — continuous confirmation that data processing stays within defined jurisdictions.
• Third-party dependency health — if you use external model APIs, monitor their availability and performance. DORA Article 28 requires monitoring of ICT third-party service providers.
• Incident detection SLA — how quickly are anomalies detected? DORA expects timely detection. Define and measure time-to-detect for different severity levels.

The Alerting Framework

Monitoring without alerting is just data hoarding. Structure alerts by severity:

Incident Reporting Under DORA

DORA Article 19 requires reporting of major ICT-related incidents. For AI systems, define what constitutes a reportable incident:

• AI system outage affecting customer-facing services
• Confirmed data breach through AI system (prompt injection leading to data exposure)
• AI system producing systematically incorrect outputs affecting financial decisions
• Security incident targeting AI infrastructure (model theft attempt, adversarial attack)
• Third-party AI provider outage affecting critical business processes

Each of these requires defined runbooks, escalation paths, and reporting templates. Your observability stack must detect these conditions and trigger the appropriate response workflow automatically.

Dashboard Design

Different stakeholders need different views:

• Operations team — real-time infrastructure and performance metrics. Focus: is it running? Is it fast enough?
• Data science team — model quality metrics, drift analysis, evaluation results. Focus: is the model performing well?
• Security team — threat detection, access patterns, anomaly alerts. Focus: is it secure?
• Compliance team — SLA adherence, audit log status, regulatory metric summaries. Focus: can we prove compliance?
• Executive level — aggregate health scores, incident counts, cost trends. Focus: should I be worried?

Implementation Priorities

If you're starting from scratch, prioritise in this order:

1. Infrastructure metrics + alerting — you need to know when things are down. This is table stakes.
2. Audit logging — complete input/output logging for compliance. Hard to add retroactively.
3. Security monitoring — prompt injection and access anomaly detection. The threat is real and growing.
4. Model quality monitoring — drift detection, quality scoring. Prevents silent degradation.
5. Compliance dashboards — aggregate views for regulators and auditors. Builds on everything above.

DORA compliance isn't a one-time certification. It's a continuous operational capability. Your AI observability stack is the evidence that you're meeting that obligation — every day, for every inference, across every AI system in your organisation.