OSFI E-23 model risk management applies to the LLMs and agents you deploy, not just to credit scorecards and capital models. If your institution is federally regulated in Canada, an examiner can ask who validated your GenAI system, on what data, and whether that party was independent of the team that built it. The honest answer needs evidence, and that evidence comes from evaluation.
TL;DR
- OSFI’s Guideline E-23 sets model risk management expectations for federally regulated financial institutions in Canada, covering the model lifecycle, validation, and governance, with scope broad enough to include AI and machine learning models.
- Validation that a supervisor can trust has to be independent of the build team, run on data the model could not have memorized, and reproducible after the fact.
- LLMs and agents strain classic validation: outputs vary run to run, public benchmarks are contaminated, and multi-step agents compound small error rates into large ones.
- The durable pattern is independent, contamination-free evaluation run inside your own VPC, with error bars and an audit trail. Build that evidence once and map it to E-23 and to global frameworks.
- Canada also proposed the Artificial Intelligence and Data Act, or AIDA, whose status continues to evolve. Confirm current obligations against source text.
What does OSFI E-23 expect for model risk?
Read E-23 as principles for managing model risk across a model’s life, not as a checklist. It expects federally regulated financial institutions to identify their models, govern them, and validate them, with accountability that a supervisor can inspect. The guideline addresses the full lifecycle: design, development, deployment, ongoing monitoring, and eventual retirement.
We describe the guideline at the level of principles on purpose. The specific structure of its expectations is best read from OSFI’s published material, so anchor to the source text rather than any secondhand paraphrase.
The through-line for evaluation is straightforward. You should be able to explain, with evidence, how a model was tested, why you trust the result, and who was accountable for the conclusion. That framing is not unique to Canada. It is the same logic behind US model risk guidance, the EU AI Act, and the ISO/IEC 42001 AI management standard.
Does E-23 cover AI and machine learning models?
Yes, in the sense that its model risk principles are broad enough to reach AI and ML systems, not only traditional statistical models. A large language model used inside a decision process is still a model producing outputs your institution relies on, so it falls within model risk management rather than outside it.
That matters because the properties that made classic validation tractable are mostly absent in GenAI. A logistic regression is deterministic and inspectable. An LLM or agent is neither, and each gap is a validation problem you have to close with testing rather than assume away.
Where does independent validation fit under E-23?
Independence is the load-bearing property. A team that both builds and grades its own system is optimizing for a passing score, not a true one. That is not an accusation of bad faith. It is the structural conflict that independent validation exists to remove, and it is why model risk guidance across jurisdictions asks for a critical review by parties separate from the model’s development.
Running an open-source eval harness in your own environment does not resolve this. You still author the checks and interpret the numbers, which is grading your own homework with a nicer rubric. The separation E-23’s governance intent points toward is exactly the separation a self-scored process cannot create. We cover where internal and independent evaluation each belong in our method, and the banking-specific case in independent model validation for banks.
Independence is about who owns the test, not where the compute runs.
What evidence do OSFI-regulated firms need?
Reproducible evidence tied to a decision, not a dashboard of metrics. When a validator or examiner asks whether a GenAI system was validated, a passing demo does not clear the bar. The artifacts that do are concrete and repeatable.
| Model risk question | Weak evidence | Audit-ready evidence |
|---|---|---|
| Who validated the model? | The build team’s own dashboard | An independent evaluation with a named, separate owner |
| Was the test gamed? | “We used a standard benchmark” | A contamination-free corpus dated after the model’s cutoff |
| Is the result reliable? | A single accuracy number | Scores with error bars and paired comparisons |
| Does the agent behave safely? | A one-shot demo | A trajectory and tool-selection evaluation across steps |
| Where did the data go? | An informal assurance | An in-VPC run with a documented, redacted data flow |
| Can you reproduce it? | A screenshot | Versioned model, dataset slice, harness, and scores |
Two points deserve emphasis. Contamination: when evaluation data has leaked into training data, a high score can measure recall rather than capability, and detection of that leakage is unreliable, so freshness beats forensics. Test on data that postdates the model’s cutoff, drawn from your own traffic. Compounding: an agent that is 95% reliable at a single step is only about 36% reliable across twenty steps, since 0.95 to the twentieth power is roughly 0.36. A single-shot demo hides exactly the failures a risk committee cares about.
Why contamination-free and in-VPC evaluation?
Treat both as table stakes, not features. A contaminated score is not a measurement of your model’s capability, so it cannot support a validation conclusion an examiner will accept. And your production data, often your model weights too, are exactly the assets your policy and Canadian data expectations ask you to keep inside your trust boundary.
Independence and privacy are compatible. Run the evaluation where the data lives so raw records never cross your boundary. Minimize and redact personal data at ingestion, keeping only what grading requires. Document the data flow so the evaluation is auditable rather than a new exposure. That is also what keeps a contamination-free corpus safe over time: private data holds its value only while it stays private. Where agent reliability sits in all this is covered in the field.
How does one evaluation map to E-23 and global frameworks?
You build the evidence once and reuse it, because these regimes ask the same underlying question in different vocabularies. Independence, clean data, reliability, and auditability satisfy the intent of OSFI E-23, US model risk guidance, the EU AI Act, and ISO/IEC 42001 at the same time.
The mapping is not a shortcut around any single regime, and each still carries specific obligations your compliance team must confirm against source text. But the evaluation artifact is common infrastructure. Design the evaluation once and tag each result to the control it satisfies, rather than running four disconnected exercises. We lay out that one-to-many mapping in one eval, four frameworks.
FAQ
Does OSFI E-23 apply to LLMs and agents? Its model risk principles are broad enough to reach AI and ML models, and an LLM used in a decision process is a model your institution relies on. Confirm the current scope and expectations against OSFI’s published guideline with your risk team.
Can our internal model risk function be the independent validator? Often yes, if it is genuinely independent of the development and use of the model. The test is separation of incentive and function, not whether the party is inside or outside the institution. Where internal capacity or specialist LLM tooling is thin, an external evaluator supplements it without replacing your governance.
Do we have to send data or models to a vendor to get independent validation? No. A properly designed evaluation runs inside your VPC so weights and production data stay under your control, with personal data redacted at the boundary. What makes it independent is that the evaluator designs and grades the test, not where the compute sits.
How does AIDA relate to E-23 here? Canada proposed the Artificial Intelligence and Data Act, and its status continues to evolve. Treat E-23 as your model risk anchor today and track AIDA separately, confirming both against source text rather than assuming a fixed relationship between them.
E-23 does not ask you to prove your AI is perfect. It asks you to prove you measured it honestly, independently, and can show your work. Book a free eval diagnostic and we will scope an audit-ready evaluation from your own traffic, inside your own VPC.