{"id":257,"date":"2026-10-23T16:43:17","date_gmt":"2026-10-23T16:43:17","guid":{"rendered":"https:\/\/smartapp.co.uk\/blog\/?p=257"},"modified":"2026-10-23T16:43:18","modified_gmt":"2026-10-23T16:43:18","slug":"the-real-test-of-ai-governance-is-what-happens-in-production","status":"publish","type":"post","link":"https:\/\/smartapp.co.uk\/blog\/the-real-test-of-ai-governance-is-what-happens-in-production\/","title":{"rendered":"The real test of AI governance is what happens in production"},"content":{"rendered":"\n

What the FCA’s latest findings mean for the engineering of reliable AI in financial services<\/em><\/p>\n\n\n\n

An AI system can pass its technical checks and still deliver a poor outcome. It can answer without inventing facts, stay within its instructions and remain available throughout the day, yet fail the person or business process it was meant to serve.<\/p>\n\n\n\n

That tension runs through the FCA’s AI in financial services: good practice and areas for improvement<\/em><\/a>, published on 19 October 2026 and updated the next day. The regulator draws on real examples from firms to examine consumer outcomes, governance and operational resilience.<\/p>\n\n\n\n

This is not a new AI rulebook. The FCA says its examples are illustrative, not mandatory templates. Existing obligations, including the Consumer Duty and relevant governance, outsourcing and resilience requirements, continue to apply according to the firm’s activities and circumstances.<\/p>\n\n\n\n

The publication nevertheless raises a practical question for anyone building or overseeing these systems: what turns a governance decision into a control that works when the AI is live?<\/strong><\/p>\n\n\n\n

Policies and approval records establish responsibility. They matter. But they cannot, on their own, prevent a model from using the wrong information, an agent from taking an unauthorised action or a failing service from continuing unnoticed. Those problems need engineering solutions, backed by business judgement and evidence.<\/p>\n\n\n\n

1. A successful technical control can produce the wrong outcome<\/h2>\n\n\n\n

One of the FCA’s most telling examples is Case Study C<\/strong>. A firm introduced guardrails that reduced model drift and unexpected responses. Yet the system then gave consumers overly generic outcomes and steered some towards inappropriate options.<\/p>\n\n\n\n

The reported result is important because the controls had succeeded at something measurable. They made behaviour more constrained. What they had not done was protect the purpose of the service.<\/p>\n\n\n\n

A similar mistake is possible whenever teams define success too narrowly: fewer unusual outputs, fewer exceptions or more consistent responses. These measures say something about system behaviour. They do not establish that people receive useful, appropriate support.<\/p>\n\n\n\n

The engineering lesson is that every material AI control needs two tests. First, does it reduce the targeted failure? Second, does it introduce another failure elsewhere in the journey?<\/p>\n\n\n\n

For a consumer-facing service, testing should include situations where a safe response is not a generic refusal but a helpful explanation, a clarifying question or a referral to a person. It should also cover differences in consumer needs, financial capability and vulnerability where relevant. A guardrail that avoids difficult cases by excluding the people who need the service most may look effective on a technical dashboard while undermining the business objective.<\/p>\n\n\n\n

This is why the starting point is not a model specification. It is a clear account of the task, the intended outcome, the foreseeable harm and the boundary between what AI may do and what must remain with a person or another controlled system.<\/p>\n\n\n\n

The FCA describes firms deciding where AI should support, automate or abstain<\/strong>. That is a useful design decision, not just a governance classification. It tells engineers which paths must exist in the application, including what happens when the system cannot safely complete a request.<\/p>\n\n\n\n

A firm’s policies should define these choices. Its software should make them real.<\/p>\n\n\n\n

2. Put authority and evidence into the architecture<\/h2>\n\n\n\n

There is a difference between instructing a model to behave and preventing an application from doing something it is not authorised to do.<\/p>\n\n\n\n

A prompt can tell an AI assistant to use approved information, avoid financial advice outside its remit and escalate uncertain requests. Those instructions are useful, but they are not reliable access controls or transaction authorisation.<\/p>\n\n\n\n

For material workflows, the model should operate inside boundaries enforced by other application components. Identity and permissions govern which data can be retrieved. A policy layer checks proposed actions against business rules. Sensitive changes go through an authorised workflow, including human approval where appropriate. The model proposes; the application decides what it may execute.<\/p>\n\n\n\n

The same principle applies to agentic AI<\/strong>, where software can select tools and take several steps without a person directing each one. An agent’s available tools should be limited to those necessary for its job, with narrowly scoped credentials, transaction limits, timeouts and defined escalation points. If an agent only needs to read records, it should not have write access. If a workflow can update accounts or send binding communications, approval should apply to the specific action, not to a general instruction issued earlier.<\/p>\n\n\n\n

These controls also reduce the damage possible from prompt injection, where text in a document, email or tool response attempts to redirect an AI system. The OWASP guidance on prompt injection<\/a> and excessive agency<\/a> is a useful technical reference. Content retrieved for analysis must not acquire authority to change permissions or bypass workflow rules merely because the model has read it.<\/p>\n\n\n\n

Information quality needs similar care. Many financial AI applications retrieve material from approved documents and cite it in their answers. This is often better than relying on general model knowledge, but a citation is not proof of correctness.<\/p>\n\n\n\n

The retrieved passage might be outdated, accessible to the wrong user or irrelevant to the claim beside it. The model may quote a genuine source while omitting a qualification elsewhere in the same document. A reliable design therefore needs versioned sources, access checks and evaluation of whether each material claim is actually supported. Where a figure or rule can be checked deterministically, use conventional code rather than asking a language model to certify its own result.<\/p>\n\n\n\n

Consider a policy requiring every AI-generated contractual finding to be supported by an authorised document. The application checks the user’s permissions and retrieves only approved document versions. The model returns findings in a structured format, with clause references and supporting quotations. Software verifies that each passage exists at the stated location and flags claims that cannot be substantiated for human review. Unsupported findings are not presented as verified conclusions. The workflow records source versions, validation outcomes and reviewer decisions, with access and retention controls. A governance statement has become a testable process.<\/p>\n\n\n\n

This is an architectural distinction with business consequences. The AI can assist with interpretation; it should not be the sole judge of its own authority or accuracy.<\/strong><\/p>\n\n\n\n

Case Study L<\/strong> shows how accountability can be divided without being diluted: the business area’s senior manager retained ultimate accountability, while a model owner oversaw the technical build. Business owners set outcomes, risk appetite and decision authority; engineers implement, test and operate the controls.<\/p>\n\n\n\n

3. Evaluation should test the service, not the demonstration<\/h2>\n\n\n\n

The FCA’s Case Study K<\/strong> describes a firm assessing AI-generated claims information against several criteria: completeness, faithfulness to sources, conciseness, adequacy of reasoning and safety. It combined staff checks with automated assessment, including an LLM used as a judge.<\/p>\n\n\n\n

That example is valuable because it moves beyond a single accuracy score. The quality of an AI output has several dimensions, and some matter more than others depending on the task.<\/p>\n\n\n\n

A concise response can omit a material qualification. A complete-looking answer can contain a false statement. A response can be factually sound but unhelpful for the person receiving it. Testing must reflect these differences and the cost of each failure.<\/p>\n\n\n\n

A practical evaluation set should contain ordinary tasks, difficult cases, conflicting information, missing evidence and examples of previous failures. For consumer applications it should also test materially different consumer circumstances, not merely variations in wording. The samples need known expectations reviewed by people who understand the business process.<\/p>\n\n\n\n

Automated evaluators help teams run tests frequently, but they must be checked too. An LLM judging another LLM may miss errors, favour familiar phrasing or share blind spots with the model being assessed. For important tests, compare automated scores with independent human judgements and revisit disagreements. A fluent explanation from an evaluator is not itself evidence that the judgement was right.<\/p>\n\n\n\n

Crucially, evaluate the whole workflow<\/em>. Test whether the system found the correct source, applied the right permissions, detected insufficient evidence, followed the proper escalation route and left the final decision with the appropriate authority. An isolated model benchmark will not catch a failed handoff or an unauthorised tool call.<\/p>\n\n\n\n

Evaluation should also be part of change control. A model upgrade, revised prompt, new document collection or changed tool permission can alter outcomes even when the application code is unchanged. Keep representative test cases under version control, compare results with an agreed baseline and make material failures block release or require an explicit, recorded risk decision.<\/p>\n\n\n\n

This is an engineering recommendation, not a newly prescribed FCA release process. It follows a familiar software principle: a test only protects production if someone acts on its result.<\/strong><\/p>\n\n\n\n

4. Human oversight and monitoring need operational consequences<\/h2>\n\n\n\n

In Case Study O<\/strong>, the FCA found that a human review role had become a tick-box exercise. Reviewers increasingly accepted AI recommendations without adequate scrutiny, even when errors or limitations were apparent. Unclear escalation criteria also allowed some outputs to proceed despite uncertainty.<\/p>\n\n\n\n

The lesson is not that human review is unhelpful. It is that attaching an approval button to an AI workflow does not create effective oversight.<\/p>\n\n\n\n

Reviewers need the right expertise, enough information to challenge the output, authority to change the outcome and a realistic workload. They should see the evidence supporting a recommendation and any material limitations. The system should record whether they accepted, changed, rejected or escalated it, rather than merely recording that someone clicked ‘approve’. Periodic independent reviews can help uncover automation bias, especially where staff have grown accustomed to agreeing with the system.<\/p>\n\n\n\n

Controls must continue after deployment. Conventional monitoring shows whether a service is running: availability, errors, latency and cost. AI monitoring must also ask whether it is doing the right things: unsupported claims, unsafe actions attempted, missed referrals, changes in output patterns and actual business outcomes.<\/p>\n\n\n\n

The FCA’s Case Study P<\/strong> offers a practical connection between governance and operation. A firm agreed red, amber and green performance thresholds between its business and model-risk teams. It defined actions when results entered amber or red, monitored performance through a dashboard and escalated recurring concerns.<\/p>\n\n\n\n

The thresholds were the firm’s own control design, not universal regulatory limits. That distinction matters. There is no meaningful industry-wide ‘acceptable hallucination rate’ for every application. A drafting assistant and an automated decision affecting a customer may have very different tolerances and intervention needs.<\/p>\n\n\n\n

In engineering terms, each important threshold needs an owner and an action. Increased sampling might be sensible at one level. A serious breach might restrict functionality, require specialist review or suspend the service. Monitoring that never changes how the system operates is reporting, not effective control.<\/p>\n\n\n\n

Case Study M<\/strong> adds incident planning: senior management and business teams approved a playbook covering outages, unexpected outputs and underperformance before deployment. This gives engineers agreed triggers for intervention.<\/p>\n\n\n\n

Production evidence should make a material interaction reconstructable: relevant model and workflow versions, source references, checks applied, actions proposed and executed, and human decisions. Distributed tracing can help connect model and tool calls across a workflow.<\/p>\n\n\n\n

But collecting every prompt and document in ordinary logs would introduce privacy and security risks. Sensitive content should be minimised, protected and retained only where justified. Traces show what the application did; they do not necessarily explain a model’s internal reasoning. Their value is to support investigation, challenge and improvement, not to claim perfect explainability.<\/p>\n\n\n\n

Finally, measure what the monitoring misses. A count of blocked requests shows that a filter is active, not how many harmful requests escaped it. Targeted sampling and reviews of subsequent complaints or incidents are needed to test the monitoring itself.<\/p>\n\n\n\n

5. Proportionate governance is an engineering choice, not a weaker standard<\/h2>\n\n\n\n

The FCA highlights smaller firms tailoring their governance arrangements rather than copying large-firm models. Its examples include firms testing samples of model outputs, adding vulnerability indicators to test cases and using carefully selected, human-reviewed data as reference answers. It also describes smaller firms building AI oversight into existing governance, assigning named owners and recording whether staff reviewed, overrode or escalated outputs.<\/p>\n\n\n\n

These are modest arrangements, but they can provide real evidence.<\/p>\n\n\n\n

A smaller financial firm does not need a central AI platform to begin sensibly. It can start with a narrowly defined use case, a responsible business owner, a short record of risks and permitted actions, a curated evaluation set, a simple approval process and scheduled reviews of live results. An issues log and a tested route back to manual handling may be more valuable than an elaborate dashboard nobody owns.<\/p>\n\n\n\n

These principles also apply to off-the-shelf AI products such as Microsoft Copilot or Claude, not only custom applications. Firms should check which version and settings staff use, what confidential data may be entered, how the vendor handles it and whether access permissions are appropriate. Staff still need to review outputs before consequential use. Commercial software may provide built-in safeguards, but accountability remains with the firm.<\/p>\n\n\n\n

The key is to concentrate effort where errors have consequences. An internal drafting assistant and a tool influencing lending or customer support should not receive identical scrutiny. Narrow scope also makes failures easier to recognise and contains their effect while the firm learns.<\/p>\n\n\n\n

Major institutions have different problems. Multiple business lines, models, suppliers and jurisdictions make inconsistent controls harder to detect. They may need shared model inventories, reusable identity and policy services, standardised evaluation pipelines, centrally available observability and independent review for higher-risk uses. Even then, ownership of a business outcome should not disappear into a central AI committee.<\/p>\n\n\n\n

For firms of any size, reliance on a third-party model does not transfer accountability. The FCA reports concerns about suppliers’ assurances going untested and about services built around models that firms could not readily replace. Sensible due diligence asks what data the provider processes, how changes are communicated, what performance evidence is available and what happens if the service fails or the model is retired.<\/p>\n\n\n\n

A second provider can improve resilience, but a compatible API does not mean equivalent behaviour. An alternative model needs validation for the tasks it would actually perform. Where appropriate, a tested manual fallback may be safer than an automatic switch to an untested model. Firms subject to specific operational resilience rules must also meet those obligations; the FCA’s examples do not replace them.<\/p>\n\n\n\n

The underlying discipline is consistent across firm sizes: decide what the system may do, test whether it works, observe what happens in use and make sure someone can intervene.<\/p>\n\n\n\n

The FCA’s publication is most useful when read through that lens. A governance policy is necessary because it sets direction and accountability. Engineering gives it practical effect through permissions, validation, deployment controls, evidence and operational decisions. Neither can substitute for the other.<\/p>\n\n\n\n

The real test of AI governance is not what an organisation says its system will do. It is what the system demonstrably does, and how the organisation responds when it does something else.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

What the FCA’s latest findings mean for the engineering of reliable AI in financial services An AI system can pass its technical checks and still deliver a poor outcome. It can answer without inventing...<\/p>\n","protected":false},"author":1,"featured_media":260,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[],"class_list":["post-257","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/posts\/257","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=257"}],"version-history":[{"count":1,"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/posts\/257\/revisions"}],"predecessor-version":[{"id":261,"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/posts\/257\/revisions\/261"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/media\/260"}],"wp:attachment":[{"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=257"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=257"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/smartapp.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=257"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}