(Image credit: Mohamed_hassan/Pixabay.)
By Upendra Belhe and Marty Ellingsworth
AI has a future in production only when two conditions are simultaneously true:
- Data integrity is strong enough to establish what is “true enough” to act on, and
- Decision frameworks are explicit enough to establish who owns outcomes, how exceptions escalate, and what evidence is produced by default.
Many organizations address the first condition (data governance, quality programs, platform modernization) while leaving the second undone (unclear decision rights, ad hoc escalation, weak evidence capture). In regulated, contested environments—insurance being a prime example—that mismatch is where AI credibility breaks. Governance stops being abstract the moment a decision is contested.
Why things are different with AI— before AI, prior decision-support systems assumed advice, not action. Tools surfaced options. Humans decided. That’s why reasoning saying “it’s just a tool” worked as it sidestepped being a decision—execution, authority, and liability stayed clearly human. Once systems act at machine speed, governance that relies on post-hoc review is no longer governance—it is hindsight.
What has changed is speed and coupling. Today’s systems don’t just inform judgment. They initiate, recommend, and sometimes execute—at machine speed, before a human can meaningfully step in. That is where prior reasoning breaks down in production. When decisions are automated, they are open to being contested, and reopened years later, so the question isn’t “was the model helpful?” It’s “who was authorized to act—under what mandate— at that moment?”
Decision support improved judgment. Decision automation requires owned, admissible authority—by design.
- The production reality: why pilots don’t predict survivability
Most AI initiatives begin with controlled success. Data is curated, labels are reasonably consistent, and evaluation is conducted against stable definitions and fixed time windows. In that setting, model performance can look impressive—often impressive enough to justify expanding the program. Yet production is not a scaled-up pilot. Production is an environment where:
- Data is incomplete and arrives late (documents, endorsements, supplements, third-party sources).
- Identity is fragmented (people, businesses, properties, vehicles, providers appear under multiple identifiers).
- Operations are adversarial (fraud, misrepresentation, strategic behavior).
- Decisions are regulated and contestable (appeals, audits, litigation, consumer complaints, market conduct).
- Edge cases are not exceptions—they are a meaningful share of the workload.
This is why enterprise AI rarely fails because the model is “not smart enough.” It fails because the organization is not prepared to answer the questions that only emerge after the system begins to influence outcomes at scale: Who is accountable? What did the system “know” at the time? What was the basis for action? What happens when the system is wrong?
Insurance illustrates this dynamic particularly well because underwriting, claims, and pricing decisions are routinely contested months or years later, making evidentiary governance a frontline operating requirement. In insurance, these questions are not theoretical. Underwriting and claims decisions are frequently revisited, contested, and inspected. The moment AI shifts from ‘analysis’ to ‘influence,’ the burden of proof shifts as well. Accuracy matters, but it is no longer sufficient.
This pattern is consistent across enterprises: confident pilot success gives way to production complexity, decisions become contested, and trust breaks down—not because the model failed, but because accountability was never operationalized.
- Two layers of production viability: integrity and decision architecture
One establishes what is true enough to act on. The other determines whether actions are defensible once outcomes are questioned.
Layer 1: Data integrity (what is true enough to act on)
This is the foundation leaders tend to recognize: quality, lineage, access controls, security, and compliant data handling. But “integrity” in production goes beyond hygiene. It is the ability to establish stable meaning and reliable identity under operational stress.
Layer 2: Decision architecture (what is defensible enough to stand up to scrutiny)
This is the layer organizations often leave implicit: decision ownership, escalation paths, human judgment boundaries, and evidence capture. It determines whether an organization can defend an outcome when a customer, regulator, auditor, or court asks, “Why did you do that?”
Most governance programs disproportionately emphasize Layer 1 and assume Layer 2 will “follow.” In regulated domains, it often doesn’t. And when it doesn’t, the model becomes the scapegoat for what is ultimately a leadership and operating-model gap.
- Layer 1 in practice: integrity is identity, meaning, and provenance
Production AI is only as coherent as the world it believes it is operating in. Three integrity disciplines matter most.
3.1 Entity resolution is not a technical feature—it is a business control
Insurance decisions are primarily contractual legal entity-driven: who the insured is, what they own, where it is, who is related to whom, what has changed, and what behavior patterns exist over time. Yet real enterprise systems create identity fragmentation:
- Multiple policy admin systems and lines of business
- Variations in name, address, phone, email, employer, taxonomy
- M&A, vendor transitions, and legacy “shadow” records
- Fraud and strategic misrepresentation
When entities cannot be resolved reliably, models inherit ambiguity. That ambiguity appears as inconsistent features, unstable labels, and non-reproducible reasoning. Over time, the organization begins to argue about “what happened” instead of improving outcomes.
Executive reality: entity coherence is a prerequisite for responsible automation. It determines whether the organization is acting on a consistent representation of the customer, risk, and event.
3.2 Semantic coherence is the difference between data movement and decision reliability
Federated data products, domain ownership, and democratized analytics only work when there is a shared grammar. Absent that grammar, the organization creates a Tower of Babel: “loss ratio” means one thing in one domain, another elsewhere; “claim severity” is defined differently across systems; “fraud” becomes both a label and a suspicion indicator; “coverage” is both contractual language and operational shorthand.
Production AI magnifies semantic drift because it operationalizes meaning. The model doesn’t simply analyze fields—it turns them into actions. If definitions differ across domains, the organization scales inconsistency, not intelligence.
Semantic coherence is often mistaken for “metadata work.” In production, it is an operational control: stable meaning enables stable decisions, auditability, and continuous learning.
3.3 Lineage and provenance are the evidentiary backbone
In high-scrutiny environments, it is not enough to say a model performed well historically. Organizations must be able to reconstruct:
- What data was available at the time of the decision
- How it was transformed and which versions were used
- Which model, rules, thresholds, and policies were applied
- Whether the decision was fully automated, assisted, or overridden
- Why an exception was escalated or not escalated
Without this, an organization can neither defend decisions nor learn responsibly. “We don’t know which version ran” is not a technical inconvenience; it is a governance failure with regulatory and reputational consequences.
- Why traditional observability is insufficient
Most organizations equate production readiness with observability defined by latency, uptime, and drift. These metrics are necessary but insufficient. In regulated decision environments, leaders also need observability of decision behavior:
- Rates of human override by segment and reason
- Volume and type of exceptions and escalations
- Outcomes of escalated cases and disputes
- Distributional behavior across protected or sensitive groups (bias monitoring)
- Geographic and regulatory variance (jurisdictional compliance)
- Evidence completeness rates (is a decision package audit-ready by default?)
This is not “more dashboards.” It instruments governance into runtime operations—so leaders can see whether decisions remain within controlled bounds.
- Layer 2 in practice: governance becomes real when a decision is contested
The critical transition for AI in production is the transition from “model risk” to “decision risk.” This is where the typical governance conversation becomes inadequate. Policies, principles, and committees do not resolve contestability. Decision architecture does.
The key proposition: governance stops being abstract the moment a decision is contested. At that moment, the organization must produce four things—quickly and consistently.
5.1 Ownership: who is accountable for the outcome?
When AI contributes to a decision, ownership often diffuses. The model team blames data. Data blames business definitions. The business blames the vendor. Compliance blames documentation. In contested scenarios, diffuse accountability is operationally fatal.
Production survivability requires explicit decision ownership—named roles, decision rights, and clear accountability boundaries. If a decision is high consequence, ownership cannot be “the AI program.” It must be an accountable operating leader with defined authority and responsibility. If no named operating leader can own a decision end-to-end, the organization has not automated responsibly—it has merely displaced accountability.
5.2 Escalation: what happens when the system is uncertain or out of bounds?
Production environments are full of incomplete signals and edge cases. Mature systems do not attempt to eliminate uncertainty. They route it. The organization must define escalation rules that are:
- Consistent (not dependent on who is on shift)
- Fast enough for operational reality
- Calibrated to decision consequence (higher stakes, stricter routing)
- Documented in a way that can be defended under scrutiny
This is where many organizations fail: they build models that can generate outputs but not systems that can manage uncertainty safely.
5.3 Human judgment: where is it required, and how is it exercised?
Human-in-the-loop is often treated as a moral statement rather than an operating design. In production, the question is not “Is a human involved?” but “Where must human judgment intervene, and what does that intervention look like?”
Key design choices include:
- Which decisions are assist-only vs auto-action
- What constitutes “sufficient integrity” to allow automation
- What evidence must be presented to the human reviewer
- How overrides are captured and fed back into learning loops
- How reviewer behavior is audited for consistency and bias
Human judgment is not a failsafe if it is not structured. Unstructured override patterns become another source of hidden risk.
5.4 Evidence: what can you prove, by default?
In regulated systems, coherence is not philosophical; it is auditable. If an organization cannot produce an “evidence packet” for a decision—inputs, versions, rationale, thresholds, and escalation history—then governance is performative.
Evidence must be produced as a byproduct of execution, not as a retroactive exercise. Retroactive evidence collection fails under time pressure, increases operational cost, and creates inconsistent narratives.
- Designed restraint: the hallmark of mature AI
A common misconception is that AI maturity equals autonomy. In practice, maturity equals designed restraint: the system acts only when integrity prerequisites are met and decision rights are clear; otherwise it defers, escalates, or requests additional information.
In practice, designed restraint shows up in system design choices:
- Integrity thresholds determine when the system may act
- Decision rights determine what the system is allowed to do
- Escalation pathways determine what happens when limits are reached
- Evidence capture ensures actions can be defended later
This posture is what converts AI from “impressive” to “trusted.” It accepts that uncertainty is normal and makes uncertainty manageable.
- From pilots to defensible systems: a practical path
Organizations that actually scale AI treat production viability as a managerial discipline. A practical approach includes six steps.
Step 1: Inventory decisions, not use cases
Identify high-frequency, high-consequence decisions: underwriting referrals, claim triage, fraud alerts, subrogation opportunities, premium audit escalations, pricing adjustments. Treat each decision as an operating asset with owners and controls.
Step 2: Define integrity prerequisites by decision consequence
Not all decisions require the same integrity threshold. A low-stakes workflow can tolerate ambiguity; a regulated denial cannot. Establish truth thresholds aligned to consequence.
Step 3: Design escalation architecture
Define the default routing for uncertainty. Ensure escalation is not informal. Measure escalation rates and outcomes.
Step 4: Build evidence capture into runtime
Establish an audit-ready decision package. Automate lineage capture and versioning. Make evidence production part of the execution path.
Step 5: Instrument decision behavior
Monitor overrides, dispute rates, bias indicators, and jurisdictional differences. Treat decision drift as seriously as model drift.
Step 6: Govern learning loops
Continuous learning must be governed as carefully as deployment. Retraining without traceability and validation discipline is not innovation—it is uncontrolled change.
- Managerial implications: how leaders should evaluate AI readiness
Senior leaders can cut through noise with three readiness tests.
Test 1: Can the organization defend a decision under contest?
If a customer challenges an outcome, can you explain what happened, who owned it, and what evidence supports it—without assembling a manual narrative?
Test 2: Is governance embedded in execution or layered on afterward?
Governance that exists primarily in documentation will not survive production pressure. Look for runtime controls: thresholds, routing, evidence, and monitoring.
Test 3: Does the operating model match the system’s level of autonomy?
As autonomy increases, decision accountability must become more explicit, not less. If escalation and decision rights are vague, autonomy will amplify risk.
- Conclusion: the case for defensible AI
The next phase of enterprise AI is not defined by smarter models. It is defined by systems that can withstand scrutiny. Production survivability emerges when organizations treat AI as an operating capability grounded in two disciplines:
- Integrity: coherent identity, stable meaning, and provable provenance
- Decision architecture: explicit ownership, structured escalation, bounded human judgment, and default evidence
When those disciplines are designed together, AI can scale responsibly. When they are not, organizations may still deploy models—but they will struggle to defend outcomes, learn safely, and sustain trust.
In regulated environments, the standard is clear: governance becomes real at the point of decision, not in the framework. And in production, coherence is not philosophical—it is auditable. The question for executives is no longer whether AI is accurate, but whether their operating model can survive scrutiny once AI decisions are challenged
Executive takeaway: “Decision Integrity Wins”
Editors Note: This article was co-authored by Martin Ellingsworth. Martin Ellingsworth has held prestigious and industry influencer positions across the insurance landscape. Most recently as the go-to-market leader for J.D.Power and previously with industry leaders USAA and Verisk Analytics (ISO). Marty brings extensive career and industry experience in creating value from data and advanced analytics. His longstanding accomplishments in both personal and commercial lines include productizing deep industry knowledge in risk assessment, risk segmentation, marketing, rating, customer experience, claims servicing, and fraud prevention. Marty’s research focuses on helping insurance companies build data-driven, customer-centric digital enterprises. Marty has a Bachelor of Science in operations research from the United States Air Force Academy and a Master of Science degree in operations research from the Air Force Institute of Technology. He enjoys reading, family, and his pets in the calm weather of Southern California, but sadly, never learned to surf.