Fulfillment Exception Causality Engine Operations intelligence
Understanding why fulfillment exceptions happen.
A fulfillment exception reports what failed. It rarely says which upstream condition caused it, or which delivery promises are now at risk.
An independent prototype that starts from a fulfillment exception and works backward. It scores every recorded upstream condition (labor, inventory, equipment, carrier notices, queue backlogs) on timing, linkage, process stage and contrast with unaffected orders, then ranks the evidence-supported candidates or reports that the evidence is insufficient. It traces effects forward to orders and delivery promises and sets an investigation priority for the operator.
At a glance
- Problem
- Operators see the symptom (a missed pickup, a late delivery) long after, and far from, the condition that caused it.
- Built
- A simulator, an event-normalization pipeline, a deterministic causal engine and an investigation interface that explains every conclusion.
- Evidence
- A benchmark across 40 simulated datasets, including 15 fresh held-out seeds, with every wrong answer listed.
Scope. Synthetic data only. Every order, node, carrier and incident comes from a deterministic simulator; no real operation has been analyzed. Not affiliated with, sponsored by or based on any company's systems. It runs locally; there is no hosted demo.
The workflow problem
When orders miss a carrier pickup or a delivery promise, the exception queue shows the symptom. The person investigating has to answer three questions quickly. Which upstream condition caused this? Which other orders and promises did it reach? What should be checked first? The trigger often sits somewhere else: a staffing gap in packing, a stockout, a conveyor outage, a carrier hub problem. The signal that looks closest is often the wrong one.
What makes it hard
The loudest signal can be the wrong one
A carrier advisory can name the right carrier and region while the delay actually accrued in packing, hours before the shipment reached the carrier.
Causes and consequences look alike
A packing backlog is either the problem or the result of a labor shortfall upstream. Both show up as recorded conditions at the same node.
Event data is messy
Duplicates, legacy formats and malformed records have to be validated, deduplicated or quarantined before any reasoning starts.
Sometimes nothing recorded explains it
Silent transit delays leave no upstream record. Naming a candidate anyway would present a guess as a finding.
My approach
I split the question into a pipeline an operator can follow: exception → evidence → candidate conditions → chain of effects → downstream impact → investigation priority. Rather than sorting events by time and blaming the first one, the engine treats each recorded condition as a candidate and scores it. The candidate must line up with the exception in time, share orders, nodes or carriers with it, and constrain the process stage where the delay built up. Orders exposed to the condition must also have deviated more than comparable orders that were not exposed. If no candidate reaches moderate evidence, the answer is "insufficient evidence", along with what was observed and what evidence is missing.
Because real fulfillment data was not available to me, I built a queueing simulator that produces delays from injected conditions. Ground truth comes from re-running each scenario without its condition, and the engine is never allowed to read it.
System architecture
Each stage is labeled by what it is: interface, deterministic logic, stored data, generated data, or the language model. The list reads in the order data flows.
- 01 DataSynthetic simulatorModels allocation, pick waves, pick and pack queues, dock staging, carrier cutoffs and transit. Injects conditions and background noise, including malformed records.
- 02 Deterministic logicNormalizationValidates raw events against a schema, deduplicates aliases and quarantines bad records.
- 03 Stored dataEvent storePGlite (PostgreSQL in WebAssembly) holds the observable world. Ground truth sits in a separate schema that only the evaluation reads.
- 04 Deterministic logicCausal engineDeterministic TypeScript: exception discovery, candidate scoring on eight signals, chain construction, impact tracing, severity and priority.
- 05 InterfaceTyped API and investigation UIException queue, evidence timeline, candidate comparison, causal chain, impact and a rule-by-rule severity breakdown.
- 06 Language modelOptional AI summaryClaude can rewrite a finished result in plain language, subject to a grounding check. It never chooses or ranks the candidate conditions.
The engine imports only the domain model and small utilities. It has no dependency on React, the database or the AI SDK, and a unit test enforces that boundary.
Walkthrough
Example Synthetic scenario from the default demo dataset (seed fcx-demo-2026). Values are as computed by the engine and describe simulated data, not a real incident.
Input
36 shipments missed their planned carrier pickup at node NV-05; delivery promises are at risk.
- 98% of the excess delay accrued in packing
- Carriers involved: MRD, NSP
System processing
The engine scores every condition recorded at NV-05 in the preceding 36 hours and compares exposed orders with unexposed ones.
- The labor system recorded 1 of 7 planned packing staff (staffed ratio 0.14).
- 97% of 76 orders entering packing during the shortfall deviated, against 0% of 437 outside it.
- The packing backlog is ranked as a downstream effect of the shortfall, not an independent candidate.
- A carrier network advisory is set aside because none of its 27 linked orders deviated in transit.
Output
Top-ranked candidate condition: labor shortfall in packing at NV-05, strong evidence (contribution score 0.99, a ranking score rather than a probability).
- Chain of effects traced from that candidate: labor shortfall → pack delayed → carrier cutoff missed → handoff delayed → promise at risk
Next action
Investigation priority P1 (score 86): high severity, with 19 at-risk promises due within 24 hours. The operator's first check is packing staffing at NV-05.
Key engineering decisions
Deterministic, documented scoring ranks the candidate conditions, not a language model.
- Why
- An operator has to be able to check every conclusion against the evidence behind it. Each statement in the explanation is tagged as fact, derived, inference or next step.
- Tradeoff
- The weights and thresholds are hand-set and tested, not learned or calibrated. The contribution score ranks candidates; it is not a probability.
- Where
docs/methodology.md
Alignment works as a soft AND.
- Why
- Timing, entity linkage and process dependency are combined with a geometric mean. A condition that is unlinked, starts too late or constrains the wrong stage then scores near zero, however loud it is.
- Tradeoff
- A condition that genuinely mattered can be under-scored when a link is missing from the recorded data.
"Insufficient evidence" is a first-class answer.
- Why
- Without a moderate or strong candidate, naming one would be a guess. The interface instead reports what was observed, what was set aside and what evidence is missing.
- Tradeoff
- Some real problems stay unexplained, by design.
The engine cannot see the answer key.
- Why
- Ground truth lives in a separate schema that only the evaluation reads, and no scenario identity appears in observable data. An import-boundary test keeps the engine independent of it.
- Tradeoff
- The simulator had to be realistic enough to produce delays on its own, which took more work than scripting them.
Reliability and evaluation
What this measures. These results measure agreement with the simulator's own counterfactual ground truth. They say nothing about accuracy on real operations.
Test conditions
- 40 generated datasets. 10 tuning seeds were used while designing the method; 15 original held-out seeds were read only after thresholds were frozen; 15 fresh held-out seeds were added with the current method and never evaluated before its logic was frozen.
- Seven scenario types, including a planted false lead, concurrent conditions and a case whose correct answer is insufficient evidence.
- Method fcx-causal-1.1, generator fcx-sim-1.2, generated by npm run evaluate on 2026-09-17.
| Measure | Result | Out of |
|---|---|---|
| Top-ranked candidate matches the injected condition (fresh held-out) | 100.0% | 166 exceptions with a known injected condition |
| Asserted candidates that were wrong (fresh held-out; the repo's "false-cause rate") | 1.2% | of 168 candidates asserted |
| Top-ranked candidate matches the injected condition (original held-out) | 98.8% | 172 exceptions with a known injected condition |
| Affected-order recall (fresh held-out) | 94.4% | orders materially harmed per ground truth |
| Severity agreement, exact level (fresh held-out) | 86.1% | exceptions scored for severity |
Observed findings
- Six held-out answers are wrong, and each is listed with its mechanism. Two are schedule-slack cases that the observable data cannot reveal. The others blame a coincident labor dip, mostly through the new small-cluster route.
- Two injected scenarios in the fresh held-out set were not detected, both single-order distractor advisories.
- Severity is the weakest measure. Exact agreement is 82–86%; agreement within one level is 97.6% on both held-out sets.
Source: docs/evaluation-results.md and docs/evaluation.md in the repository
Limitation. A synthetic benchmark can only show that the engine recovers what the simulator injected. Hidden confounders, schedule-margin erosion and conditions that no system records are outside what it can measure.
What I learned
Recall and precision traded against each other in ways the aggregate numbers hid. Version 1.1 kept one- and two-order exceptions when a condition explained them strongly, and split carrier clusters by region. On the original held-out seeds, scenario detection rose from 98.9% to 100% and merged same-carrier incidents fell from 4 to 0. The false-cause rate rose from 0.6% to 1.2%, because one new small exception ranked a nearby labor dip first although it did not materially change the order. I designed the change on tuning seeds only, disclosed the regression, and added a fresh held-out set because the original one had already been read.
Current boundary and next step
Status: Independent prototype · runs locally.
Not yet validated or not built
- Synthetic data only; real-world accuracy is unknown.
- Observational inference. Only recorded conditions can be named, and a coincident recorded condition can be misattributed.
- Simplified operations: single-node fulfillment, one pickup per carrier per day, no weekends, returns or split shipments.
- Not production software. There is no authentication or multi-tenancy, and data lives in memory with file snapshots.
- The optional AI summary was tested against a mocked SDK; a real Claude API call was not verified in this environment.
Source and navigation
- Source code and README Architecture, pipeline, scoring formulas and how to run it locally.
- Methodology Every signal, weight and threshold, and why it was chosen.
- Evaluation and failure analysis Metric definitions, the revision history, and each held-out failure with its mechanism.
- Synthetic data design How the simulator produces delays instead of scripting them.
Talk about this work
Questions about the design, the tradeoffs or the evaluation are welcome.