Case study
An architecture case study: LLM-assisted reconciliation with deterministic correctness.
Finance teams reconcile transaction data across systems that were never designed to agree. The work is high-volume, low-tolerance: thousands of rows, where a single silent mismatch can cost hours of downstream investigation or a misstated ledger.
The obvious move โ hand the rows to an LLM โ fails in two predictable ways. Cost scales linearly with volume, and correctness becomes probabilistic exactly where it must be absolute.
The system I built keeps the LLM off the data path entirely. It works like this:
Sample. From the input files, extract a statistically significant random sample of rows. This sample โ not the dataset โ is what the model sees.
Parse and allocate. The LLM reads the sample and does one job: it works out the structure of the reconciliation. What the columns mean, how the two sides correspond, and which rules from a small domain-specific language apply โ exact match here, amount tolerance there, windowed date logic on this pair. The model's output is not matches. It is a rule allocation.
Execute deterministically. The allocated DSL rules then run across all rows โ every one, not the sample โ in plain deterministic code. Every match carries full traceability: what matched what, why, and which rule made the decision. When someone asks "why did these two rows pair?", the answer is a named rule, not a probability.
The cost consequence falls out immediately: the LLM's work is a function of sample size, which is fixed, not file size, which isn't. A 500-row file and a 500,000-row file cost roughly the same to reconcile. That is the difference between a demo and a business.
Here is the part that took building the system to learn.
Early on, the goal was 100% match rates. Then it became clear that a system reporting 100% completion, 100% of the time, is almost certainly hiding errors โ because in real financial data, genuine ambiguity always exists. A tool that never says "I don't know" hasn't eliminated uncertainty; it has laundered it into false matches, which are the most expensive kind.
So the system was redesigned around an explicit confidence boundary. Rules only fire where they are genuinely decisive. Everything below the threshold surfaces as a first-class exception queue โ visible, annotated, human-reviewed. The residual isn't a limitation being apologised for. It is the system telling the truth about where truth runs out.
This inverts how most AI tools are evaluated. The question isn't "what percentage did it complete?" It's "is the boundary between known and unknown honest, explicit, and in the right place?" A 96% match rate with a trustworthy 4% exception queue is worth more than a 100% rate you can't audit โ because the first number describes your data, and the second describes your tool's willingness to guess.
This is one instance of a pattern I apply everywhere: LLMs operate at the fuzzy boundaries; deterministic systems hold correctness. The model interprets messy structure and proposes rules โ the tasks it's genuinely good at. The deterministic layer executes, traces, and decides โ the tasks that demand exactness.
Most AI tools that demo well and fail in production fail because the model is in the middle, and because they smooth over the confidence boundary instead of exposing it. The fix is rarely a better model. It is a better boundary โ and the honesty to show where it sits.
If you're diagnosing an AI tool that isn't delivering, start with two questions: what does the model actually own, and where does the system admit it doesn't know? I write more about this in Mo' Automation, Mo' Irony.