Enterprise AI: The Automation Evolution Enterprises Skip - Manual Process to API Integration to AI-Assisted Automation
Why AI-automates-everything pilots stall: they skip the scripted/API stage that fixes the data contract, baseline error rate and logging. 3 worked examples.
The pitch deck says “AI will automate the invoice process”. The process in question is a shared mailbox, a spreadsheet of vendor-to-account mappings that lives on one clerk’s laptop, and a monthly reconciliation that finds the mistakes. Nobody has measured how many mistakes. There is no API on either end. And the plan is to put a language model in the middle of it.
I have now watched three of these pilots stall at the same place, in three different organisations, for the same reason. They jumped from stage 1 to stage 3 of an evolution that has three stages, and the stage they skipped is the one that makes the other two measurable:
- Manual process. People, mailboxes, spreadsheets, tacit rules.
- Scripted / API integration. The process gets a data contract, deterministic rules, idempotent writes, logs and a baseline error rate.
- AI-assisted automation. A model handles the part the rules cannot, behind the same contract and the same logging, with a human on the low-confidence path.
This post is about what stage 2 actually buys you, why skipping it makes stage 3 fail quietly, and what the three stages looked like for the three workflows this blog has covered most: vendor invoice line coding, support ticket routing, and code review.
The three misconceptions behind the jump
“AI is the integration”
The most expensive misunderstanding is that the model replaces the plumbing. It does not. A model can read an invoice PDF and produce a JSON object, but somebody still has to decide which fields that object has, which system of record receives it, what happens when the same invoice arrives twice, and who gets paged when the write fails. Those are integration questions, and they have the same answers whether a model or a regex produced the JSON.
In the invoice pilot that skipped stage 2, the team spent eleven weeks on prompt engineering and two days on the write-back. The write-back was a CSV import the clerk ran by hand. When the model produced a main account that did not exist in the chart of accounts (it did, about 3 % of the time), the import silently created a line with an empty dimension and the monthly reconciliation found it four weeks later. The model was blamed. The missing piece was the enum of valid values in the structured output contract, which is a stage 2 artefact: you only know the valid value set once you have an API that can give it to you.
“We can skip the data contract because the model is flexible”
The flexibility of a language model is precisely why you need a stricter contract around it, not a looser one. A rules engine fails loudly on an unexpected input; a model produces a plausible answer. If the process has never been scripted, nobody has written down what “a valid output” means, so there is nothing to validate the plausible answer against.
Stage 2 forces that definition. To write the script that codes 68 % of invoice lines from vendor history, you have to define: which entity and which fields, which values are legal, what “the same invoice” means (vendor + invoice number + date, as it turned out, not the document hash), and what the script does when it is not sure (nothing, and a flag). Every one of those definitions is reused unchanged when the model replaces the rules on the remaining 32 %.
“We will add observability later”
In stage 1 the error rate is unknown. In a stage-3-without-stage-2 pilot it is still unknown, except now the errors are generated faster and look more confident. The question “is the AI better than the clerk?” cannot be answered because there is no number for the clerk.
The pilots that worked all had the same boring property: by the time the model was switched on, every decision in the workflow already emitted a log line with an id, an input hash, the rule or model that decided, the decision, and whether a human later changed it. That is the LLM observability setup, but the point is that it existed before the LLM and was built for the scripts. The model inherited it.
Three workflows through the three stages
Vendor invoice line coding (D365FO)
Manual. Clerk picks main account, cost centre, department and project from memory, 41 s median per line. Nobody knew the error rate; a sample of 400 posted lines against the auditor’s corrections gave 4.1 % wrong account or cost centre.
Scripted / API. An Azure Function reads pending VendorInvoiceLines over OData and applies two deterministic rules: if the last 20 posted lines for this vendor all share the same coding for the same description prefix, propose it; if the vendor has a contract with a fixed project, fill the project. That covered 68 % of lines with 1.2 % errors, every proposal written back as a draft with PATCH, every write logged with the rule id. Six weeks, including fixing the OData paging and the idempotency key.
AI-assisted. The model only sees the 32 % the rules leave behind, with the same schema, the same enum of valid dimension values and the same draft-then-approve write-back. The result reported in the original post - 86 % accepted unchanged, 9 s per line, $0.0021 per line - is the stage 3 number on top of the stage 2 pipeline. Four weeks, because almost nothing new had to be built.
The pilot that skipped the middle took eleven weeks to reach a less reliable version of this and was stopped by finance.
Support ticket routing
Manual. A triage queue, 3.5 h median from ticket creation to first assignment, 22 % of tickets re-routed at least once. The 22 % was discovered only when we started counting.
Scripted / API. Rules on the ticketing system’s webhook: customer tier, product field, and a keyword list the support leads maintained anyway in a wiki. 55 % auto-routed, re-route rate down to 18 %, and - the important bit - every routing decision stored on the ticket as a custom field with the rule id, so re-routes could be attributed to a rule.
AI-assisted. A classifier (a small Azure OpenAI prompt with the team list as an enum) on the 45 % the rules could not place. 91 % auto-routed, 9 % re-routed, 25 min median to assignment. When re-routes spike, the stored decision field tells us whether a rule or the model was wrong, which is how we caught a product rename that broke two keyword rules and one prompt example in the same week.
Code review
Manual. Human-only review, 18 h median wait for a first review on the main repository, no shared taxonomy for what a review comment was about.
Scripted / API. The unglamorous stage: linters, SAST, a coverage gate and a conventional-commit check in CI, each finding tagged with a rule id and posted through the same review-comment API a human would use. Around 40 % of review comments became automated, and because they were tagged we knew which rules developers routinely dismissed.
AI-assisted. An agent reviewing logic and test gaps, on top of those gates, measured in this post and gated before merge here. 11 h median wait and 31 % fewer escaped defects. The agent’s comments carry the same tag field as the linter findings, so the dismissal rate dashboard did not need a second version.
What stage 2 removes before the model shows up
Looking across the three, the scripted stage removed the same four things each time:
- The unknown baseline. You get a measured error rate and latency for the manual process as a side effect of instrumenting the script. Without it, “the AI is 86 % accurate” is not a result, it is a number.
- The easy majority. Between 40 % and 68 % of the volume was deterministic. Sending that through a model costs tokens, adds latency and, worse, adds a non-zero chance of a confident wrong answer to cases that had a right answer available by lookup.
- The undefined contract. Entity, fields, legal values, idempotency key, failure behaviour. The model’s JSON schema is the contract you wrote for the script, with a
ConfidenceandReasonfield appended. - The missing feedback loop. “Did a human change this decision?” is the only training signal that matters in production, and it only exists if decisions are stored with an id the human’s correction can be joined to.
When to skip, honestly
Jumping straight to the model is defensible in two situations. First, when the input is unstructured and there is genuinely no deterministic majority - free-text contract clauses, for instance - so stage 2 would be a contract and logging layer with no rules in it. Build that layer anyway; it is two weeks, not six. Second, when the process is low volume and low stakes enough that the “observability” can be a human looking at every output, which is a stage 3 pilot in name only.
Everywhere else, the sequencing is the project. The model is the smallest part of it, and in all three workflows above it arrived last, cost the least, and was the only part that looked good on a slide.