In a controlled demo, watching a 5-step ReAct agent take 18 seconds to produce an answer feels acceptable. Drop that same architecture into a manufacturing procurement line processing 4,000 non-standard RFQs a day, and P95 latency spikes to 26s, retry storms cascade, and token bills explode before week two ends.
This field note breaks down how our Forward Deployed Engineers (FDE) compressed end-to-end P95 latency from 21.4s down to 5.9s (-72.4%) during a three-week production hardening sprint.
Why Production Agents Stall Under Real Load
OpenTelemetry flamegraphs from week one revealed three uncomfortable bottlenecks:
| Pipeline Stage | Baseline P95 | Share | Root Cause |
|---|---|---|---|
| Serial ReAct planning loop | 9.8s | 45.8% | Deterministic SQL & vendor lookups routed through LLM turns |
| Unbounded prompt context | 6.4s | 29.9% | Raw 40-page vendor PDFs injected on every reasoning step |
| Unbounded tool calls | 5.2s | 24.3% | Legacy ERP endpoints called sequentially without timeouts |
The takeaway was unambiguous: 80% of our latency had nothing to do with model speed—we were asking probabilistic inference to execute deterministic business logic.
Principle 1: Extract the Deterministic Spine
We replaced the open-ended ReAct loop with a Deterministic State Graph + Bounded Semantic Nodes:
- Layout-aware PDF table parsing, FX conversion, and 3-year price benchmark queries moved into pure deterministic code.
- LLMs were restricted to the two sub-tasks that genuinely require language understanding: non-standard spec disambiguation and legal clause risk synthesis.
interface RfqNodeResult {
lineItems: NormalizedLineItem[];
riskFlags: ComplianceFlag[];
confidence: number;
}
export async function runDeterministicPipeline(doc: ParsedPdf): Promise<RfqNodeResult> {
// 1. Parallel deterministic fan-out: historical pricing + vendor compliance
const [priceBenchmarks, vendorStatus] = await Promise.all([
queryHistoricalPrices(doc.partNumbers),
verifyVendorCompliance(doc.vendorCode),
]);
// 2. Invoke fast schema-constrained inference only for ambiguous line items
const ambiguousItems = doc.items.filter((item) => item.specConfidence < 0.92);
const resolved = ambiguousItems.length > 0
? await disambiguateSpecsWithSchema(ambiguousItems, priceBenchmarks)
: [];
return mergeAndScore({ doc, priceBenchmarks, vendorStatus, resolved });
}That single architectural shift dropped average LLM calls per RFQ from 6.4 to 1.3, while simultaneously lifting golden-set audit accuracy from 91.2% to 99.1%.
Principle 2: Tiered Routing & Confidence Escalation
Using a 420-ticket anonymized golden dataset, we ablated each node and deployed a three-tier routing policy:
1. Extraction & Normalization Tier (TTFT < 350ms)
High-throughput compact models locked to strict JSON schemas handle multilingual field normalization.
2. Deep Clause Audit Tier (Conditional Escalation)
Only the 14% of RFQs triggering liability-cap or IP-ownership heuristics escalate to a heavyweight reasoning model.
3. Human-in-the-Loop Confidence Gate
Any output scoring below 0.88 confidence skips blind retries and routes directly to a specialist queue—complete with exact source PDF bounding-box highlights.
Principle 3: Chaos Testing Before Cutover
Before switching live traffic, we ran OpenFDE-Bench across 28 injected failure modes—ERP 504 timeouts, upstream 429 rate limits, and skewed scanned PDFs—verifying sub-800ms graceful degradation across every dependency.
Verified Production Delta
| Metric | Initial ReAct Prototype | Hardened Production Graph | Delta |
|---|---|---|---|
| End-to-End P95 Latency | 21.4s | 5.9s | -72.4% |
| Token Spend / Ticket | $0.12 | $0.043 | -64.3% |
| Risk Clause Recall | 91.2% | 99.1% | +7.9% |
| Manual Review Workload | 100% | 22% | -78.0% |
When an AI pilot looks great in slides but chokes in production, put the workflow on a whiteboard and cross out every decision that code can make deterministically. Reliability almost always comes from subtraction.
FDE Bootcamp & Enterprise AI Training
Built for engineering leads and technical teams: production AI Agent architecture, deterministic evals, and forward-deployed delivery playbooks.