JIANCHENG STUDIOJC-01
Back to Insights
NOTE-01Published 2026-11-12Read time 8 min

FDE Field Note: How We Cut Production AI Agent Latency by 72%

Inside a real global supply-chain deployment: replacing unconstrained ReAct loops with deterministic state graphs, tiered model routing, and chaos benchmarks.

FDEAI AgentLatency

In a controlled demo, watching a 5-step ReAct agent take 18 seconds to produce an answer feels acceptable. Drop that same architecture into a manufacturing procurement line processing 4,000 non-standard RFQs a day, and P95 latency spikes to 26s, retry storms cascade, and token bills explode before week two ends.

This field note breaks down how our Forward Deployed Engineers (FDE) compressed end-to-end P95 latency from 21.4s down to 5.9s (-72.4%) during a three-week production hardening sprint.

Why Production Agents Stall Under Real Load

OpenTelemetry flamegraphs from week one revealed three uncomfortable bottlenecks:

Pipeline StageBaseline P95ShareRoot Cause
Serial ReAct planning loop9.8s45.8%Deterministic SQL & vendor lookups routed through LLM turns
Unbounded prompt context6.4s29.9%Raw 40-page vendor PDFs injected on every reasoning step
Unbounded tool calls5.2s24.3%Legacy ERP endpoints called sequentially without timeouts

The takeaway was unambiguous: 80% of our latency had nothing to do with model speed—we were asking probabilistic inference to execute deterministic business logic.

Principle 1: Extract the Deterministic Spine

We replaced the open-ended ReAct loop with a Deterministic State Graph + Bounded Semantic Nodes:

  • Layout-aware PDF table parsing, FX conversion, and 3-year price benchmark queries moved into pure deterministic code.
  • LLMs were restricted to the two sub-tasks that genuinely require language understanding: non-standard spec disambiguation and legal clause risk synthesis.
interface RfqNodeResult {
  lineItems: NormalizedLineItem[];
  riskFlags: ComplianceFlag[];
  confidence: number;
}
 
export async function runDeterministicPipeline(doc: ParsedPdf): Promise<RfqNodeResult> {
  // 1. Parallel deterministic fan-out: historical pricing + vendor compliance
  const [priceBenchmarks, vendorStatus] = await Promise.all([
    queryHistoricalPrices(doc.partNumbers),
    verifyVendorCompliance(doc.vendorCode),
  ]);
 
  // 2. Invoke fast schema-constrained inference only for ambiguous line items
  const ambiguousItems = doc.items.filter((item) => item.specConfidence < 0.92);
  const resolved = ambiguousItems.length > 0
    ? await disambiguateSpecsWithSchema(ambiguousItems, priceBenchmarks)
    : [];
 
  return mergeAndScore({ doc, priceBenchmarks, vendorStatus, resolved });
}

That single architectural shift dropped average LLM calls per RFQ from 6.4 to 1.3, while simultaneously lifting golden-set audit accuracy from 91.2% to 99.1%.

Principle 2: Tiered Routing & Confidence Escalation

Using a 420-ticket anonymized golden dataset, we ablated each node and deployed a three-tier routing policy:

1. Extraction & Normalization Tier (TTFT < 350ms)

High-throughput compact models locked to strict JSON schemas handle multilingual field normalization.

2. Deep Clause Audit Tier (Conditional Escalation)

Only the 14% of RFQs triggering liability-cap or IP-ownership heuristics escalate to a heavyweight reasoning model.

3. Human-in-the-Loop Confidence Gate

Any output scoring below 0.88 confidence skips blind retries and routes directly to a specialist queue—complete with exact source PDF bounding-box highlights.

Principle 3: Chaos Testing Before Cutover

Before switching live traffic, we ran OpenFDE-Bench across 28 injected failure modes—ERP 504 timeouts, upstream 429 rate limits, and skewed scanned PDFs—verifying sub-800ms graceful degradation across every dependency.

Verified Production Delta

MetricInitial ReAct PrototypeHardened Production GraphDelta
End-to-End P95 Latency21.4s5.9s-72.4%
Token Spend / Ticket$0.12$0.043-64.3%
Risk Clause Recall91.2%99.1%+7.9%
Manual Review Workload100%22%-78.0%

When an AI pilot looks great in slides but chokes in production, put the workflow on a whiteboard and cross out every decision that code can make deterministically. Reliability almost always comes from subtraction.

TRAINING & BOOTCAMP // WAITLIST

FDE Bootcamp & Enterprise AI Training

Built for engineering leads and technical teams: production AI Agent architecture, deterministic evals, and forward-deployed delivery playbooks.