A practical, low-latency architecture for real-time voice AI. Separating the
live call path from the control plane and the post-call systems. Providing security, scaleability and observability.
Reading guide: Layers 01–05 are on the live call path.
Layer 06 is the control plane that configures how the live path behaves.
Layers 07–08 are mostly async, safety and operational systems that support the
runtime without slowing down the caller experience.
Security 9/10
Latency 8.5/10
Observability 9/10
Scalability 8/10
Runtime Maturity 8.5/10
Cost Control 8/10
Live path
Everything the caller feels directly: audio ingest, speech detection, routing, model response and playback.
Control plane
Configuration that shapes runtime behavior: prompts, policies, tenant settings, budgets and rollout control.
Async systems
Evaluation, audits, analytics and synthetic checks that happen beside or after the call instead of on the critical path.
Key principle
The caller should never wait for things that do not need to be synchronous. Silence is failure in voice systems.
CLICK ON CARDS TO VIEW DETAILSeo
01 · Realtime Audio Ingress & Turn Control
LIVE PATH · AUDIO IN → TURN DETECTION
This layer is responsible for bringing live audio into the system, keeping it stable,
and deciding when the user has started or finished a turn. In voice systems, this is where
a lot of the “feels magical” experience is won or lost.
↓
02 · Security & Policy Boundary P0
EVERY TURN PASSES HERE BEFORE MODEL EXECUTION
This boundary exists to stop unsafe, abusive or policy-breaking content before it reaches the model or is written to logs.
It is much cheaper to block bad input here than to let it leak deeper into the system.
↓
03 · Realtime Understanding & Routing
FAST PATH · CHEAP AND DETERMINISTIC
The goal of the fast path is simple: do not wake up expensive reasoning if a lightweight rule or small model can make the decision safely.
DEEP PATH · WHEN UNDERSTANDING MATTERS
When the turn is ambiguous or multi-step, the system uses a stronger model to classify intent and choose the safest route.
↓
04 · Domain Context & Tool-Oriented Orchestration
LIVE PATH · CONTEXT ASSEMBLY BEFORE REASONING
The model should not have to guess the business context from scratch. This layer identifies domain entities,
injects tenant-specific rules and prepares the request so the orchestrator sees the right facts at the right time.
↓
05 · Response Execution P0
LIVE PATH · MODEL, TOOLS, TTS, INTERRUPTION HANDLING
This is the heart of the runtime. It chooses the right model path, runs tools if required and streams a response back quickly.
In a voice system, interruption handling matters almost as much as answer quality.
↓
06 · Control Plane P1
NOT ON THE CRITICAL PATH · SHAPES RUNTIME BEHAVIOR
These systems do not answer the caller directly, but they control how the live runtime behaves. This separation makes the
architecture easier to operate and safer to change.
↓
07 · Knowledge, Actions & Human Safety Nets
KNOWLEDGE RETRIEVAL
Retrieval should be fast, selective and bounded. The goal is to improve correctness, not dump documents into context.
SYSTEM ACTIONS
Tools are where the system stops being a chatbot and starts becoming operational infrastructure. That power needs strong limits.
HUMAN SAFETY NET
The system should know when not to continue. Good escalation is a sign of maturity, not weakness.
These systems keep the platform trustworthy over time. They are what let operators answer hard questions such as
“what failed?”, “what changed?”, “how much did it cost?” and “is quality drifting?”