In-Process Retrieval Turns RAG Into Agent Working Memory
Networked vector stores are the quiet bottleneck inside every language-agent loop. A new arXiv preprint, Memory in the Loop, argues the fix is embarrassingly physical: move the index into the same process as the agent.
In-Process Retrieval Turns RAG Into Agent Working Memory
Networked vector stores are the quiet bottleneck inside every language-agent loop. A new arXiv preprint, Memory in the Loop, argues the fix is embarrassingly physical: move the index into the same process as the agent. Their in-process store answers in roughly 100 microseconds — three orders of magnitude faster than a networked call — and pushes retrieval into the observe-reason-act cycle instead of parking it as an offline preprocessing stage.
The engineering case is direct. When retrieval costs 80–200 ms per hop over gRPC, agents avoid asking. When it costs 100 µs, the agent can query memory between every reasoning step without noticing. That changes what “context” means: instead of one bulk retrieval before the prompt, memory becomes something the model consults on demand, the way a human consults a notebook mid-sentence.
What the numbers actually show
The paper reports recall on a synthetic multi-hop task climbing from 0/5 to between 3.6 and 4.8 out of 5 when GPT-5-class models are given in-loop access to the vector store. Redundant tool calls drop — the agent stops re-fetching what it already has — because the price of “let me check again” collapses.
The new dominant cost surfaces immediately: the embedder. A remote embedding API call now dwarfs the retrieval itself. The authors mitigate this by pairing the in-process index with a small local embedder, landing the full embed-plus-search cycle around 40 µs.
retrieval:
mode: in_process
index: hnsw
vectors: 2_400_000
dim: 384
embedder:
kind: local
model: bge-small-en-v1.5
device: cpu
p50_us: 38
p99_us: 71
store:
p50_us: 96
p99_us: 210Architecture implications for agent teams
Two design consequences follow. First, the agent runtime and the index must share address space, which rules out serverless FaaS deployments where cold-start amortization kills the µs budget. Second, the embedder becomes a first-class latency SLO — not a “we’ll swap it later” component. Teams building production RAG systems should treat embedder residency the way databases treat buffer-pool sizing.
There is a security angle the paper does not address. An in-process store shares the failure domain of the agent: prompt injection that reaches the retrieval layer can now write, not just read. Guardrails that assumed a network hop between agent and memory need re-examination in light of retrieval-layer threat models documented by ENISA’s AI threat landscape and NIST’s AI Risk Management Framework.
Where CAI Technology sits
We build agent runtimes for regulated Romanian mid-market clients under Directive (EU) 2022/2555 obligations. In-process memory is faster, but it collapses the audit boundary between “the model” and “the knowledge base” — a boundary that NIS2 auditors care about. Our position: in-loop retrieval is the right latency choice, but it demands per-call retrieval logging that survives outside the process. Speed without a paper trail is a liability, not a feature. If you are pricing this trade-off for a live agent, talk to our RAG engineering team.