We Built a Persistent Agent Memory Layer on Elasticsearch with 0.89 Recall: What Product Builders Need to Know

TL;DR

• Persistent memory is the missing infrastructure layer for production AI agents: Most agent frameworks treat memory as ephemeral or store it in simplistic key-value stores, but Elasticsearch's semantic search capabilities enable agents to recall relevant context across sessions with 0.89 recall rates.
• The memory layer architecture requires three components: A storage backend (Elasticsearch), an embedding model for semantic search, and a retrieval strategy that balances recency, relevance, and relationship context—not just vector similarity.
• 0.89 recall means your agent remembers 89% of relevant past interactions: This benchmark, achieved through careful tuning of retrieval parameters and index configuration, represents a practical threshold where agents feel "contextually aware" rather than perpetually forgetful.
• Product builders should prioritize memory architecture early: Retrofitting memory into an existing agent is significantly harder than designing it from the start, and the choice of memory backend fundamentally shapes what kinds of agent behaviors you can support.

The Memory Problem Nobody Talks About

We're building increasingly sophisticated AI agents—systems that can book flights, manage customer support tickets, and even write code. But there's a fundamental problem that most teams discover too late: agents without memory are just expensive chatbots.

Think about it. You wouldn't hire a customer service representative who forgot every conversation the moment it ended. You wouldn't work with a personal assistant who needed you to re-explain your preferences every single day. Yet that's exactly how most AI agents operate today.

The team at Elastic recently published their approach to building a memory layer for AI agents using Elasticsearch, and it's one of the most practical explorations of this problem I've seen. They achieved 0.89 recall—meaning their agent successfully retrieved relevant past context 89% of the time. That number matters because it represents a threshold where memory stops feeling like a party trick and starts feeling like actual continuity.

Let me break down what they built, why it matters, and what you should consider if you're building agents that need to remember.

What Is Agent Memory, Really?

Before we dive into implementation details, let's establish what we mean by "agent memory." This isn't about expanding context windows or caching recent messages. Agent memory is a persistent, searchable store of past interactions, decisions, and learned preferences that an agent can query to inform future actions.

There are roughly three types of memory that matter for agents:

Short-term (Working) Memory

This is your conversation context—the last few exchanges that sit in the LLM's context window. Every agent has this by default. It's fast, it's simple, but it evaporates the moment the session ends.

Episodic Memory

Specific past interactions: "The user asked about pricing on March 15th," or "We discussed their preference for email over Slack notifications." This is what most people mean when they talk about agent memory. The Elasticsearch implementation focuses heavily here.

Semantic Memory

Broader knowledge and patterns: "This user typically prefers technical documentation over video tutorials," or "When this customer says 'urgent,' they mean within 2 hours, not 24." This requires aggregation and pattern recognition across many episodes.

The Elastic team's approach addresses episodic memory first—and that's the right call. You need a solid foundation for storing and retrieving specific interactions before you can build higher-order semantic understanding.

The Architecture: More Than Just Vector Search

Here's where things get interesting. The naive approach to agent memory is to embed every conversation, throw it in a vector database, and retrieve the top-k most similar past interactions when needed. The Elastic implementation is more sophisticated.

Storage Layer: Why Elasticsearch?

Elasticsearch isn't the obvious choice for agent memory—purpose-built vector databases like Pinecone or Weaviate seem more natural. But Elasticsearch brings something crucial: hybrid search.

You can combine dense vector similarity (semantic meaning) with sparse retrieval (keyword matching) and traditional database filters (time ranges, user IDs, conversation types). This matters because relevant memories aren't always semantically similar to your current query.

Imagine an agent helping with expense reports. The user asks: "What did I spend on that Chicago trip?" The semantically similar memories might be other questions about trips or spending. But the relevant memory is the specific conversation about Chicago—which might be better retrieved through keyword matching on "Chicago" plus a time filter for recent business travel.

The Elastic team's architecture leverages this hybrid approach, and their 0.89 recall rate suggests it's working.

Embedding and Retrieval Strategy

The implementation uses embedding models to convert conversations into vectors, but the retrieval strategy is where the real engineering happens. They're not just doing a simple k-nearest-neighbor search.

According to their approach, retrieval considers:

1. Semantic similarity: How closely does this past memory match the current query's meaning?
2. Recency: More recent interactions are weighted higher (with decay functions)
3. Relationship context: Memories connected to the same user, project, or thread get boosted

This multi-factor retrieval is crucial. Pure semantic search often surfaces memories that sound similar but aren't actually relevant. Pure recency bias means you ignore important but older context. The magic is in the combination.

The 0.89 Recall Benchmark

Let's talk about that 0.89 recall number. In information retrieval, recall measures what percentage of relevant items you successfully retrieved. If there are 10 past conversations that should inform the current interaction, and your system retrieves 9 of them, that's 0.90 recall.

0.89 is genuinely impressive for a production system. It means the agent is missing about 11% of relevant context—which sounds bad until you realize that humans probably do worse. I certainly don't remember every relevant past conversation when talking to a colleague.

My take: 0.89 recall is the threshold where memory becomes useful rather than frustrating. Below about 0.75, users notice the gaps constantly ("Didn't I already tell you this?"). Above 0.90, you're hitting diminishing returns—the engineering effort to improve recall further rarely justifies the marginal benefit.

The real question for product builders isn't whether 0.89 is "good enough" in the abstract. It's whether your specific use case requires higher recall, and whether you're willing to accept the complexity and latency trade-offs that come with it.

Practical Implications for Product Builders

If you're building AI agents, here's what this implementation teaches us:

Start with Memory Architecture, Not Memory as an Afterthought

The biggest mistake I see teams make is building an agent, getting it working with short-term context, and then trying to bolt on memory later. This is backwards.

Your memory architecture shapes fundamental product decisions:

• What can your agent promise to remember? (User preferences? Past decisions? Entire conversation histories?)
• How does memory affect latency? (Every retrieval adds milliseconds to response time)
• What happens when memory conflicts with current input? (User says they prefer email, but past memory says they prefer Slack)

Design these answers into your system from day one. The Elastic team clearly did—their architecture assumes memory queries are part of every agent interaction, not an optional enhancement.

Choose Your Storage Backend Based on Query Patterns, Not Hype

Vector databases are trendy right now, but they're not always the right choice. Ask yourself:

• Do you need hybrid search (combining semantic and keyword)?
• Do you need complex filtering (time ranges, user attributes, conversation metadata)?
• Do you need to update or delete specific memories (for privacy/compliance)?
• What's your scale? (Millions of memories? Billions?)

Elasticsearch makes sense for the team's use case because they need all of the above. If you're building something simpler—say, a personal AI assistant with a single user and a few hundred memories—you might be better off with a simpler stack.

I think too many teams reach for the most sophisticated infrastructure before understanding their actual requirements. Start with the simplest thing that could work, measure it, then upgrade if needed.

Measure What Matters: Recall, Precision, and Latency

The 0.89 recall number is useful because it's measurable and meaningful. But you need to track three metrics:

1. Recall: What percentage of relevant memories did you retrieve?
2. Precision: What percentage of retrieved memories were actually relevant?
3. Latency: How long does memory retrieval add to response time?

These metrics trade off against each other. You can boost recall by retrieving more memories, but precision drops (more noise) and latency increases. The Elastic implementation found a sweet spot, but your sweet spot might be different.

For a customer support agent, you might tolerate higher latency for better recall—missing context is worse than waiting an extra second. For a coding assistant, you might optimize for lower latency even if it means slightly lower recall.

Handle Memory Conflicts Explicitly

Here's a scenario the Elastic article doesn't deeply explore, but you'll encounter immediately: What happens when memories contradict each other or conflict with current input?

User memory from last month: "I prefer detailed technical explanations." User input today: "Just give me the quick version."

Your agent needs a strategy. Options include:

• Recency wins: Always prioritize current input over past memory
• Explicit override: Ask the user to confirm when memory conflicts with input
• Context-dependent: Use different strategies for different types of information

I lean toward recency wins for preferences and explicit override for factual information. But this is a product decision, not a technical one. Your memory architecture needs to support whatever strategy you choose.

The Bigger Picture: Memory as Product Differentiation

Here's what excites me about persistent agent memory: it's becoming a genuine product differentiator.

Right now, most AI products compete on model quality ("We use GPT-4!" "We fine-tuned Claude!"). But as foundation models commoditize, the differentiation shifts to everything around the model. Memory is a huge part of that.

An agent that remembers your preferences, learns from past interactions, and builds continuity across sessions feels fundamentally different from one that starts fresh every time. It's the difference between talking to a colleague versus talking to a stranger.

The Elastic team's work shows that building this kind of memory is tractable today. You don't need to wait for some future breakthrough. The tools exist, the patterns are emerging, and the benchmarks (like 0.89 recall) give you targets to aim for.

Implementation Considerations and Trade-offs

Let's get tactical. If you're implementing something similar, here are the key decisions:

Embedding Model Selection

You need an embedding model that balances quality and speed. The Elastic implementation doesn't specify which model they used, but common choices include:

• OpenAI's text-embedding-3: High quality, reasonable cost, API latency
• Open-source models (e.g., sentence-transformers): Lower cost, more control, self-hosting required
• Domain-specific models: Better performance if you fine-tune for your use case

For most product builders, I'd start with OpenAI's embeddings. They're good enough, easy to integrate, and you can always swap them out later if you need better performance or lower costs.

Index Configuration

Elasticsearch (and similar systems) require careful index configuration. Key parameters:

• Vector dimensions: Match your embedding model (1536 for OpenAI text-embedding-3-small)
• Similarity function: Cosine similarity is standard, but dot product can be faster
• Number of shards: Affects search speed and scalability
• Refresh interval: How quickly new memories become searchable

The Elastic team's configuration prioritizes search speed and recall. If you're operating at smaller scale, you might simplify.

Memory Lifecycle Management

Memories aren't static. You need policies for:

• Retention: How long do you keep memories? Forever? 90 days? User-configurable?
• Updates: What happens when information changes? (User moves cities, changes preferences)
• Deletion: How do you handle "forget this" requests? (Privacy, compliance)
• Summarization: Do you compress old memories to save space?

These aren't just technical questions—they're product and legal questions. GDPR gives users the "right to be forgotten." Your memory architecture needs to support that.

What This Means for the Future of Agents

The Elastic team's work is part of a broader shift: agents are becoming stateful systems, not stateless functions.

Early AI assistants were essentially fancy APIs—you sent a prompt, got a response, done. Modern agents maintain state across interactions. They remember, they learn, they build context over time.

This changes what's possible:

• Personal AI that actually knows you: Not just "AI assistant," but "your AI assistant" in a meaningful sense
• Agents that improve with use: Every interaction makes future interactions better
• Collaborative AI: Multiple users sharing context through a common agent memory

But it also introduces new challenges:

• Privacy and security: Persistent memory means persistent risk if breached
• Bias and drift: Agents might reinforce past patterns even when they should adapt
• Transparency: Users need to understand what the agent remembers and why

I think we're at the beginning of figuring out these challenges. The technical foundations—like the Elasticsearch memory layer—are being built now. The product and ethical frameworks will take longer.

Getting Started: A Practical Roadmap

If you're convinced that your agent needs persistent memory, here's how to start:

Phase 1: Define Your Memory Requirements

• What types of information should your agent remember?
• How long should memories persist?
• What recall/precision targets make sense for your use case?
• What's your acceptable latency budget?

Phase 2: Choose Your Stack

• Elasticsearch if you need hybrid search and complex filtering
• Purpose-built vector DB (Pinecone, Weaviate, Qdrant) if you're doing pure semantic search
• Simpler solutions (Postgres with pgvector) if you're starting small

Phase 3: Implement and Measure

• Start with a simple retrieval strategy (top-k semantic search)
• Instrument everything (log queries, retrievals, relevance)
• Measure your actual recall/precision with real users
• Iterate based on data, not intuition

Phase 4: Optimize

• Add hybrid search if semantic search alone isn't working
• Tune recency weighting based on your domain
• Implement memory lifecycle policies
• Consider summarization or compression for scale

The Elastic team's 0.89 recall didn't happen by accident. It's the result of careful architecture, thoughtful retrieval strategies, and iterative optimization. You can achieve similar results, but it requires treating memory as a first-class system component, not an afterthought.

Final Thoughts

Persistent agent memory is moving from research curiosity to production requirement. The Elastic team's implementation proves that you can build practical, high-recall memory systems with existing tools.

But here's my honest take: most teams will underestimate the complexity of memory until they try to build it. It's not just about storing and retrieving vectors. It's about handling conflicts, managing lifecycle, balancing trade-offs, and making product decisions about what your agent should and shouldn't remember.

The good news? We're past the point where you need to invent everything from scratch. Patterns are emerging, benchmarks are being established (like that 0.89 recall), and infrastructure is maturing. The Elastic implementation is one example; there will be many more.

If you're building AI agents, start thinking about memory now. Not next quarter, not after you've validated product-market fit. Now. Because the agents that win won't just be the ones with the best models—they'll be the ones that remember.