A Beginner's Guide to Hindsight: Memory for AI Agents

Artificial intelligence, Data Engineering, Science - thoughts, stories and ideas

27 Aug 2026 • 4 min read

If you've built a chatbot or an AI agent, you've probably run into the same wall everyone does: the moment a conversation ends, the AI forgets everything. Ask it about something you discussed yesterday, and it stares back blankly. That's the problem Hindsight is built to solve.

Hindsight is an open-source memory system for AI agents, built by Vectorize. Instead of just replaying old chat logs back to a model, it's designed to help agents actually learn from experience over time — spotting patterns, building understanding, and getting better at their job the longer they run.

This guide walks through what Hindsight is, how it works, and how to get it running for the first time.

Why Memory Is Hard for AI Agents

Most "memory" in AI products today is really just retrieval. A common approach — often built with RAG (retrieval-augmented generation) or a knowledge graph — stores snippets of past conversation and pulls back the ones that seem relevant to a new question. That works reasonably well for simple recall, but it has a ceiling: the agent is only ever repeating things it was told, not forming any deeper understanding of them.

Hindsight takes a different approach. It's modeled loosely on how human memory organizes information, separating what an agent has been told from what it has experienced, and adding a layer that reflects on both to form higher-level insights. According to the project's benchmark results on LongMemEval — a standard test for long-term conversational memory — Hindsight currently performs at the top of the field, with results reportedly reproduced independently by researchers at Virginia Tech and by The Washington Post.

The Three Building Blocks

Hindsight organizes memory into three types:

World facts — general information about the world, like "the office closes at 6pm."
Experiences — things the agent itself went through, like "I tried that approach and it failed."
Mental models — higher-level understanding the system builds by reflecting on the raw facts and experiences it has collected.

These get stored in a "memory bank," represented as a mix of entities, relationships, and time-based data, indexed for fast retrieval later.

Three Simple Operations

You interact with Hindsight through three core actions:

Retain — feed information in. This is how you tell Hindsight something worth remembering, like a fact about a user or an event that happened. Behind the scenes, an LLM pulls out key facts, entities, and timing details, then organizes them for later use.

Recall — pull information back out. When your agent needs context, Recall searches across memory using several methods at once — semantic vector search, keyword matching, graph traversal for related entities, and time-based filtering — then merges and ranks the results before handing back the most relevant memories.

Reflect — go deeper. Rather than just fetching stored facts, Reflect asks the system to reason over what it knows and produce new insights. This is useful for things like a support agent noticing a recurring gap in documentation, or a sales agent figuring out why some outreach messages land better than others.

Getting Started

The fastest way to try Hindsight is with Docker. You'll need an API key from an LLM provider (OpenAI is the default, but Anthropic, Gemini, Groq, Ollama, LM Studio, and MiniMax are also supported).

export OPENAI_API_KEY=sk-xxx

docker run -it --pull always --name hindsight --restart unless-stopped -p 8888:8888 -p 9999:9999 \
  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest

Once it's running, the API is available at http://localhost:8888 and a web UI at http://localhost:9999.

If you'd rather skip Docker entirely, there's also a Python package that runs Hindsight embedded in your own process, no separate server required:

pip install hindsight-all -U

import os
from hindsight import HindsightServer, HindsightClient

with HindsightServer(
    llm_provider="openai",
    llm_model="gpt-5-mini",
    llm_api_key=os.environ["OPENAI_API_KEY"]
) as server:
    client = HindsightClient(base_url=server.url)
    client.retain(bank_id="my-bank", content="Alice works at Google")
    results = client.recall(bank_id="my-bank", query="Where does Alice work?")

A First Example

Once you have a server running (via Docker or embedded), install the client library:

pip install hindsight-client -U

Then try the three core operations:

from hindsight_client import Hindsight

client = Hindsight(base_url="http://localhost:8888")

# Store something
client.retain(bank_id="my-bank", content="Alice works at Google as a software engineer")

# Ask a question about it
client.recall(bank_id="my-bank", query="What does Alice do?")

# Ask for a deeper, reasoned-out answer
client.reflect(bank_id="my-bank", query="Tell me about Alice")

A Node.js/TypeScript client is available too, if that's more your world.

Adding It to an Agent You Already Have

If you already have an agent calling an LLM directly, Hindsight offers a "wrapper" mode: you swap your existing LLM client for the Hindsight wrapper, and memory gets stored and recalled automatically as your agent runs — no need to manually call retain or recall yourself. If you want finer control over exactly when memories are saved or fetched, the SDKs and REST API give you that directly.

When Should You Actually Use It?

Hindsight is aimed at agents that need to do more than just chat — things like AI employees handling open-ended tasks, adjusting behavior based on feedback, or getting better at a job over repeated use. A simple, single-purpose automation (say, something built in a no-code tool like n8n) probably doesn't need this level of memory infrastructure. But for agents meant to run continuously, remember individual users, and improve over time, it's built for exactly that.

A common starting use case is per-user memory in a chatbot: storing facts tied to a specific user via metadata, then filtering recall so each person only gets their own memories back — useful for personalizing conversations across sessions without mixing up different users' information.

Where to Go Next

Docs: hindsight.vectorize.io
Cookbook (examples): hindsight.vectorize.io/cookbook
Community Slack: linked from the GitHub repo
Source code: github.com/vectorize-io/hindsight

Hindsight is MIT-licensed and actively developed, with frequent releases. If you're building an agent that needs to remember — and actually improve — rather than just recall, it's worth a look.