Build a delightful AI Chatbot: Part 1: Reliable long-term memory

Introduction

At OpenUp, I’m currently building a wellbeing chatbot. Among many, many challenges around usability, safety and capability, I’ve also been exploring and experimenting with different ways to give this chatbot long term memory about a user. It’s a great way to propel your product and make it stand out of the crowd.

The Short Answer

The best way I’ve found so far, is to start with straightforward structured memory. It is tempting to jump to using a vector database for potentially saving a large amount of memory and having intant fuzzy recall, but from lots of user testing I have reached the conclusion that to make a magical product, your bot should remember a few simple facts well.

Three-layer memory model

Layer #1: Structured Memory

This is a simple data record that tracks what you know, or just found out, about the user. It is fast, cheap and easy to access. It’s also easy to surface to your users in a memory management view. It’s malleable, easily updatable via background jobs, and filter-able.

CREATE TABLE user_memories (
    id PRIMARY KEY,
    category TEXT CHECK (category IN (<enums>)), -- Could be anything - 'fact', 'preference', 'goal', 'speaking-pattern'
    key TEXT, -- eg "pet_name"
    value TEXT, -- eg "Casper"
    confidence FLOAT, -- optional, say between 0.0 to 1.0, for extraction quality and RAG confidence
    createAt DATE,
    lastAccessedAt DATE,
    expiresAt DATE, -- optional, eg. for transient memories that should fade
)

In conversation, every few turns, I run a tiny asynchronous extraction prompt against recent messages. This is meant to extract any new facts about a user (with some rules) that may be useful in the future.

[
  {
    "category": "relationship",
    "key": "sister_name",
    "value": "Millie",
    "confidence": 0.95
  },
  {
    "category": "emotion_pattern",
    "key": "morning_anxiety",
    "value": "User feels most anxious on Sunday evenings before the work week",
    "confidence": 0.88
  }
]

It’s certainly a good way to surface factual information when necessary, but also to let a user see what’s stored about them (and let them remove it) - a big win for transparency. Retrieval is instant.

Layer 2: Structured (or Episodic) Memory

For more complex follow ups, or for the agent to actively bring something up by itself (again, seeming like magic ✨), you will need to store memory of specific conversations, maybe even pieced together through multiple conversations. This will need careful tuning, because much of most conversations are probably not worth remembering. This really needs to be dynamic, almost rolling thoughts, where prior conversatoins that are recently surfaced are amended and saved differently.

One nice approach is at the end of a full conversation (if you can automatically detect the end of a user’s conversation, kudos! I still haven’t figured that out…) or asynchronosly multiple times a day from conversation history, generate a short paragraph of a conversation and add it to a time series storage.

Time Series Storage

There’s good reasons to make this data time specific, and time bound to be surfaced again.

The shape here can really be anything, probably the more domain specific the better.

{
    "id:" ""
    "date": "2026-04-21",
    "mood_at_start": "overwhelmed",
    "mood_at_end": "slightly calmer",
    "summary": "User was stressed about an upcoming presentation. We did a box-breathing exercise. They mentioned their manager tends to micromanage. They planned to set one boundary tomorrow.",
    "topics": ["work stress", "boundaries", "breathing"],
    "follow_up_needed": {
        "question": "Did you manage to talk to your manager?",
        "due_date": "2025-04-20",
        "status": "pending"
    }
}

Proactive Follow Up

The nice thing with this is that with a simple cron job or at the start of a new session, the app can look at follow_up_needed entries where the due_date is today.

Something like this would look lovely to a user

“Last time you were here, you were thinking about setting a boundary with your manager. No pressure at all - just wanted to check in if you want to talk about how it went.”

Layer 3: Fuzzy Vector Memory (Long Term)

The best use case I’ve found for vector storage is for specific recollection. These really may not be relevant for your domain, so give this thought before jumping in. You can make this vector store as large as you like (maybe everything the user has ever said. Claude certainly does it)

When the user asks a question like “What did I say about feeling invisible at family dinners?” This is a very specific question, that does not exist in your recent context or in a structured fact table (maybe it will?). A vector search at will let you “remember” deep details about a user.

The implementation is relatively simple: break down the user’s messages (and maybe yoiur agent’s) into chunks, embed and store.

For my use case though, I’ve decided I don’t need it. It’s hard to really know when it’s time to tap into this long term memory, and easy to find irrelevant information. The user is also bound to forget what they were talking about anyway, and it’s tough to make a use case for when this might be useful.

Combining These Steps together

async const recallMemory = (userMessage: string) => {
  // 1. Check structured memory first (instant, factual)
  const facts = await db.select("user_memories").where({ user_id }); // I quite like DuckDB for this

  // 2. Check for any pending follow-ups
  const followUps = await db.select("structured_memories")
                      .where({ user_id, follow_up_status: "pending" }); // or whatever conditions make sense

  // 3. Vector search
  const embedding = await embed(userQuery);
  const episodes = await sqliteVec.search(embedding, { limit: 3 });

  return { facts, followUps, episodes };
}

Bonus: Building a fresh user context

In a new conversation, simply build a small (<500 tokens) memory object. Recent facts, context and maybe long term instruction. Here’s a fun one for a friendly user

You are a helpful assistant, speaking to Milo

Facts:

- They are a dog
- They have trouble sitting still

Recent context:

- On May 18, their owner tricked them into thinking they were going on a walk. They were, in fact, not.
- [Pending follow-up] Ask gently about this if they seem ready.

Tone guidance:

- They prefer short, warm responses. "Sit" is their favorite word