Memory is all you need? (for an AI coach)

When ChatGPT was still new one of the common frustrations users experienced was that every conversation with started from zero. It felt like talking to a goldfish: you had to explain yourself over and over again every time you opened a new chat. It didn’t take too long before most LLM chat apps started adding simple memory functions - allowing the AI to remember key things about you. Over time this builds your experience into a much more personalised one - so much so that I think that your memories in ChatGPT or Claude is now the main reason people are reluctant to switch tools.

When we started building Friday one of our first decisions was that Friday needed the best possible memory we could build for it. As a creative coach, poor memory is fatal - a coach who doesn’t remember you isn’t much of a coach. It’s just a chatbot with a nice tone, and every session would be a first session. And first sessions are mostly warm-up. The good coaching - the kind where someone pushes you past a stuck point because they know your patterns - only happens when there’s accumulated understanding.

So we built a memory system. Here’s how it works, what we got wrong, what surprised us, and where we are heading next.

What “memory” means for a coach

There’s a difference between recording what was said and understanding what matters. A good human coach doesn’t replay your last session word for word. They hold your intentions, your patterns, what you’re reaching toward. They know, for example, that you always get stuck when the emotional stakes get high. They know if you work with images (as opposed to, say, text), so they frame feedback visually. They know you’ve been circling around the same theme for three sessions.

Friday has to be able to do the same thing. Not store journal notes of the last session - but hold an actual understanding of where you are and where you are heading on your coaching journey.

The naive approach, and why it didn’t work

Our obvious first idea: make a summary of every conversation you have with Friday and stuff those into the LLM’s context window. Every previous message, every exchange, everything.

This failed for three reasons. Cost… every token you send to the LLM costs money, and sending 50 sessions of summarised conversation history into your next conversation with Friday gets expensive, fast. Quality… language models actually get worse when you overload them with too much context. The signal gets buried in noise. They lose focus. And probably most importantly, relevance: most of what was said three sessions ago doesn’t matter right now. What matters is the insight that came out of that session and your intentions moving forward.

What we built instead

Friday’s memory has three tiers, each serving a different purpose.

Tier 1: Your profile. This is identity-level information - who you are, what you’re working on, your creative background, your preferences. Stuff that rarely changes over time. Think of it as a card that Friday always sees at the start of every conversation. When you tell Friday you’re a photographer working on a memoir, that goes into your profile memory. It’s not a growing list; it’s a living document that gets updated only as things change. If you switch from photography to screenwriting, your profile is updated to reflect that.

Tier 2: Episodic memories. These are observations from coaching sessions - breakthroughs, feedback given by Friday, exercises attempted, emotional patterns noticed. Unlike the profile, these accumulate. They’re the coaching journal. “User had a breakthrough connecting childhood memories to current project” or “User tends to deflect when discussing their father.” “Framing feedback through visual metaphors works well for this user.” These memories grow over time and are never overwritten, only consolidated when they start to overlap.

Tier 3: Communication history. Friday can send follow-up emails between sessions - gentle nudges, session summaries, prompts to keep momentum going. This tier tracks what’s been sent as emails, so Friday doesn’t repeat itself or lose track of what it’s already communicated. This tier is mainly to keep e-mail communication fresh and relevant - it gets boring pretty quickly if you get the same motivational email every week.

Spoiler: we are currently working on a fourth tier for what are called prospective memories. Current AI memory systems are always retrospective - that is, they remember things that have happened or are happening. We also want Friday to remember things that are going to happen or might happen - your intentions, for example - and pick these up at the appropriate moment in the future. Just like a real coach would. If you express an intention “I want to get better at X, but lets focus on Y for now” we want Friday to remember that it should return to X at some point in the future. If you say “I’m working on a TED talk I’m giving in June” then Friday would know in May to ask you how it is progressing.

Who decides what to remember

This is the interesting part: Friday decides, not you the user, and not some kind of algorithmic rule engine. The AI behind Friday uses five memory tools we’ve given it.

First we use the system prompt (think of the system prompt as a long chat message at the beginning of every conversation that is never shown to you as a user but is shown to Friday - it is the rule book for how Friday should behave like Friday) to instruct Friday to save memories invisibly. The user should never what Friday is storing as memories during the chat. They should experience a coach who naturally remembers, the way a human coach would - Friday “just knows”. When you mention something important, Friday silently notes it. When it needs context from a past session, it silently searches. This way the conversational flow is not disturbed with technical AI details.

The key instruction is simple: save when you learn something worth remembering. Skip the throwaway details. If the user mentions they had coffee this morning, that’s not a memory. If they mention they’ve been stuck on chapter three for two weeks, that is.

The de-duplication problem

This is where we ran into our first real challenge.

Early on, we noticed Friday saving near-duplicate memories. A user would talk about a successful book reading, and Friday would save three separate memories: “The reading went well,” “User felt confident at the reading,” and “The Rönnells reading was a success.” (real example here) Three sticky notes about the same thing, each memory slightly longer than the last as more details are gleaned in conversation with the user.

The problem is mechanical. When the AI generates a response, it plans all its tool calls at once - before seeing the results of any of them. So it can’t check whether the first memory it just retrieved overlaps with the second one it’s about to save, because both are dispatched simultaneously.

Our solution has two parts. First, a de-duplication check: when Friday wants to save a memory, we first convert it into a numerical representation (a vector embedding) using OpenAI’s text-embedding-3-large model. Then we compare the suggested new memory against all existing memories (using cosine similarity, a mathematical measure of how similar two pieces of text are in meaning, not just wording). If the similarity score is above a certain threshold level (meaning the new memory is very close in meaning to something already stored) we flag the overlap and tell Friday about it. Friday then decides whether to merge the two memories into a single new memory, replace the old memory with a new one (“I am about to do a reading at Rönnels” becomes “I have done my a reading at Rönnels”) , or skip saving this memory.

The second part of de-duplication was keeping memories compact and focussed. We solved this with careful prompting. The system prompt now explicitly tells Friday: distill your observations into one memory per topic. One notebook entry, not three sticky notes. This simple instruction dramatically reduced duplicate memories.

How memory feeds into coaching

So that is how we save memories. But how does Friday use them in the future? At the start of every conversation, Friday’s system prompt is dynamically assembled. In other words - the system prompt is unique for each user, for each conversation you have with Friday. It includes: all of your profile (Tier 1) memories, the most recent episodic memories (Tier 2), and a crucial piece of metadata: the relationship stage.

The relationship stage is derived from memory count. Zero to two memories indicates to Friday that you are a new user. Three to ten memories means your relationship is developing. Eleven or more means it’s established. This single categorical label (new, developing, or established) changes how Friday coaches.

A new user gets listened to, not coached. Friday’s priority is understanding who they are, what they’re working on, what they hope to get out of this. The bar for saving memories for new users is deliberately low - Friday needs to capture the basics so that your next session has relevant context.

A developing relationship gets more tailored suggestions. Friday starts referencing past conversations naturally: “Last time you mentioned struggling with the chapter about your dad, how’s that going?”

An established relationship gets pushed. Friday can be more direct with feedback, notice patterns across sessions, call out avoidance. “You tend to get stuck at the same point - right when the emotional stakes get highest. Want to talk about why?”

These thresholds are also loose, on purpose. We don’t want the AI rigidly switching modes at exactly memory number three. Friday always knows about all three stages and their descriptions give it a gradient, which the LLM naturally interpolates during sessions so that you experience a natural progression the more Friday learns about you.

Memory transparency

Users can see and manage everything Friday remembers about them at their account page where they can see every memory Friday sorted by which tier each memory belongs to, the content of the memory, and when it was created. They can edit any memory (which regenerates its vector embedding) or delete entries outright.

This transparency felt important. The memory system works best when it’s invisible during a coaching session - you don’t want to see the machinery and break the illusion. But you absolutely should be able to inspect it whenever you want to, correct things that are wrong, and delete things you’d rather Friday forget.

The hard calls

There are genuine trade-offs in this design that we haven’t fully resolved.

Privacy vs. usefulness. The more Friday remembers, the better it coaches. But more memory also means more data stored about you. Naturally we store everything encrypted and take every technical precaution we can to ensure privacy. But we’ve also chosen to give users full visibility and control - there should be nothing in the system that Friday knows about you that you can’t see somewhere in the website.

Remembering too much vs. too little. A coach who remembers everything can feel like big brother. A coach who forgets too much feels shallow. Finding the right threshold for what’s worth saving. This an ongoing calibration for us - we are constantly adjusting Friday’s instructions here based on our own testing and user feedback.

Misclassification. Friday sometimes saves something as a profile fact when it should be an episodic memory, or vice versa. A stated intention (”I want to write about my mother”) might get saved as a permanent profile entry, when it’s really a moment-in-time observation. Again, we constantly tweak the prompt guidance, but it’s not perfect yet.

The thing that surprised us

The biggest surprise was how a clearly defined relationship stage matters. The difference between a user’s first session and their third session is dramatic. Not because we’ve done anything clever in the engineering, but because three sessions worth of memories gives Friday enough context to stop being generic.

One user told us Friday “knows I work with images, so all feedback is image-adapted.” We didn’t code that. Friday figured it out from the memories it had accumulated and started framing its coaching visually.

That’s our sign that the memory system is working. Not when you can see it, but when you forget it’s there.

This is the second in a series about what an AI coach needs to be useful. The first post covered safety guardrails in Friday:

Friday is the AI creativity coach at the heart of Revontale, built on the belief that coaching requires continuity, not just conversation. If you’re interested in how we’re building AI that remembers, subscribe for more in this series.