I didn't set out to build an AI memory architecture. I set out to stop reconstructing the same context for the 5th time in a week.
That's roughly how it started. Work was spread across AI chats, tickets, documents, code, notes, messages, and whatever half-formed research I'd left in a browser tab 3 weeks earlier. None of it was lost, exactly. It was just scattered enough that finding it again cost more than re-deriving it from scratch, so I usually just re-derived it. Which is its own kind of loss, repeated quietly, every week.
The fix people reach for is "give the agent memory." Fine, but memory of what, stored how, kept for how long? Those turned out to be the actual questions, and I've spent more time on them than I expected to.
3 kinds of true
It isn't chat history, and it isn't quite a knowledge base either. What I've ended up with is 3 different kinds of thing worth keeping, and they don't behave the same way.
Journal entries sit closest to episodic memory: what happened, what I tried, what I thought at the time. They don't need to be right. "I believed X" is useful even when X turned out to be wrong later, maybe especially then, because it tells me something about how I was reasoning, not just what I concluded.
Curated knowledge sits closer to semantic memory, the stuff expected to stay useful: how a system works, a decision that still holds, a convention worth not re-litigating. This is also the one that gets abstracted the most, which is where it gets dangerous.
Current state isn't fact so much as current belief. "This will probably take 3 weeks" isn't a verified claim, it's my present estimate, and it should be recorded as exactly that and no more.
Belief, not truth
That last distinction matters more than it sounds like it should. It's tempting to talk about a memory system as if it knows what's true. It doesn't. It knows what's been recorded, or inferred, as true, which is a quieter and much less confident thing.
"The deployment failed at 14:32" is an observation. "I think it failed because of a config issue" is an interpretation. "At the time, I thought the config was the cause" is a historical belief, and it stays useful even after the interpretation turns out wrong, because it tells me what I was working with at the time.
A good work memory needs to hold all 3 without flattening them into a sentence that only sounds confident. Which is harder than it sounds, because the flattened sentence always reads better than the hedged one.
The actual hard part
Here's the part that took longer to see clearly than I'd like to admit: the hard problem was never storage. It's deciding what survives the trip from "thing that happened" to "thing that's remembered."
Say a decision gets made in a conversation. There were alternatives considered, constraints that ruled them out, an edge case somebody flagged and half dismissed. An agent asked to remember that conversation might store: "Team decided to use X."
Which is true. It's also nearly useless, because the reasoning is gone. 6 weeks later, when the decision gets questioned, the memory can confirm what was decided and say nothing about why, which is exactly the moment why matters most.
That's a different failure to anything I ran into with manual notes. With manual notes, I decide what gets written and I control the wording, badly sometimes, but at least it's my judgement making the cut. When an agent manages the memory instead, it decides what survives, and it can be confidently, fluently wrong about what mattered. Semantically correct, practically useless, and there's no obvious signal at write time telling you which one you're getting.
Testing against raw memory
I've started calling this lossy memory, if only to myself: the agent gets the general idea right and still discards exactly the detail that turns out to matter 8 months later.
I've been running an experiment with Hugging Face's Funes alongside my own curated setup, mostly to test that exact worry. Funes leans episodic: instead of trusting a model to compress a session into a summary, it indexes and retrieves from the actual prior traces. Provenance stays intact, because nothing got compressed away in the first place. The trade is obvious once you say it out loud. Curated memory is cheap to read and expensive to trust. Raw memory is expensive to read, more context, more noise, and easier to verify, because you can see exactly where a claim came from. I don't think one wins outright. I think the honest answer is that they fail in different places, and running both side by side has told me more than picking one and hoping would have.
Accidental connections
Most of what memory gets praised for is retrieval: I know I wrote something about this, and the system finds it. Useful, but not the interesting part.
The more interesting behaviour turns up when the system connects 2 things I wasn't trying to connect. A small task reveals an architectural concern, nothing dramatic, filed and forgotten. Weeks later, a completely unrelated task runs into the same concern. Neither conversation looks important on its own. Put them in the same memory and the pattern becomes visible in a way it never would from either conversation alone.
Even the small version of this is worth having. One session records a name one way, another records it slightly differently, and persistent memory is what notices the 2 don't match. Not exciting, but it's the same underlying thing: information from different contexts, actually being compared, instead of sitting in 2 places that never talk to each other.
No correct schema
I'd like to end this with the shape of the ideal memory schema. I don't have it, and I've mostly given up expecting to find it in advance.
The reason isn't that schema design is hard in the usual sense. It's that the future value of a piece of information is unknowable at the moment you encounter it. You don't know which throwaway comment about a colleague will matter in 8 months. You don't know which small implementation detail is actually an architectural decision wearing a disguise. Trying to design the perfect structure upfront means guessing at relevance you can't yet see.
What's worked better is starting loose and letting repeated pain point at where structure is actually missing. The same person recorded under 2 different names, that's a signal. A decision I can't reconstruct the reasoning for, that's a signal. A pattern that shows up across 2 conversations that never should have known about each other, and doesn't, because nothing connected them, that's the strongest signal of all.
None of that is a finished answer. It's closer to a standing practice: notice what keeps failing, and let that tell you what deserves structure next. Which, now that I've written it down, sounds a lot less like building a memory system and a lot more like paying attention.