A model can answer as if it knows you, but today’s AI memory is mostly an illusion. It doesn’t know you. It is reading the chat that is open in front of it. Close that chat and the facts leave with it. Start a new one next week and the model has no way to tell a fact that is still true from one that stopped being true. It may repeat the old one with the same confidence, because nothing inside the model marked it as old.
That is the problem this piece is about. A long prompt is not a memory. A cache is not a memory. And no shipping model was found, as of 3 October 2026, that keeps the right facts across weeks and drops the ones that went stale.
Picture a reply as many cheap steps adding up to one thought, if that helps. Treat it as a metaphor. It is not a claim that models work like brains.
Context window, prompt cache and side store: three kinds of AI memory
Three different things get called memory, and only one of them is the hard problem.
A long context window is a bigger sheet of paper for a single call. Gemini 3.8 Flash, latest update September 2026, accepts up to 1,048,576 input tokens and can write up to 65,536. Mixtral’s window, from Mistral’s note of 11 December 2023, is 32,000 tokens. DeepSeek-V3’s model card says 128,000. More room for one conversation. Nothing about next month.
A prompt cache is a discount, not a diary. If you send the same opening text again, exactly, the repeat can be cheap. Change the text, or add tokens after the break, and you pay the normal price. Anthropic’s caching page, read on 3 October 2026, prices a 5-minute cache write at 1.25 times normal input, a 1-hour write at 2 times, and a read at 0.1 times. The read is 0.025 times on Claude Fable 5.1 and Claude Mythos 5.1, and 0.05 times on Claude Opus 5.5 [6].
OpenAI’s caching page, read the same day, says cached input is discounted by up to 95 percent. For GPT-5.6 and later, a write is 1.25 times the normal input price and a read is 0.1 times, or 0.05 times on GPT-6.1 Sol. Output tokens are not part of the deal [3]. Some infrastructure firms now try to stretch caching further: Tensormesh reuses the KV cache across servers, but that is still a saving on repeated text, not a memory.
A store beside the model is the thing that actually keeps notes. It is still not the model remembering.
Graphiti and Mem0: the memory tools you can already build with
Several open-source tools sit next to the model and keep notes for it. Two show the limits clearly.
Graphiti keeps facts on a timeline: each fact has a validity window, and when information changes the old fact is invalidated rather than deleted [13]. You bring the database. That is the closest shipping version of “this used to be true, and now it isn’t.”
Mem0’s memory algorithm of April 2026 uses single-pass, add-only extraction: memories accumulate and nothing is overwritten [15]. Its hosted test scores belong to the hosted product. They are not the open-source build, and they should not be used to judge the code you run yourself.
Other agent tools write notes, hooks, or files an agent can read next time. Useful. Still a notebook. A chat log is still a chat log. Tencent’s Hy-Memory 2.0 agent memory SDK, released after this piece, is another store of this kind.
Three gaps between a notebook and real model memory
Three gaps are still open. None of them showed up as a product you can call.
The model would have to update, and forget. Google’s Titans paper, arXiv 2501.00663, is the research version of that idea. It updates a neural memory while it runs, and it has a forget gate. The experiments are research-scale models, not an API, and the paper should not be read as a deployed memory that replaces a chat log.
The practical version is narrower than the paper. Write a fact, stamp when it was true, and stop using it when a newer fact replaces it. Graphiti already does the expiring part in a graph you operate. Doing it inside the model, with no side store, was not found.
The hidden steps would have to stop inflating the bill. On 12 September 2024, OpenAI said o1 gets better when it spends more time thinking. The post does not say that time is cheap. The current reasoning guide says those thinking tokens are hidden from you, still take up context, and are billed as output tokens. A lower effort setting uses fewer of them. A higher one thinks more. Pro mode bills the extra work at that model’s normal rates, and it raises both token use and cost. [1]
The bill is a price times tokens. A lower price does not cap it. Spend more tokens and you can pay more. No primary page checked here measured a named customer’s bill rising after a specific price cut. No page said the provider absorbs the hidden tokens.
There are cheaper paths for some calls, and only for those calls. OpenAI’s Batch API is 50 percent off the normal price if the job can wait up to 24 hours. Microsoft’s Foundry model router, version 2025-11-18, can send an easier call to a cheaper model. Balanced is the default. The page’s 1 to 2 percent and 5 to 6 percent figures are quality gaps against the best model for that prompt, not a cost discount. It does not say the router itself is free.
A long internal chain would have to cost about as much as one reply. No 2025 or 2026 primary source opened for this piece says that. NVIDIA’s NIM documentation, read on 3 October 2026, describes something smaller. A draft model guesses several tokens and the larger model checks them in one pass. When the guess is right, several tokens cost about one check. When it is wrong, decoding continues as usual. That is not a chain of memory updates [10].
OpenAI’s Predicted Outputs is a real feature on the GPT-4o and GPT-4.1 series named on its page. The page does not call it speculative decoding. Tokens it rejects are still billed, so the feature can cost more, not less.
Mixture-of-experts is a different kind of saving, and it is about cost per token, not about remembering. Mistral, on 11 December 2023, said Mixtral picks 2 of 8 experts for every token. Mistral’s own figures are 46.7 billion parameters in total and 12.9 billion used, at the same speed and cost as a 12.9 billion parameter model. DeepSeek-V3’s card says 671 billion parameters in total and 37 billion active per token. [8][9]
What an honest AI memory would look like
If someone built the missing piece, it would not look like a model that “just knows.” It would look like this.
Facts would carry a time. A new fact would replace the old one, so nothing stayed true forever by accident. The store would sit apart from the chat, so a long prompt would not be mistaken for a memory. Hosted scores would be labelled as hosted. Text you send again could use the cache. A fresh fact would be written at the normal price, because a write is not a read. Extra thinking would stay on the bill until a provider prices it at zero. None of them has said they do.
Update, 8 October 2026: Devin’s memory and dreaming
Two days after this piece was published, Cognition launched a memory feature for its Devin coding agent that meets part of the “honest version” described above [14].
Devin’s memories are short notes kept in a Git repository, separate from the chat. At the start of a session Devin reads only a small index file and looks up other notes when they seem relevant [14]. That matches one test above: the store sits apart from the chat, so a long prompt is not mistaken for a memory. Users can browse and correct what has been stored [14].
Each night a background “dreaming” session merges overlapping notes, strips out passing details, looks for lessons that were missed, and removes “stale memories records that weren’t used by any sessions” [14]. When two sessions edit the same note in conflicting ways, the conflict is flagged rather than overwritten [14].
What it does not do, on Cognition’s description, is mark each fact with the time it stopped being true. “Stale” here means unused, not out of date. A note can be unused and correct, or used and wrong. So this is an agent with a well-kept notebook, not the model-level memory this piece found missing. It is, though, a shipping agent that prunes its own notes on purpose. Cognition has not published results showing whether the notes improve Devin’s work [14].
Why AI memory is still a notebook beside the model
You can already give an agent a notebook, a very long sheet of paper, and a discount when you repeat yourself. You cannot yet hand it a memory that updates and forgets. The research sketch exists in papers such as Titans [12]. The product does not. And until the hidden steps are priced differently, remembering more will keep showing up on the invoice as more tokens, not as one cheap thought.
That matters for anyone building on these systems. A notebook can be inspected, corrected and deleted, which is a real advantage for privacy and for fixing mistakes; a memory buried inside model weights would be much harder to audit. The practical gap is time. Graphiti shows a store can mark when a fact stopped being true [13], while Mem0’s add-only design and Devin’s pruning of unused notes show that many tools still treat “stale” as “old” or “unused” rather than “no longer correct” [14][15]. Shared stores such as Anchor’s Action Index for computer-use agents push the same idea outward, to workflows many agents can reuse. Until a provider ships timestamped facts that expire by default, users should assume an assistant may repeat something that was true last month with the same confidence as something true today.
Sources
Undated pages were read on 3 October 2026.
- OpenAI, reasoning models. https://developers.openai.com/api/docs/guides/reasoning
- OpenAI, “Learning to reason with LLMs,” 12 September 2024. https://openai.com/index/learning-to-reason-with-llms/
- OpenAI, prompt caching. https://developers.openai.com/api/docs/guides/prompt-caching
- OpenAI, Batch API. https://developers.openai.com/api/docs/guides/batch
- OpenAI, Predicted Outputs. https://developers.openai.com/api/docs/guides/predicted-outputs
- Anthropic, prompt caching. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- Microsoft, Foundry model router, version 2025-11-18. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-router
- Mistral, “Mixtral of experts,” 11 December 2023. https://mistral.ai/news/mixtral-of-experts
- DeepSeek-V3 model card, citing arXiv:2412.19437. https://huggingface.co/deepseek-ai/DeepSeek-V3
- NVIDIA NIM, speculative decoding. https://docs.nvidia.com/nim/large-language-models/latest/advanced-use-cases/speculative-decoding.html
- Gemini 3.8 Flash. https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash
- Titans, arXiv 2501.00663. https://arxiv.org/abs/2501.00663
- Zep, Graphiti repository README (temporal context graph, fact validity windows), read 3 November 2026. https://github.com/getzep/graphiti
- Cognition, “Memory and dreaming: how Devin learns from working with you,” Devin blog, 5 October 2026. https://devin.ai/blog/memory-and-dreaming
- Mem0, repository README (“New Memory Algorithm (April 2026)”, single-pass add-only extraction), read 3 November 2026. https://github.com/mem0ai/mem0
Get TSN’s biggest tech stories, explained.
News and insights across AI, tech, crypto and cybersecurity. Every source linked. Unsubscribe anytime.
Get the newsletterFollow @tsnmediahub on X for breaking AI news as it happens.
Follow on X


