Milla Jovovich Solved Memory for AI Agents. Wait, What?

How a viral GitHub project inflated its benchmark scores through seven compounding tricks — and what it tells us about how we consume AI research in 2026.

Today, a link started circulating in my feed. A GitHub project called MemPalace — by a user named, yes, milla-jovovich — claimed to have essentially solved the long-term memory problem for AI agents. The headline number: 96.6% on LongMemEval with zero API calls, beating systems from well-funded research labs. With an optional LLM rerank pass, it hits 100%.

The project has a beautiful README. An elegant architecture inspired by the ancient method of loci — wings, rooms, halls, closets, drawers. A comparison table showing it above Supermemory, Mastra, and Hindsight. Open-source. Runs locally. No GPU required.

I almost shared it with a few friends. Then I read the source code.

What I found isn’t fraud. Every number is technically reproducible. But the gap between what the numbers say and what they mean is a masterclass in benchmark inflation. Let me walk you through it.

What is LongMemEval?

LongMemEval is a benchmark from ICLR 2025 that tests whether AI systems can remember things across long conversation histories. It has 500 carefully designed questions across five categories: factual recall, multi-session reasoning, knowledge updates, temporal reasoning, and abstention (knowing when you don’t know).

The benchmark comes in two sizes:

• LongMemEval_S: ~40 conversation sessions (the small, easy version)

• LongMemEval_M: ~500 sessions (the real deal)

It measures two things: can your system retrieve the right conversation, and can it answer the question correctly (judged by GPT-4o)?

Keep these details in mind. Every one of them matters.

Finding 1: The Wrong Metric

This is the biggest one, and it’s subtle enough that most people would miss it.

LongMemEval’s official evaluation script (print_retrieval_metrics.py in the original repo) reports this:

sess_metric_names = ['recall_all@5', 'ndcg_any@5', 'recall_all@10', 'ndcg_any@10']

The key metric is recall_all@5: do ALL the sessions needed to answer a question appear in your top 5 results?

MemPalace reports recall_any@5: does at least ONE of them show up?

ra = sum(metrics_session[f"recall_any@{k}"]) / len(metrics_session[f"recall_any@{k}"])

print(f" Recall@{k:2}: {ra:.3f}")

For single-fact questions (where the answer lives in one session), these metrics are identical. But LongMemEval has multi-hop questions. If you need evidence from sessions A, B, and C, and your system retrieves only A — that’s 100% on recall_any but 0% on recall_all.

The README just says “R@5 = 96.6%.” It never mentions it’s recall_any. The reader naturally assumes it’s the standard metric. It is not.

And here’s the kicker: MemPalace’s code actually computes both metrics. It calculates recall_all — and then doesn’t print it. The value is computed, stored in a list… and silently dropped from the output. You can see the variable rl (recall_all) being appended to metrics_session[f”recall_all@{k}”], but the print loop only reads from recall_any.

Finding 2: The Easy Mode Dataset

MemPalace runs exclusively on LongMemEval_S — the variant with ~40 sessions per instance. The data file is literally called longmemeval_s_cleaned.json.

Here’s why this matters: when your haystack has only 40 sessions and you’re returning top-5 results, you’re looking at 12.5% of the entire corpus. Any decent embedding model will get most single-hop questions right just from semantic similarity. It’s less “finding a needle in a haystack” and more “finding a needle in a small toolbox.”

LongMemEval_M, with ~500 sessions per instance, is where retrieval becomes genuinely hard. MemPalace never runs on it. The README never mentions that this larger, harder variant exists. The reader sees “LongMemEval” and assumes it’s the full benchmark.

Finding 3: Memorizing the Test Set, One Question at a Time

This is where it gets technically fascinating.

MemPalace has a progression of retrieval modes: raw → hybrid v1 → v2 → v3 → v4. Each version adds features. Let’s trace how those features were developed.

hybrid_v2 docstring:

“three targeted fixes for the remaining 11 misses”

They examined 11 specific failing questions and engineered three new features to fix them: temporal date boosting, a two-pass retrieval for “you told me” style questions, and preference broadening.

hybrid_v3 docstring:

“two targeted improvements for remaining misses”

They added 16 regex patterns for detecting user preferences and expanded the LLM rerank pool from 10 to 20, specifically because “the two remaining assistant failures have their correct session at rank 11–12.”

And then hybrid_v4. This is the one. The docstring literally names the remaining three failing questions by their unique IDs:

Miss 1 — ‘high school reunion’ (d6233ab6)

Miss 2 — ‘Rachel/ukulele’ (4dfccbf8)

Miss 3 — ‘sexual compulsions’ (ceb54acb)

For each miss, a bespoke fix:

• For the high school reunion question, they added nostalgia-themed regex patterns: r”i (?:still )?remember (?:the |my )?…” and r”(?:happy|fond|good|positive) (?:high school|college|childhood) (?:experience|memory|memories)…”

• For the Rachel/ukulele question, they added person-name extraction from queries, boosting sessions containing “Rachel” by 40%

• For the sexual compulsions question, they added quoted-phrase detection: if the query contains ’sexual compulsions’ in quotes, boost documents containing that exact phrase by 60%

Each of these fixes is reasonable in isolation. Person name matching is a good idea. Quoted phrase matching is a good idea. But they weren’t discovered through principled analysis or ablation studies. They were reverse-engineered from three specific test questions.

This is the ML equivalent of a student who got access to the final exam and studies only those questions.

Finding 4: The “Held-Out” Set That Isn’t

MemPalace claims a “clean” held-out score: 98.4% R@5 on 450 unseen questions. They created a 50/450 dev/held-out split to address contamination concerns.

The problem: the split was created after hybrid versions 1 through 3 were already developed by examining failures across all 500 questions. The 16 preference extraction patterns, the temporal boosting, the assistant two-pass logic — all of it was tuned with visibility into the full dataset.

Only the three v4 fixes claim to respect the split boundary. But the foundation they’re built on — v1 through v3 — was trained on the “held-out” set.

That’s like a student who practiced with the answer key for six months, then says “but the last three questions I solved without looking!” The held-out score inherits all the contamination from earlier development.

Finding 5: Half the Benchmark is Missing

LongMemEval measures two things: retrieval (can you find the right session?) and QA accuracy (can you generate the correct answer?). The QA evaluation uses GPT-4o as a judge.

MemPalace only does retrieval. It never generates an answer. It never runs the QA evaluation.

This matters because retrieval and QA accuracy can diverge substantially. You might find the right session but fail to extract the right fact. You might retrieve a session with both old and updated information and pick the wrong one. You might find a temporal reference but fail the date arithmetic.

By skipping QA evaluation, MemPalace avoids the part of the benchmark where the hard questions actually bite.

Finding 6: The Comparison Table

The README has a comparison table:

This looks like MemPalace is competitive with or better than all these systems. But the table doesn’t specify what metric each system reports. MemPalace’s number is recall_any@5 (retrieval only, easy metric, small dataset). The other systems may be reporting recall_all or QA accuracy or something else entirely. Placing these numbers in the same column without clarifying the metric creates a false equivalence.

Finding 7: The Benchmark That Tells the Real Story

LoCoMo is another memory benchmark — multi-hop reasoning across 10 extended conversations, 1,986 question-answer pairs. It’s harder than LongMemEval_S because it genuinely requires cross-conversation reasoning.

MemPalace’s raw baseline on LoCoMo: 60.3% recall.

That’s the same ChromaDB + all-MiniLM architecture that gets 96.6% on LongMemEval_S. On a genuinely hard retrieval task, it loses 36 percentage points.

The README doesn’t mention 60.3%. Instead, it highlights an 88.9% number (achieved after the same iterative patching process) and a “100%” variant that the authors themselves acknowledge is trivial because “top-50 exceeds session count” — meaning they returned the entire corpus.

What MemPalace Actually Is

Strip away the benchmark inflation, and MemPalace is a local ChromaDB wrapper with heuristic reranking. That’s not nothing — local, zero-API memory with decent retrieval is useful. The palace metaphor (halls for topic classification, closets for user turns, drawers for assistant turns) maps to straightforward keyword-based session routing.

But the hall classification is a bag-of-words heuristic:

TOPIC_KEYWORDS = {
    "technical": ["code", "python", "function", "bug", "error", "api", …],
    "personal": ["family", "friend", "birthday", "vacation", "hobby", …],
    "knowledge": ["learn", "study", "degree", "school", "university", …],
}

This is not a breakthrough in AI memory. It’s a reasonable engineering project dressed in benchmark clothing that doesn’t fit.

The Bigger Picture: Why We Keep Falling for This

Every few weeks, a new project hits the AI feed with a benchmark number that sounds too good to be true. And every few weeks, someone reads the code and finds the same patterns: easy dataset variant, wrong metric, iterative test-set fitting, apples-to-oranges comparison tables.

Why does it keep working?

We’ve outsourced our judgment to numbers. “96.6%” looks objective. It looks scientific. It triggers the same trust response as a peer-reviewed result, even when it’s self-reported, on a self-selected metric, on a self-selected dataset variant.

Our attention span is one README long. The headline number is in the first paragraph. The methodology caveats — if they exist at all — are buried in a /benchmarks subdirectory that 99% of people will never open.

We believe in vibes. The project looks good. The architecture has a cool metaphor. The comparison table has the right shape. The username is memorable. In 2026, when half the code is vibe-coded, we’ve developed a dangerous intuition that if something looks right, it probably is right. The flip side is that we’ve lost the instinct to read the source.

We want heroes. A solo developer beating funded research labs is a great story. A ChromaDB wrapper with keyword heuristics is not. We select for the story we want to tell.

The fix isn’t complicated: read the code. Check which metric is reported. Check which dataset variant was used. Check whether the comparison table compares the same things. It takes an evening, not a PhD.

The 96.6% is real. You can run the script and get it. But real and meaningful are not the same thing.