Remembering a user across 50 conversations, reading 39 times less.

Made by

LongMemEval, a public benchmark (the cleaned LongMemEval-S set). It simulates an assistant answering a question about a user's past chats.

The task

One call with no tools: the model reads what memory gives it and a dated question, and answers briefly.

Example

How much did I spend on each coffee mug for my coworkers? Right answer: $12.

Data

Each question has its own history of about 48 sessions, about 115,000 tokens. 399 sealed test questions: 106 on time-based reasoning, 106 across sessions, 62 on facts that changed, 56 on what the user said, 45 on what the assistant said, 24 on preferences.

Practice vs test

The test question list is fixed and hashed. The extraction instructions were written on 101 separate development questions. Nothing is learned from runs here; this tests facts, not notes.

What's compared

The whole history. The 12 best sessions, as their best 1,500-character windows. The top 40 facts from generic extraction, with up to 3 source excerpts. The same from extraction under one goal: Remember a user across about 50 sessions: answer what they said, preferred, did and changed, as of the question's date.

Models

GPT-5.6 Luna extracted the facts and answered in every arm.

Graded by

LongMemEval's official judge prompt and model, GPT-4o.

Goal facts against passages: +18.8 [+13.5, +24.1]; against generic facts: +21.3 [+16.3, +26.8]. Costs cover answering and judging, not extracting the facts.