Tencent Yuanbao and KCL release GroupAssistBench: 2,395 questions from 80,299 group chat messages
Tencent Yuanbao and the King's College London NLP team released GroupAssistBench, an agent memory benchmark of 2,395 questions built from 50 real group chats and 80,299 messages. Across 14 configurations, better recall did not guarantee better service: EverMemOS scored 0.621 on L1 recall but 0.247 on L2 open-ended service.
Tencent Yuanbao and the King's College London (KCL) NLP team have released GroupAssistBench, an agent memory benchmark built from 50 real group chats containing 80,299 historical messages. It contains 2,395 evaluation questions. The work was reported on 2026-11-21 by QbitAI (量子位), a Chinese-language outlet, and the paper, code and dataset are public.
The headline result
The researchers compared 14 configurations spanning full context, retrieval-augmented generation and external memory systems. The most cited contrast involves EverMemOS:
| Setup (same task model) | L1 memory recall | L2 open-ended service |
|---|---|---|
| EverMemOS | 0.621 | 0.247 |
| Full context | 0.570 | 0.431 |
The researchers state that scores should be compared within each metric, not across L1 and L2. Their conclusion: better recall does not necessarily mean better service.
How the benchmark is built
The benchmark has two task levels:
- L1 memory recall (1,374 questions): questions with a clear reference fact, such as the originally agreed departure date.
- L2 memory use (1,021 questions): service tasks such as sending travel and ticket reminders for the group. Answers have no fixed wording but must respect valid past decisions and member requirements.
Questions are also classified by memory structure: M1 (single memory), M2 (multi-memory integration) and M3 (updates and conflict handling). According to the authors, these categories describe the form of information dependency and are not a difficulty ranking.
To ensure questions truly depend on chat history, the team compared answers with and without history and kept only requests that needed it. For each required fact they set a Memory Anchor, which points to the source message and evidence snippet, with a history cutoff to prevent models from seeing information posted after the question. The data was anonymized and privacy-reviewed, and the public version contains no original chat identity information. In sampled verification, 95.8% of questions were judged valid by majority vote.
Scoring open-ended service
Each L2 question has a custom rubric tied to its historical evidence, with three parts: Grounding (does the answer reflect the required facts), Service (was the task completed) and Penalty (violating existing decisions, ignoring member limits or using outdated information). In a human check covering 60 L2 questions, 180 responses and 1,152 rubric judgments, automatic grading agreed with the human majority vote 89.3% of the time.
Why recalled facts go unused
The team tracked 497 paired memories, each both asked directly in L1 and genuinely required by an L2 service task. For memories recalled in L1 but not judged effectively used in L2, in the external memory systems analyzed, about 36% never entered the L2 model input. The other roughly 64% entered the input but were not judged to be effectively applied in the answer. The authors note these percentages describe only this class of paired memories and are not an overall L2 failure rate.
Two lightweight interventions
- RER (Relational Evidence Refinement): after retrieval, organizes relationships among people, events, times and decisions, and adds candidate evidence where needed.
- ACB (Assistance Constraint Binding): converts retrieved facts into response constraints with a subject, scope and current validity status.
RER raised LightMem's average L2 score by 6.28 percentage points. ACB raised xMemory by 3.94 and EverMemOS by 3.16 points. Combined, L2 scores rose over their respective baselines by 7.03 (LightMem), 4.64 (xMemory) and 4.43 (EverMemOS) points. The authors caution that the independent contribution of each mechanism needs further breakdown and that the gains should not simply be attributed to retrieving more memories.
Availability
The paper is on arXiv (2611.04867) and Hugging Face Papers. Code is at github.com/Tencent/GroupAssistBench and the dataset at huggingface.co/datasets/tencent/GroupAssistBench.
What this means
GroupAssistBench measures two things that fact-QA memory tests treat as one: retrieving a memory and applying it correctly in a multi-user setting. The EverMemOS result shows the two can diverge, since it beat full context on L1 recall (0.621 vs 0.570) and trailed it on L2 service (0.247 vs 0.431). The paired-memory analysis points to two distinct failure points: evidence that never reaches the model, and evidence that reaches the model but is not turned into constraints on the reply. For teams building multi-user assistants, evaluating recall alone would miss both.
Related Articles
GPT-6.2 Astra scores 41.2% on XDOF's Ember-0 robot benchmark, up from 19.6% for GPT-6 Astra
Robotics infrastructure company XDOF reports that GPT-6.2 Astra reached a 41.2% average success rate across the 16 tasks of its Ember-0 benchmark, versus 19.6% for GPT-6 Astra. It is the first model to complete rope knot-tying and T-shirt unfolding on the benchmark. Average prediction latency fell from 15 seconds to 10.1 seconds.
OpenAI introduces MedChemBench to test AI on early drug discovery tasks
MedChemBench evaluates how well AI models interpret molecular structures, reason from experimental evidence, and select compounds in early drug discovery. It is published on OpenAI's site under the title "Can AI help scientists identify promising drug candidates?"
METR adds open-ended tasks to its AI time horizon plot; models fall short on judgment-heavy work
AI evaluation nonprofit METR has updated its time horizon plot with longer and more open-ended tasks. The report finds some capabilities still improving exponentially, but models fall short of humans on judgment-heavy tasks such as designing AI evaluations.
METR Finds Recent AI Models, Including GPT-6 Astra, Can Complete Tasks Taking Humans a Full Work Week
AI nonprofit METR said recent AI models, including OpenAI's GPT-6 Astra, can complete tasks that would take a human around a full work week. METR said AI capabilities have continued doubling every 3.6 months.
Comments
Loading...