Skip to content

Results

Published evaluations of Evolve cover task completion, repeat-run consistency, and memory efficiency on AppWorld and τ²-bench. Performance depends on the agent, model, memory configuration, and evaluation protocol. Each section identifies the setup and links to its reported evidence.

The studies use four metrics:

  • Task Goal Completion (TGC): the percentage of individual AppWorld tasks completed successfully, as checked by the benchmark's programmatic grader.
  • Scenario Goal Completion (SGC): the percentage of AppWorld scenarios for which every task variant succeeds, usually three variants per scenario.
  • Pass^k: success across all k attempts at the same task. The repeated-run studies below use five attempts on AppWorld and four on τ²-bench.
  • Mean accuracy (Mean@k): success averaged across those k attempts.

SGC measures success across related task variants; Pass^k measures success across repeated attempts. Completion and accuracy values are percentages, and absolute gains are reported in percentage points (pp).

Generalization to unseen tasks

A GPT-4.1 ReAct agent was evaluated on AppWorld's held-out test-normal partition. Guidelines were generated from trajectories on the train and dev partitions. The configuration below uses subtask-level guidelines with LLM-guided selection, retrieving the top five guidelines before reasoning, with a maximum of 30 reasoning-action steps per task. It achieves the highest aggregate SGC among the configurations reported in the technical paper.

Difficulty Baseline TGC + Evolve TGC Baseline SGC + Evolve SGC
Easy 89.5% 91.2% 79.0% 89.5%
Medium 66.7% 70.8% 56.2% 56.2%
Hard 54.0% 58.7% 19.1% 47.6%
Aggregate 69.6% 73.2% 50.0% 64.3%

Aggregate task completion improves by +3.6 pp, and scenario completion by +14.3 pp. Hard-task scenario completion rises from 19.1% to 47.6% (+28.5 pp). Medium-task scenario completion is unchanged, even though individual task completion improves. The effect therefore depends on both difficulty and metric.

These results test transfer from train/dev experience to unseen tasks. The technical paper reports additional extraction and retrieval configurations; the Memory as Middleware paper reproduces this table in its preliminary evaluation.

Repeat-run consistency

A separate study evaluates a GPT-4.1 ReAct agent on AppWorld test_normal (168 tasks), with five attempts per task. Consistency guidelines are generated from one baseline trajectory per task. Same-task evaluation applies those guidelines to fresh attempts at that task; sibling-task evaluation applies guidelines from other variants of the same scenario.

Guidelines from Pass^5 Gain vs. baseline Mean@5 Gain vs. baseline
No guidelines (baseline) 53.0% — 77.4% —
Same task 69.0% +16.0 pp 81.0% +3.6 pp
Sibling task 66.0% +13.0 pp 79.5% +2.1 pp

The 53% → 66% result measures transfer between related scenario variants. The 53% → 69% result measures improved repeat use of the same task. Both improve aggregate mean accuracy, alongside the stricter all-five-attempts metric.

Guideline generation has an additional offline cost: the paper's Consistency Analyzer uses 30 resamples per inference step, requiring roughly 30 times the tokens for each trajectory analyzed. That analysis cost is separate from the runtime inference-token figures in the next section.

Sources: Closing the Consistency Gap, especially the evaluation and Appendix Table 1; Your Agent Aced the Task. Will It Do It Again?.

Model coverage and efficiency

The reported eight-model sweep evaluates different ways to deliver learned guidelines. The examples below use AppWorld test_normal; the article describes guideline mining from the training split only. Each row shows the best of the tested delivery configurations for that model.

Curated retrieval supplies a fixed, high-confidence core plus task-relevant guidelines. The full guideline set supplies every mined guideline at each ReAct step.

Model Best tested delivery TGC: baseline → Evolve SGC: baseline → Evolve
gpt-oss-120b Curated retrieval 39.9% → 56.0% 21.4% → 37.5%
DeepSeek-V3.2 Full guideline set 79.8% → 89.3% 64.3% → 80.4%
Claude Opus 4.6 Full guideline set 90.5% → 94.6% 87.5% → 94.6%
GPT-5.5 Full guideline set 92.3% → 95.2% 82.1% → 89.3%
GLM-5 Full guideline set 87.5% → 87.5% 80.4% → 80.4%

The results show benefits for several models, including ones with high baseline completion, and no measured gain for GLM-5 in this setup. Memory delivery is a configuration choice to evaluate for the model and task distribution.

Runtime token usage

The article reports average inference tokens per task, accumulated across agent steps. These figures describe runtime usage; guideline generation is an additional operation. Token counts are rounded as reported (K = thousand tokens).

Model Delivery No-memory tokens/task Evolve tokens/task
gpt-oss-120b Curated retrieval 110K 116K
gpt-oss-120b Full guideline set 110K 166K
DeepSeek-V3.2 Full guideline set 148K 263K

With curated retrieval, gpt-oss-120b gains +16.1 pp task completion. Supplying all guidelines incurs substantially more tokens for both models in the table.

Source: How Much Memory Does Your Agent Actually Need?.

In-house comparison with ACE

The ACE comparison uses in-house, single-run evaluations on AppWorld test_normal (168 tasks), with the same base models and ReAct harness. Memory is mined from train/dev. The systems use different prompt templates and have different no-memory baselines; the table shows their reported post-memory performance and inference tokens.

For the repeated Evolve scores, the articles describe different guideline-mining splits: train only in the memory-dosage article and train/dev in the ACE comparison. The relationship between those runs remains unclear in the published descriptions.

Model ACE TGC / SGC Evolve TGC / SGC ACE tokens/task Evolve tokens/task
gpt-oss-120b 54.8% / 35.7% 56.0% / 37.5% 777K 116K
DeepSeek-V3.2 80.4% / 73.2% 89.3% / 80.4% 634K 263K

For gpt-oss-120b, Evolve uses about one-seventh of the inference tokens. The article treats task completion as approximately tied: an Evolve repeat run scored 54.8%, matching ACE. For DeepSeek-V3.2, Evolve reports higher completion with about 41% of ACE's inference tokens. These are token comparisons; monetary cost depends on model pricing and caching.

Source: Thinking of ACE? We Can Do It with Fewer Tokens, including its method notes and reference tables.

Customer-service tasks

The τ²-bench experiments use the benchmark's native ReAct scaffold (THINK → ACT), with learned guidelines appended after the domain policy in the system prompt. Guidelines are learned from each domain's train split using the agent model. Evaluation uses τ²-bench v1.0.1, GPT-5.5 as the customer simulator, and four attempts per task.

Held-out test results

The table below uses 40 retail test tasks and 20 airline test tasks. sup3 retains guidelines supported by at least three training tasks; sup2 uses a threshold of two. “All” retains every consolidated guideline, including support-one guidelines. Counts appear in parentheses.

Domain Agent model Guidelines Mean accuracy Pass^4
Retail (40 test tasks) GPT-4.1 None 78.1% 57.5%
Retail (40 test tasks) GPT-4.1 sup3 (31) 78.8% 60.0%
Retail (40 test tasks) GPT-4.1 All (292) 82.5% 67.5%
Airline (20 test tasks) GPT-4.1 None 57.5% 35.0%
Airline (20 test tasks) GPT-4.1 sup3 (14) 60.0% 45.0%
Airline (20 test tasks) GPT-4.1 All (101) 65.0% 40.0%
Airline (20 test tasks) GPT-5.5 None 72.5% 55.0%
Airline (20 test tasks) GPT-5.5 sup2 (103) 80.0% 70.0%
Airline (20 test tasks) GPT-5.5 All (146) 82.5% 70.0%

On retail, GPT-4.1 gains +4.4 pp mean accuracy and +10.0 pp Pass^4 with all guidelines. On airline, GPT-5.5 gains +10.0 pp mean accuracy and +15.0 pp Pass^4 with all guidelines. GPT-4.1's airline result illustrates the delivery tradeoff: all guidelines give the highest mean accuracy, while the smaller sup3 set gives the highest Pass^4.

The article also reports cost on the full base split: retail's combined agent and customer-simulator cost increases from approximately $0.084 to $0.139 per conversation (+65%) with all guidelines. This cost table covers a different task set from the held-out table above.

Source: ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model, particularly Tables 2, 3, and 5.

Submitted full-base results and leaderboard snapshot

The October 20 article compares full-base submissions with a leaderboard snapshot from October 12, 2026. The benchmark's base split includes both train and test tasks, so it includes tasks used to learn guidelines and differs from the held-out evaluation above.

Domain Agent model Base tasks × attempts Submitted Pass^4 Submission
Retail GPT-4.1 114 × 4 67.54% PR #542
Airline GPT-5.5 50 × 4 82.00% PR #551

As of October 27, 2026, both submission PRs are open. Both disclose modified_prompts: true, training-task overlap, and an expected Unverified status. The article's ranking comparisons refer to its dated snapshot; the held-out tables provide the evidence for transfer to unseen tasks.

April launch evaluation

The April 8 introduction reports the following AppWorld SGC table. The April 7 IBM announcement highlights the aggregate +8.9 pp and hard-task +14.2 pp gains. The introduction describes a ReAct agent receiving the top five guidelines generated from prior train/dev trajectories and evaluated on unseen test-normal tasks.

Difficulty Baseline SGC + Evolve SGC Gain
Easy 79.0% 84.2% +5.2 pp
Medium 56.2% 62.5% +6.3 pp
Hard 19.1% 33.3% +14.2 pp
Aggregate 50.0% 58.9% +8.9 pp

These are the historical figures behind the original homepage headline, including the approximately 74% relative improvement on hard tasks. The generalization section presents the best SGC configuration reported in the technical papers with its stated setup.