IBM Research Reports Learned Memory Tops τ²-Bench Service Benchmark

PUBLISHER ARTICLE PREVIEW

From the original article

In an October 20, 2026 post on Hugging Face, IBM Research reported that guidelines learned automatically from an agent’s own training runs and appended to its system prompt lifted a stock GPT-4.1 customer-service agent to 67.5% Pass^4 on τ²-bench’s retail domain, above every retail entry in the team’s October 12, 2026 leaderboard snapshot.

The result comes from ALTK-Evolve, the lab’s method for building what the authors describe as a learned playbook. The method analyzes an agent’s attempts on training tasks and uses the same model to distill successes and failures into short behavioral rules, covering situations such as authenticating a customer, confirming a transaction, or handling a return.

According to the post, the same model powers both guideline extraction and the agent itself, and the process requires no weight updates or human annotation. The team previously demonstrated the approach on coding agents; the new study applies it to customer service. The post lists eleven IBM Research authors, among them Ashwath Vaithinathan Aravindan, Vatche Isahagian, and Vinod Muthusamy.

τ²-bench, maintained by Sierra Research, is a simulation framework for evaluating customer-service agents across domains. According to the τ²-bench repository, each domain specifies a policy the agent must follow, a set of tools, and a set of evaluation tasks, and the benchmark’s core paper describes a dual-control environment. IBM’s study uses the retail and airline domains.

Each task pairs the agent with an LLM-simulated customer, domain tools, and a written policy document; a task counts as solved only when the final database state matches the expected outcome, which the post describes as a strict binary reward.

The evaluation ran τ²-bench’s own implementation of a basic ReAct agent with a two-phase THINK→ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model, with four trials per task. The only modification was a plain markdown list of learned guidelines appended to the system prompt after the domain policy.