SkillShift-Bench
GitHub →
Menu
paper.cover Camera-ready

SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents

  • Bo-Yu Chen 1,2
  • Zhi Zhou 1,2 *
  • Yu-Feng Li 1,2 *
  1. School of Artificial Intelligence, Nanjing University
  2. National Key Laboratory for Novel Software Technology, Nanjing University

{chenby, zhouz, liyf}@lamda.nju.edu.cn

* Corresponding authors.

PDF forthcoming Cite ↓

paper.abstract Abstract

Abstract

Skill learning enables web agents to reuse prior experience for long-horizon interactive tasks. However, learned skills are typically evaluated in closed environments, overlooking the fact that real-world web deployment involves continuous shifts in invocation context, perceptual grounding, and execution feasibility. To this end, we introduce SkillShift-Bench, the first benchmark for evaluating skill learning with controlled open-environment shifts. SkillShift-Bench simulates Contextual Shift, Perceptual Shift, and Execution Shift across three self-hosted web systems, and evaluates agents with diagnostics covering task success, interaction cost, skill use, and skill failure. Comprehensive experiments reveal systematic robustness degradation under controlled shifts, characterized by lower success rates, higher interaction costs, and more frequent skill failures, with Execution Shift exerting the strongest pressure. We further formulate shifted skill execution as a nonstationary local executability problem and provide both a dynamic-regret analysis and a practical test-time adaptation method, demonstrating the feasibility of improving learned-skill execution without relearning the skill library. Code, tasks, environments, documentation, and the leaderboard are available at https://skillshift-bench.github.io/.

paper.contrib 3 items

Contributions

  1. Bench

    We introduce SkillShift-Bench, a controlled benchmark for learned skill executability under Contextual, Perceptual, and Execution Shifts.

  2. SWOP

    We formulate shifted skill execution as nonstationary local executability and analyze SWOP with a variation-dependent dynamic regret guarantee.

  3. MPCR

    We propose MPCR, a test-time, fixed-library executor-side reranker that improves transfer through feasibility-aware action selection.

paper.figure Fig. 1

Shift modes

Three controlled shift modes: Contextual, Perceptual, and Execution
Three controlled shift modes in SkillShift-Bench. Aligned variants preserve task intent, task-critical entities, required functionality, and success criteria, while Contextual, Perceptual, and Execution Shifts respectively target skill invocation, grounding evidence, and runtime feasibility.
paper.findings RQ1 · RQ3

Findings

Success decreases from 40.97% in the Base view to 35.23% in Perceptual and 18.69% in Execution, with Execution Shift exerting the strongest pressure. On a frozen ASI library, MPCR raises overall SR from 37.90% to 41.26% (relative +8.87%).

Base

40.97%

pooled SR

Perceptual

35.23%

pooled SR

Execution

18.69%

pooled SR

Fixed library · GPT-5 mini

ASI 37.90% → MPCR 41.26% +8.87%

paper.method SWOP · MPCR

Method

SWOP Analysis

Sliding-Window Optimistic Prior-Guided Planning

An idealized prior-guided adaptation model with a variation-dependent dynamic-regret guarantee. The learned library is the prior; recent feasibility evidence updates which actions remain reliable.

MPCR Executor

Masked Prior-Guided Candidate Reranking

A test-time, fixed-library executor-side method that reranks candidate actions using current-page feasibility evidence. Gains reflect action selection, not additional skill acquisition.

cite.bib BibTeX

Cite

bib 
@unpublished{chen2026skillshift,
  title  = {SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents},
  author = {Chen, Bo-Yu and Zhou, Zhi and Li, Yu-Feng},
  year   = {2026},
  note   = {Code, tasks, environments, and leaderboard: https://skillshift-bench.github.io/}
}