355 families · 3 websites · 6 conditions

skillshift.cover Issue 10-slot

SkillShift-Bench measures whether a web agent’s learned skills still execute after the website itself has evolved.

GitLab 162 · Magento 114 · WordPress 79 · src, src-perc, src-exec, tgt, tgt-perc, tgt-exec

SR readout

ASI · Mixed

Overall SR 36.29

GitLab 20.81
Magento 27.78
WordPress 60.27

src

Contextual 53.65 src

Perceptual 48.80 src-perc

Execution 24.43 src-exec

tgt

Contextual 41.84 tgt

Perceptual 33.19 tgt-perc

Execution 15.79 tgt-exec

AWM · Mixed

Overall SR 35.29

GitLab 17.71
Magento 25.49
WordPress 62.66

src

Contextual 50.07 src

Perceptual 43.35 src-perc

Execution 22.45 src-exec

tgt

Contextual 41.28 tgt

Perceptual 37.65 tgt-perc

Execution 16.90 tgt-exec

WALT · Mixed

Overall SR 34.80

GitLab 21.07
Magento 27.95
WordPress 55.38

src

Contextual 46.82 src

Perceptual 43.53 src-perc

Execution 25.46 src-exec

tgt

Contextual 41.17 tgt

Perceptual 32.74 tgt-perc

Execution 19.09 tgt-exec

SkillWeaver · Mixed

Overall SR 20.15

GitLab 13.78
Magento 14.37
WordPress 32.29

src

Contextual 35.13 src

Perceptual 28.75 src-perc

Execution 15.63 src-exec

tgt

Contextual 17.80 tgt

Perceptual 13.85 tgt-perc

Execution 9.71 tgt-exec

MPCR · GPT-5 mini

Overall SR 41.26

GitLab 23.20
Magento 39.18
WordPress 61.39

src

Contextual 56.67 src

Perceptual 57.09 src-perc

Execution 33.23 src-exec

tgt

Contextual 43.32 tgt

Perceptual 37.24 tgt-perc

Execution 19.99 tgt-exec

ASI · GPT-5 mini

Overall SR 37.90

GitLab 23.00
Magento 32.26
WordPress 58.44

src

Contextual 52.79 src

Perceptual 53.02 src-perc

Execution 26.68 src-exec

tgt

Contextual 42.83 tgt

Perceptual 34.32 tgt-perc

Execution 17.75 tgt-exec

Empty = unmeasured · -- = N/A · oxide = execution

Open the leaderboard →

shift.modes 3 named

Three named shifts

A skill has to be invoked, grounded, and executed. SkillShift-Bench names those three failure modes as run conditions. Execution is the steepest SR drop in every launch series.

Contextual: Same task after the site’s version or theme changes. Routes and chrome move; the goal does not.
Perceptual: Same task under altered layout, DOM, and locator evidence. The objects stay aligned.
Execution oxide: Same task with reproducible runtime blockers. This is where learned skills collapse on the launch board.

Schematic toy admin UI, not a live capture. Contextual moves chrome, routes, and object placement. Perceptual scrambles locators, then settles on the target. Execution raises an oxide runtime blocker. 0 is the source the skill was learned on.

slope.json Table 1 · 3

Execution drop, from the launch JSON

Two instrument banks, not one ranking. Lines are SR by condition. The table is the source of truth.

Mixed backbones — Table 1

Mixed backbones (Table 1). Oxide bands mark src-exec and tgt-exec. Drawn from launch submissions, not a new aggregation.

SR (%) by condition, mixed backbones

Method	src	src-perc	src-exec	tgt	tgt-perc	tgt-exec
ASI	53.65	48.80	24.43	41.84	33.19	15.79
AWM	50.07	43.35	22.45	41.28	37.65	16.90
WALT	46.82	43.53	25.46	41.17	32.74	19.09
SkillWeaver	35.13	28.75	15.63	17.80	13.85	9.71

gpt-5-mini — Table 3

gpt-5-mini (Table 3 comparison). Same six conditions. Drawn from the ASI and MPCR launch submissions.

SR (%) by condition, gpt-5-mini

Method	src	src-perc	src-exec	tgt	tgt-perc	tgt-exec
MPCR	56.67	57.09	33.23	43.32	37.24	19.99
ASI	52.79	53.02	26.68	42.83	34.32	17.75

getting.started 3 lines

Getting started

pip install -e 'skillshift-bench[run]'
skillshift calibrate
skillshift run --agent your_module:Factory --all --out runs/latest

Full quickstart →

cite.bib BibTeX

Cite

bib

@unpublished{chen2026skillshift,
  title  = {SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents},
  author = {Chen, Bo-Yu and Zhou, Zhi and Li, Yu-Feng},
  year   = {2026},
  note   = {Code, tasks, environments, and leaderboard: https://skillshift-bench.github.io/}
}