arXiv preprint · 2026
KAIST
Vision-language-action (VLA) models often struggle to generalize to skill combinations absent from their fine-tuning demonstrations, even when every constituent skill has been demonstrated. We focus on a vision shortcut as one failure mode: during fine-tuning, visual observations can serve as a proxy for the instruction, so a policy may execute a demonstrated combination associated with similar observations rather than the instructed combination. This motivates training with counterfactual pairs formed by holding a demonstration observation fixed while changing the instruction to specify an undemonstrated combination. These pairs, however, lack corresponding demonstrated action targets. Crucially, the currently required skill has already been demonstrated, but actions from those executions cannot serve as direct targets because the same skill can require different actions across observations. We propose CRAFT, which transfers supervision from demonstrated executions of the required skill to counterfactual pairs using skill representations that can be reused across executions of the same skill. Across three VLA models and two simulation benchmarks, CRAFT improves success on undemonstrated combinations while maintaining high success on demonstrated ones; it also improves compositional generalization on a real robot.
Interactive
Demonstrations cover every constituent skill but only same-color combinations. Select a combination to compare FT (Full) and CRAFT; cells show CRAFT’s success rate (%).
Motivation
We define a skill as an operation (e.g., pick, place) paired with the entity specified for that operation (e.g., red cube, blue plate), such as “picking a red cube” or “placing on a blue plate.” Fine-tuned VLAs often struggle to generalize to skill combinations absent from their fine-tuning demonstrations, even when every constituent skill has been demonstrated.
The six pick-and-place tasks of LIBERO-Goal define the demonstrated combinations (✓). Of the remaining nine object–target pairs, we evaluate five undemonstrated combinations (○), such as “put the wine bottle on the stove” highlighted.
Object–target combinations in LIBERO-Goal
| cabinet top | rack | plate | stove | bowl | |
|---|---|---|---|---|---|
| wine bottle | ✓1 | ✓4executed | ○ | ○instructed | |
| bowl | ✓2 | ✓3 | ✓6 | ||
| cream cheese | ○ | ○ | ○ | ✓5 |
✓ demonstrated (task index) · ○ undemonstrated, evaluatedblank: undemonstrated, not evaluated
Across four publicly released LIBERO-fine-tuned VLAs, mean success ranges from 88.8 to 97.4% on demonstrated combinations, compared with 2.0 to 50.9% on undemonstrated combinations.
Their failure behavior is consistent with a vision shortcut: visual observations can serve as a proxy for the instruction, so a policy may execute a demonstrated combination associated with similar observations rather than the instructed one.
LIBERO-Goal success (%), mean ± std over three seeds
| Model | Demonstrated 6 tasks | Undemonstrated 5 combinations |
|---|---|---|
| π0 | 94.6±0.8 | 2.9±0.4 |
| π0.5 | 97.4±0.6 | 50.9±2.4 |
| π0-FAST | 88.8±1.0 | 19.5±0.7 |
| GR00T N1.7 | 96.1±0.2 | 2.0±1.0 |
Rollouts of the fine-tuned π0.5 policy; task indices follow the ten original LIBERO-Goal tasks.
Method
To mitigate the vision shortcut, we form counterfactual pairs during fine-tuning by keeping a demonstration observation fixed while changing the instruction to specify an undemonstrated combination. These pairs lack corresponding demonstrated action targets, and actions from demonstrated executions of the required skill cannot serve as direct targets because the same skill can require different actions across observations. CRAFT transfers supervision from demonstrated executions of the required skill to counterfactual pairs using skill representations that can be reused across executions of the same skill.
01
Skill queries attend to both image and text tokens to produce a skill representation that reflects the currently required skill; state queries attend only to image tokens to produce a state representation. Together, they condition action prediction.
02 · ℒskill
Holding the state representation fixed, we replace only the skill representation with one from another execution. A contrastive objective over the resulting prediction errors encourages skill representations to be reusable across same-skill executions and to preserve distinctions between skills.
03 · ℒcf
We use the counterfactual pair’s skill representation to condition action prediction for a demonstrated execution of the currently required skill. We supervise this prediction with that execution’s action target. Thus, ℒcf trains the counterfactual pair’s skill representation to reflect the skill required under the counterfactual instruction in action prediction.
ℒtotal = ℒfm + β ℒskill + γ ℒcfoptimized jointly in a single training stage
Benchmarks
We introduce two compositional benchmarks in which the fine-tuning demonstrations cover every constituent skill but only a subset of the possible combinations. In both, the same-color combinations are demonstrated, and we evaluate the remaining 12 in Pick-Place and 6 in Pick-Place-Press. We collect 50 demonstrations per demonstrated combination and will release both benchmarks and their demonstration datasets.
Results
Standard fine-tuning often performs well on demonstrated combinations but achieves substantially lower success on undemonstrated ones, and changing the adaptation method alone does not close this gap. Across all VLA models and benchmarks, CRAFT achieves the highest success on undemonstrated combinations while maintaining high performance on demonstrated ones.
Success rates (%) on demonstrated (Demo.) and undemonstrated (Undem.) combinations; mean ± std over three evaluation seeds.
| Method | π0.5 | π0 | GR00T N1.7 | |||
|---|---|---|---|---|---|---|
| Demo. | Undem. | Demo. | Undem. | Demo. | Undem. | |
| Pick-Place | ||||||
| FT (Full) | 100.0±0.0 | 8.4±0.3 | 100.0±0.0 | 2.8±0.2 | 98.8±0.3 | 1.1±0.1 |
| FT (LoRA) | 100.0±0.0 | 46.3±0.9 | 97.5±0.0 | 1.0±0.2 | 97.8±0.6 | 2.6±0.3 |
| FT (Frozen) | 99.0±0.0 | 21.1±0.2 | 48.2±1.4 | 3.0±0.5 | 92.3±1.3 | 2.2±0.1 |
| CAG-VA | 96.3±1.5 | 66.5±0.5 | 50.5±1.3 | 6.8±0.6 | 88.7±1.3 | 4.9±0.5 |
| Skill Composition | 100.0±0.0 | 25.7±0.4 | 99.5±0.0 | 12.7±0.4 | 99.7±0.6 | 13.1±0.7 |
| CRAFT (ours) | 99.0±0.9 | 84.2±0.3 | 94.0±0.9 | 64.3±0.5 | 99.0±0.5 | 72.1±0.9 |
| Pick-Place-Press | ||||||
| FT (Full) | 100.0±0.0 | 0.7±0.0 | 99.7±0.6 | 0.3±0.0 | 97.7±2.3 | 1.1±0.2 |
| FT (LoRA) | 100.0±0.0 | 15.4±0.8 | 99.3±0.6 | 0.7±0.0 | 99.3±1.2 | 1.0±0.0 |
| FT (Frozen) | 98.0±1.0 | 0.9±0.2 | 82.0±3.5 | 1.5±0.4 | 97.7±0.6 | 1.1±0.2 |
| CAG-VA | 92.7±3.5 | 21.8±1.9 | 69.0±2.6 | 1.1±0.2 | 88.7±2.5 | 2.1±0.2 |
| Skill Composition | 99.7±0.6 | 5.8±0.2 | 99.3±0.6 | 0.3±0.0 | 99.3±0.6 | 2.0±0.0 |
| CRAFT (ours) | 99.7±0.6 | 62.6±3.9 | 95.3±0.6 | 40.7±1.5 | 99.3±0.6 | 31.9±1.1 |
Analysis
When an instruction change leaves the currently required skill unchanged, the policy continues executing that skill. When the required skill changes, the policy begins executing the newly required skill, from a state reached while executing a different skill.
Instruction
Holding the visual observation fixed, the predicted trajectories under CRAFT separate according to the instructed cube or plate, whereas predictions from FT (Full) largely overlap across instruction changes.
PickingFixed observation: picking the blue cube“pick the red/blue/green/yellow cube”
PlacingFixed observation: placing on the red plate“place it on the red/blue/green/yellow plate”
Trajectory colors indicate the instructed entity,with predicted action chunks shown for 30 random seeds (π0.5, Pick-Place).Insets enlarge the marked regions.
Without ℒskill, representations from different entities are more mixed within the pick and place groups. With ℒskill, they form entity-specific groups within each operation.
Replacing the skill representation with one from another execution of the same skill changes the prediction by only 4.0 ± 3.1% on average across recipients. In contrast, substitutions from different skills produce substantially larger changes.
ℒcf covers both instruction changes that alter the currently required skill (ℒchange) and those that leave it unchanged (ℒpreserve). Success on undemonstrated combinations remains low when ℒskill is added and improves with ℒchange. Combining ℒchange and ℒpreserve yields substantially higher success on undemonstrated combinations than using either counterfactual term separately. The combined objective also maintains high success on demonstrated combinations.
Pick-Place success (%), mean ± std over three seeds. All variants include ℒfm.
| Auxiliary objectives | π0.5 | GR00T N1.7 | ||||
|---|---|---|---|---|---|---|
| ℒskill | ℒpreserve | ℒchange | Demo. | Undem. | Demo. | Undem. |
| 88.8±0.6 | 1.9±0.4 | 99.3±0.3 | 3.8±0.1 | |||
| ✓ | 99.2±0.6 | 0.3±0.4 | 97.2±1.0 | 6.1±0.5 | ||
| ✓ | ✓ | 99.8±0.3 | 0.4±0.2 | 97.3±0.6 | 5.3±0.3 | |
| ✓ | ✓ | 97.7±0.3 | 14.7±0.8 | 94.5±1.3 | 12.1±0.4 | |
| CRAFT (all) | 99.0±0.9 | 84.2±0.3 | 99.0±0.5 | 72.1±0.9 | ||
Real robot
A Piper 6-DoF robot performs the Pick-Place tasks, with 20 demonstrations for each of the four same-color combinations. Over five trials per combination on π0.5, CRAFT achieves higher success than FT (Full) on undemonstrated combinations while maintaining high success on demonstrated combinations.
Real-robot success (successful / total trials)
| Method | Demo. | Undem. |
|---|---|---|
| FT (Full) | 19/20 | 9/60 |
| CRAFT (ours) | 19/20 | 43/60 |
Additional real-robot rollouts of CRAFT (π0.5) on undemonstrated Pick-Place combinations; all videos play at real-time speed.
@article{yang2026craft,
title = {Same Scene, Different Task: Skill Alignment for Compositional Generalization in {VLAs}},
author = {Yang, Taegeun and Na, Youngju and Cho, Yoonki and Yoon, Sung-Eui},
journal = {arXiv preprint arXiv:2610.00524},
year = {2026}
}