Contents
Ai2’s ToyBlocks gives researchers a controlled environment for testing what robot models learn and why they fail. Its greatest value lies in reproducible experiments. Researchers can generate training data for specific goals, change one condition, and measure where problems arise in planning, perception, or physical control.
Key Points#
- ToyBlocks combines a simulated robot arm, tabletop blocks, automated demonstration generation, and detailed evaluation metrics.
- Researchers can vary training data and test unfamiliar block arrangements, appearances, stack heights, and workspace regions.
- Ai2’s experiments reveal weaknesses in spatial generalization, letter recognition, and predicting physical interactions.
- Simulation results support controlled comparisons, but the supplied report does not establish reliable transfer to real robots.
What ToyBlocks Provides#
Released on November 6, 2026, ToyBlocks is Ai2’s first FoundationWorld: a framework combining a narrowly scoped simulation with tools for generating tasks, creating training demonstrations, and evaluating behavior.
The robot pushes, flips, rotates, lifts, and stacks blocks. Letters and pictures on each face add perception and reasoning challenges. It may need to reveal hidden faces before deciding which blocks to use. Researchers can vary block sizes, table height, and other physical properties to test how performance changes.
The narrow scope makes experimental results easier to interpret. Researchers can hold tasks constant while changing the composition of training data, or hold training conditions constant while moving tests beyond the familiar workspace region. Evaluation records execution time, force, and task progress alongside success rates.
How Demonstrations Are Generated#
ToyBlocks combines reinforcement learning and motion planning to create short movement skills. It uses Nvidia’s Isaac Lab to train these skills. Policies learn contact-based actions such as pushing and flipping, while a task-and-motion planner selects and sequences skills and plans movements between them.
Executing these plans produces demonstrations labeled with tasks and skills. According to Ai2, a single GPU ran hundreds of plans in parallel, reaching speeds of up to 24 times real time. This is the reported maximum; throughput needs to be checked for individual configurations.
Data generation can use exact simulator information, such as block positions and orientations. Models trained to act from raw sensor observations, however, must solve perception problems themselves. This distinction needs to be clear when comparing the demonstration generator with learned robot policies.
What the Experiments Show#
Ai2 tests spatial generalization using ManiFlow, π0.5, and MolmoAct 2 on the Stack, Unstack, Flip, and Rotate tasks. The tests distinguish interpolation from extrapolation. Training on stacks of two and five blocks and testing on three and four differs from training in the left workspace region and testing on the right. The reported early results suggest that large-scale pretraining does not guarantee strong generalization.
For long tasks, ToyBlocks separates high-level planning from low-level execution. A planner selects skills and target blocks, and an action model executes the movements. Reference planners and controllers help identify which component caused a failure. In one letter-search and word-building experiment with a state-of-the-art agent, the agent completed the task or correctly determined it was impossible in 63% of cases. Half the scenes lacked the required letters. Confusion between similar letters occurred in 13% of tasks, and incorrect letter orientation in 15%.
Tool descriptions also affected performance. In another experiment, descriptions written by the agent after trying its own skills raised the success rate from 24% with default descriptions to 63%. Descriptions written by human experts yielded 36%. This result shows the value of checking whether a model understands its available actions before attributing failures solely to planning ability.
What the Results Do Not Establish#
ToyBlocks also evaluates action-conditioned world models, which predict how a scene changes after a robot acts. Ai2 generated 900 trajectories for each of four skill settings, split evenly between successes and failures. In these tests, Wan 2.2 performed best at predicting block motion and task outcomes. Astra struggled with contact, while LeWM accumulated errors over long trajectories.
In an additional experiment, adding 100 trajectories generated by the best-performing video model raised policy success in an unseen region from 19% to 78%. This result comes from a specific experimental setup and does not show that generated data will deliver comparable gains for other tasks or robots.
The supplied material ends before presenting real-robot evaluation results. It therefore supports conclusions about controlled simulation experiments, while transfer to the real world remains unresolved. Before adopting ToyBlocks for a real-robot project, researchers need to check the full report for transfer results, robot configurations, and failure rates.
Next Steps#
Run one transfer experiment that changes the workspace region. Train a Flip policy on blocks placed in the left region, then measure success rates over 100 held-out trials on each side while keeping block appearance and all other settings fixed.
Source: https://huggingface.co/blog/allenai/toyblocks
Contents
