Contents

LingBot-VA is a public project for robot control that combines camera video prediction and robot action estimation in a single model. It reports high success rates on RoboTwin 2.0 and LIBERO, but running it requires a compatible environment and adjustments to data formats.

Key Points#

  • Its autoregressive model alternates between changes in video and actions, aiming to handle long tasks and unfamiliar situations.
  • Set attn_mode to flex for training, and to torch or flashattn for inference and evaluation.
  • It reports success rates of 92.9% on Easy and 91.6% on Hard in RoboTwin 2.0 evaluations.
  • Fine-tuning on custom data requires conversion to LeRobot format, descriptions of action segments, and extraction of video latent representations.

Handling Video and Actions Together#

LingBot-VA’s defining feature is that it predicts visual state changes and estimates actions within the same sequence, rather than simply outputting robot actions. The model alternates between video and actions, but their roles are designed separately. The idea is to use future video as a cue for deciding the next operation.

To improve execution efficiency, it combines a dual-stream MoT architecture with asynchronous execution and a KV cache. This design targets higher success rates on long tasks and better generalization to unfamiliar situations. Its actual effectiveness depends on the tasks, data, and evaluation conditions.

Set Up the Inference Environment First#

The basic environment listed in the README is Python 3.10.16, PyTorch 2.9.0, and CUDA 12.6. Install the specified PyTorch build for CUDA 12.6, along with matching dependencies such as diffusers==0.36.0 and transformers==4.55.2. GPU use also requires flash-attn.

Before loading the model, open transformer/config.json in the model folder and change attn_mode to suit your use case. Use flex for training and torch or flashattn for inference and evaluation. The documentation warns that evaluating with flex will cause an error.

For single-GPU evaluation on RoboTwin 2.0, the estimated VRAM requirement is about 24 GB with the VAE and text encoder offloaded to the CPU. Inference that generates video and actions from an image is listed as requiring about 18 GB. These are estimates from the repository; check actual requirements for your settings and environment.

Running Evaluations#

For RoboTwin 2.0, check out the specified commit, 2eeec322, and prepare the dependencies and assets. Then start the inference server and client. For a single GPU, use evaluation/robotwin/launch_server.sh for the server and evaluation/robotwin/launch_client.sh for the client. The server and client must run on the same machine.

LIBERO also provides server and client launch scripts. Follow the official setup instructions for the benchmark you want to evaluate before running them. RoboTwin’s multi-GPU client divides tasks into seven groups; set task_group_id to a value from 0 to 6.

Interpreting the Published Results#

Across 50 RoboTwin 2.0 tasks, LingBot-VA reports average success rates of 92.9% on Easy and 91.6% on Hard. Motus, listed in the comparison table, achieves 88.7% and 87.0%, respectively. LingBot-VA’s average success rate on LIBERO is 98.5%. These are benchmark results published in the repository, however, and do not guarantee the same performance on your robot or data.

The real-world evaluation is described as covering six tasks with 50 trials each. For example, on “Unpack Delivery,” LingBot-VA achieves a 65.0% success rate, compared with 25.0% for π₀.₅. Meanwhile, its progress score on “Fold Clothes” is 48.8%, showing that results vary by task. Looking at progress scores, which indicate how far a task advanced, alongside success rates makes the evaluation easier to interpret.

Fine-Tuning on Custom Data#

Fine-tuning inputs use LeRobot format. Add action_config to each episode’s entry in meta/episodes.jsonl, recording the start and end frames of each action segment and a natural-language description. For example, a continuous action spanning 450 frames would use start_frame: 0 and end_frame: 450.

Training actions are standardized to 30 dimensions: seven each for the left and right end effectors, seven each for the left and right joints, and one each for the left and right grippers. Fill unused dimensions with zeros. As a guideline, resize video to about 256×256 pixels at 5–15 fps, then extract latent representations with the Wan2.2 VAE.

Place the latent data in latents/, following the same directory structure as the source videos, and name files episode_{index}_{start_frame}_{end_frame}.pth. Training uses FSDP, and the README provides eight-GPU examples for both RoboTwin and LIBERO. Since a single GPU may not support training under the same conditions, first check your data format and available computing resources.

Next Steps#

Start by checking the GitHub README for the environment setup instructions for your benchmark and the model settings. To try inference first, download the checkpoint, set attn_mode to torch or flashattn, and launch the evaluation scripts. For training on custom data, a concrete starting point is to convert one episode to LeRobot format and verify that action_config matches the latent files.

Source: https://github.com/robbyant/lingbot-va

Contents