Contents

To adapt TRELLIS.2 to a specific object category, train LoRA adapters for the structure and shape models, then evaluate generated meshes on held-out objects. The provided workflow runs in Docker or Hugging Face Jobs, and the footwear experiment uses 188 objects while keeping material generation frozen.

Key Points#

  • For geometry problems, train the sparse structure and shape models while keeping the texture model frozen.
  • Preprocess meshes once to create training latents and rendered images, then reuse them across experiments.
  • The reference configuration trains approximately 9.3 million adapter parameters across the two models.
  • Inspect generated meshes on held-out objects before concluding that lower training loss means better geometry.

Choose What to Train#

TRELLIS.2 separates coarse structure, detailed geometry, and materials. This separation lets you address specific defects without updating every component.

The problem reported in the footwear experiment was inaccurate sole geometry. This workflow adds LoRA adapters to two diffusion Transformers: the sparse structure model, which determines coarse occupancy, and the shape model, which generates detailed geometry. The original model weights remain frozen.

Start with the following reference configuration.

| Component | Configuration | |---|---| | Sparse structure model | Rank-8 LoRA on attention and feed-forward layers | | Shape model | Rank-16 LoRA on attention layers | | Texture model | Frozen | | Shape and structure encoders | Frozen during preprocessing |

Prepare the Dataset#

Start with meshes uploaded to a Hugging Face dataset repository. The container accepts GLB or OBJ assets and converts them into the latents needed for training. You also need an HF_TOKEN with the required repository permissions and approved access to the gated DINOv3 model.

The preprocessing container uses Blender to render conditioning images from 16 views per object and saves encoded training latents. During training, each sample uses a randomly selected rendered image. The reference configuration splits the footwear objects into 179 for training and 9 for evaluation. Keep images of held-out objects out of the training set, regardless of viewpoint.

Run preprocessing on Hugging Face Jobs.

snippet.txt
export HF_TOKEN=hf_xxx

hf jobs run -d --flavor a100-large --timeout 6h \
  --secrets HF_TOKEN \
  cr.hf.co/suvadityamuk/trellis2-finetuning-prep:latest \
  trellis2-prep \
  --meshes your-username/my-meshes \
  --push-to your-username/my-meshes-trellis2 \
  --private --holdout 0.05 \
  --views 16 --workers 8

Run Two Training Jobs#

Use the ss and shape presets to train the structure and shape adapters separately. Both use the prepared dataset but select different models and configurations. Separate upload paths keep their checkpoints distinct.

The commands below configure each run for 3,000 optimizer steps, saving every 500 steps. These are experimental settings, not values proven optimal for every dataset. Trackio logs metrics so you can monitor training progress.

snippet.txt
hf jobs run -d --flavor a100-large --timeout 12h \
  --secrets HF_TOKEN \
  cr.hf.co/suvadityamuk/trellis2-finetuning-train:latest \
  trellis2-train --config ss \
  --latents your-username/my-meshes-trellis2 \
  --push-to your-username/my-trellis2-lora \
  --run-name ss_lora --push-path ss_lora \
  --trackio-space your-username/trellis2-trackio \
  --set max_steps=3000 i_save=500

hf jobs run -d --flavor a100-large --timeout 12h \
  --secrets HF_TOKEN \
  cr.hf.co/suvadityamuk/trellis2-finetuning-train:latest \
  trellis2-train --config shape \
  --latents your-username/my-meshes-trellis2 \
  --push-to your-username/my-trellis2-lora \
  --run-name shape_lora --push-path shape_lora \
  --trackio-space your-username/trellis2-trackio \
  --set max_steps=3000 i_save=500

Understand Resource Costs#

The reported preprocessing run took about one hour to process 188 objects on an a100-large job with 80 GB of GPU memory. The six-hour preprocessing and twelve-hour training limits are job timeouts, not measured completion times. The provided materials do not specify minimum GPU requirements or total training costs.

LoRA reduces trainable parameters and their optimizer states, but the backbone still consumes memory. This implementation uses gradient checkpointing, keeps trainable adapter weights in fp32, and applies bf16 autocasting during the forward pass. The example uses a batch size of 4 with 4 gradient accumulation steps, giving an effective batch size of 16.

Locally, run the same container commands in Docker with GPU access enabled. Pass HF_TOKEN to each container that needs Hub access, and mount a persistent Hugging Face cache to avoid repeated downloads.

Review Results and Limitations#

Do not train the texture model on unsuitable training targets. The GSO assets used here contain diffuse maps with baked-in lighting rather than the separate material maps this workflow requires. Training on these could teach the model to reproduce lighting as a surface property.

Compare saved checkpoints against the unchanged base model using the same held-out conditioning images and generation settings. Check sole thickness and contours, and inspect exported meshes from other angles. Rendered appearances alone can hide geometry errors.

The provided materials describe the experiment's goals and training configuration but offer no quantitative results demonstrating improvement. The nine held-out objects are useful for an initial check but do not establish broad generalization. Also, because latest image tags can change, record the container digest and configuration used for each run.

Next Steps#

Before preprocessing your own meshes, inspect the preprocessed dataset at suvadityamuk/gso-footwear-trellis2-latents ↗ to understand the expected training data structure.

Source: https://huggingface.co/blog/3d-lora-finetuning

Contents