Contents
Hugging Face’s torch-neuronx model support Space shows which tested models run on AWS Trainium without modifications and provides scripts to reproduce each result. Check compatibility before committing to deployment, then test with your own workload. A passing result covers only a short generation test.
Key points#
- Search for your model in the Space and select the Neuron compiler and Transformers versions that match your intended environment.
- Each tested model shows its result, core count, total runtime including compilation, and peak device memory usage.
- The provided
run.pyreproduces the test and automatically launches distributed processes when the model requires sharding. - PASS covers a 32-token generation test, not production throughput, long outputs, or all model features.
Find your model#
Open Hugging Face’s torch-neuronx model support Space. The default Selected view displays models curated from those with high download counts or trending scores.
To find a specific checkpoint, switch to Trending or Downloads, select the relevant task, and enter part of the model name. These views follow Hub rankings and include untested models. Being untested is not evidence of incompatibility. For models intentionally excluded, the Space lists the reason. Some exclusions reflect priorities rather than technical constraints.
Select the neuronx-cc compiler and Transformers versions you plan to use. Treat compatibility as a result for a particular combination of model and software versions, rather than a permanent property of a checkpoint.
Interpret the results correctly#
Each tested row shows whether the model worked, the number of Trainium cores used, elapsed wall-clock time including compilation, and peak device memory usage. These figures help you reproduce the test and understand resource requirements. They do not indicate serving performance.
Each test loads the model onto a Trainium2 instance and splits it across the minimum number of cores needed to hold its weights. Most models use tensor parallelism. Mixture-of-Experts models without a tensor parallel plan use expert parallelism alone.
The test generates 32 tokens and checks the output. For single-core runs, it compares the tokens with those produced by running the same checkpoint on a CPU. For sharded models, a small judge model checks the generated text for English fluency. These checks provide different strengths of evidence. A fluency check cannot verify token-by-token agreement with a reference run.
Reproduce the test#
Selecting a model lets you inspect the Neuron stack, additional packages, and actual run.py script used for the test. Launch a Trainium2 (trn2) instance with that stack, using an AWS Neuron Deep Learning Container or a similar setup, and install the listed packages.
The script loads the model with from_pretrained, applies sharding as needed, and generates output using a static cache and torch.compile(backend="neuron"). Save the provided script as run.py and run:
python3 run.pyYou do not need to launch it with torchrun. When sharding is required, the script launches each torch.distributed rank itself. Matching the published environment closely on your first reproduction makes failures easier to diagnose.
Account for compilation#
The first run compiles the model’s graphs. The reported wall-clock time includes this work, so do not interpret it as steady-state generation latency or use it directly to estimate requests per second.
The Space also provides a command to download graphs from the public NEFF cache using the compiler version corresponding to the test. Use that command to reuse published compiled graphs and skip compilation when reproducing the run.
When evaluating your application, measure startup and generation separately. A successful cached run and a fresh instance that requires compilation before serving requests establish different things about operational behavior.
Understand the scope limits#
According to the October 23, 2026 announcement, testing covers Transformers models for the text-generation and image-text-to-text tasks. Models must fit within four Trainium2 devices, or 16 cores. Diffusers, sentence transformers, and other Transformers tasks were planned extensions and were not included in the verified support scope.
PASS does not validate longer generation or throughput. Before deploying, test your expected prompt lengths, output lengths, concurrency, and required features. For models that handle images and text, also check the image inputs you will actually use. Reported results do not guarantee support for every input format or use case.
For automated checks, query the public model support dataset ↗. Each row includes the result, any failure error, stack versions, and the script used. Result labels are defined in the dataset card, so review those definitions before treating different statuses as equivalent.
Next steps#
Find your checkpoint in the Space, select your intended compiler and Transformers versions, and run the provided run.py in the corresponding Trainium2 environment.
Source: https://huggingface.co/blog/torch-neuronx-model-support
Contents
