Contents
AWS and Google Cloud have open-sourced TorchNeuron and TorchTPU, making it easier to evaluate PyTorch workloads on Trainium and Cloud TPU. These releases could reduce migration work, but teams still need to validate compatibility, performance, and total cost against their existing NVIDIA environments.
Key Points#
- AWS released TorchNeuron, which runs PyTorch on AWS Trainium, as an open-source public beta.
- Google Cloud open-sourced TorchTPU, which runs PyTorch with hardware acceleration on Cloud TPU and its own infrastructure.
- Both companies say they support eager execution and
torch.compileto reduce changes to existing PyTorch code. - The practical test is whether the entire workload meets accuracy, speed, and cost requirements after migration.
What These Releases Change#
Announced alongside PyTorch Conference 2026 on October 20–21, these releases make previously restricted backends publicly available as open source. Developers can inspect the code and evaluate the implementations without joining the earlier limited-access programs. Running workloads still requires access to appropriate hardware.
The main change is closer integration with familiar PyTorch execution modes. According to AWS and Google Cloud, developers can retain more of their existing code while using the companies’ AI processors.
This matters because migration work can outweigh hardware price advantages. If switching processors requires rewriting operators, replacing libraries, or separately maintaining training implementations, it creates ongoing development costs.
How the Backends Work#
According to AWS, TorchNeuron can run PyTorch code on Trainium by changing the device specification to .to('neuron'), and supports eager execution, torch.compile, and standard PyTorch APIs. This should be treated as a starting point for evaluation, however, rather than a guarantee that every application needs only one change.
TorchNeuron also integrates with the Neuron Kernel Interface (NKI). This provides a way to implement custom operators and hardware-specific optimizations when standard operations do not deliver the required behavior or speed.
Google says TorchTPU helps developers run models and migrate to TPUs through eager execution optimizations, torch.compile integration, and native hardware acceleration. Claims of competitive training and inference performance need to be verified using the models, precision settings, and deployment conditions teams actually use.
Why Replacing CUDA Remains Difficult#
Getting a model to run correctly is only the first requirement. Many PyTorch applications depend on libraries, custom kernels, and tools developed around NVIDIA CUDA. Even if a backend supports standard PyTorch operations, an application may fail because one of its dependencies assumes a CUDA device.
Open-sourcing these backends allows developers to inspect their implementations and contribute to development. It does not establish compatibility across an entire application or prove that either processor is cheaper for a particular workload.
The question, then, is specific: Can your application run reliably on Trainium or TPUs at the required quality while reducing total cost? Broad claims about replacing NVIDIA cannot answer that question.
How to Validate a Migration#
Choose one representative workload and use measurements from the existing NVIDIA environment as the baseline. Keep the model, dataset, quality targets, and measurement methods consistent so the differences are meaningful.
Record results in the following four areas.
- Compatibility: Check required operators, third-party libraries, custom extensions, and checkpoint loading.
- Correctness: Compare training convergence and inference quality against predefined tolerances.
- Performance: Measure compilation and startup times separately from steady-state throughput, latency, and memory usage.
- Cost: Include accelerator runtime, supporting infrastructure, failed runs, and the development effort required to achieve acceptable results.
For training, compare the time and cost required to reach the same quality target. For inference, compare costs at the same latency target and request volume. Higher throughput alone is insufficient if output quality drops or response times exceed the application’s limits.
What Still Needs Checking#
Because TorchNeuron is in public beta, check supported configurations, known limitations, and available support before using it in production. For both projects, review licenses, installation requirements, supported hardware, and dependency versions before planning a deployment.
Validate operational procedures as well. Checkpoint recovery, profiling, distributed execution, and error diagnosis determine whether a successful experiment can become a maintainable service. Record all required code changes. Those changes are part of the cost of keeping the application running.
Next Steps#
Run one representative PyTorch workload on TorchNeuron or TorchTPU, then create a one-page comparison covering correctness, performance, total cost, and required code changes against the existing NVIDIA environment.
Source: https://www.publickey1.jp/blog/26/awsgoogle_cloudpytorchaws_trainiumgoogle_cloud_tpunvidia.html
Contents
