Contents
Transformers Model Integration Contracts (MIC) use small, fixed models to check whether code changes break real usage paths. They provide useful regression checks for pull requests, but complement tests using full checkpoints and require human-reviewed baselines.
Key Points#
- MIC bundles small model fixtures, executable usage examples, and reviewed expected outputs.
- Fixtures preserve relevant architecture and preprocessing while keeping checks inexpensive to run.
- An experimental UMT5 fixture detected a real attention regression on CPU using 66 MB of weights.
- Coverage depends on the code paths tested, and some bugs require real checkpoint weights.
What Contracts Check#
MIC tests whether a model continues to behave as expected when used. Successfully loading it and completing a forward pass is not enough. A model can run without errors yet produce incorrect results.
Each contract combines a version-pinned fixture, an executable usage path written in Markdown, and a recorded baseline. Text generation contracts can compare generated token IDs, check that training loss and gradients are finite, and verify that an optimizer step changes the weights. Baselines require human review. Recording outputs alone does not establish their correctness.
Contracts target architectures and usage paths. Compatible checkpoints can share fixtures while retaining their own expected results. The fleet dataset pins contract revisions and records the package versions used to establish baselines. Each CI run also records the fleet revision for reproducibility.
Why Execution Paths Matter#
The strongest reported example is a UMT5 regression introduced on July 30, 2026, when Transformers enabled PyTorch’s scaled dot-product attention (SDPA) for T5. This allowed the UMT5 decoder to see future tokens during training. Nothing crashed, but the model could access information it was supposed to predict.
When replaying the Transformers commit history, the small UMT5 fixture detected anomalies from the regression’s introduction until its fix on September 28. Label log probabilities differed from the reviewed baseline by 1.22, and training loss by about 0.20. This experimental fixture had 66 MB of weights, far below the original checkpoint’s 1,227 MB.
The contract caught the bug through SDPA, the default attention path. Testing only eager attention missed it. When a maintainer reverted the fix on a validation branch, the selected T5 contract failed again. Its execution step took 13.4 seconds on CPU, excluding setup.
Keeping Fixtures Representative#
Shrinking a model requires preserving the components that exercise its implementation. Reducing depth without checking layer types can remove the very behavior the test should cover.
Fixture creation follows four steps:
1. Reduce dimensions. Shrink hidden sizes, attention head counts, and depth while preserving relevant layer types, such as sliding and full attention, and dense and mixture-of-experts layers. 2. Preserve preprocessing. Keep the vocabulary, special tokens, input dimensions, and tokenizer and processor files unchanged. 3. Validate weights. Check that changing inputs changes behavior, then reload the saved model and verify that outputs, loss, and gradients are finite. 4. Freeze results. Review reproducible baselines on supported platforms and numerical precisions, then publish pinned snapshots for reuse in CI.
This process also preserves the original weight file format so loading tests exercise the relevant conversion code. The reported Llama 4 Scout fixture reduced the parameter count from 108.6 billion to 44.7 million while preserving the layer types and MoE routing scheme under test.
Creating and Reviewing MICs#
The tmic CLI drafts contracts, builds fixtures, and measures candidate baselines. For new models, an LLM drafts usage code and manifests through Inference Providers. Humans approve the expected values.
Install the runtime and documentation dependencies, authenticate with Hugging Face, and run the local workflow with your checkpoint.
pip install "transformers-mic[runtime]" \
"hf-doc-builder @ git+https://github.com/huggingface/doc-builder.git"
hf auth login
tmic run org/model --local-only --fixtures-only --no-history --bill-to ''Replace org/model with the original checkpoint. This workflow requests review at each stage and stops before uploading. LLM requests are billed to your account. Before accepting outputs, check that the usage examples exercise meaningful user operations and the intended attention backend.
Coverage Limits#
Random weights can reduce maintenance costs, but may weaken a test’s ability to detect bugs. A random Whisper fixture barely responded to changes in audio. Training it on a few sound–word pairs made behavior along that input path easier to observe. In one study, an overfitted fixture detected all 24 injected faults, while a random fixture detected 18.
This result alone does not establish better coverage of real regressions. Both missed an assisted decoding bug that surfaced with a real Whisper checkpoint. Retain full integration tests, and use overfitting when random weights provide insufficient sensitivity.
As of November 1, 2026, the Transformers MIC CI job reported failures without blocking pull requests. Justifying blocking requires continued measurement of actual detection rates, false positives, and baseline review effort.
Next Steps#
Choose an unsupported model listed in the project’s coverage Space and run the local tmic workflow. Then review whether the usage path exercises the backends and behavior that users depend on.
Source: https://huggingface.co/blog/transformers-mic
Contents
