Contents
Noetra reports that its image encoder, trained from random initialization, roughly matched SigLIP2 on English benchmarks and outperformed it in Japanese and Chinese evaluations. Multimodal LLM experiments also showed improved document understanding, though performance did not improve on every task.
Key Takeaways#
- Noetra trained on 2.44 billion filtered image–text pairs. Japanese accounted for about 9.0%, and Chinese for about 3.4%.
- Training used 256 NVIDIA B200 GPUs and a contrastive learning objective, processing 40 billion samples in five days.
- On XM3600 Japanese text-to-image retrieval, recall@5 reached 97.28%, compared with 71.92% for SigLIP2.
- Higher scores for an encoder alone do not necessarily translate into better multimodal LLM performance, making downstream evaluation essential.
What the Results Show#
Noetra used the same architecture as ViT-SO400M-14-SigLIP2, randomly initializing both the image and text encoders. The comparison therefore tested different data and training pipelines without introducing a new encoder architecture.
The English results were competitive but not uniformly better. ImageNet-1K accuracy declined slightly, while ObjectNet and several retrieval scores improved. The largest reported gains appeared in Japanese evaluations.
| Benchmark | SigLIP2 | Noetra | |---|---:|---:| | ImageNet-1K accuracy | 83.18 | 82.65 | | ObjectNet accuracy | 75.23 | 79.81 | | XM3600-ja text→image R@5 | 71.92 | 97.28 | | WAON-Bench accuracy | 79.26 | 98.20 | | XM3600-zh text→image R@5 | 71.17 | 86.78 |
Build the Data Pipeline First#
The training data combined 10 sources, including DataComp, ReLAION, WAON, OBELICS, and OmniCorpus. The reported counts for these sources total approximately 2.44 billion pairs. Documents containing multiple images and passages required an extraction step: split the text into sentences, use SigLIP2 to score how well each image matches each sentence in the document, and select the highest-scoring sentence as its caption.
The data then went through four filtering stages.
- Remove pairs missing an image or text, images outside the allowed 1:3 aspect-ratio range, and images below the 50-pixel size threshold.
- Remove text with five or fewer characters, retaining only English, Japanese, and Chinese.
- Filter by image–text similarity using SigLIP2 for English and Chinese and
clip-japanese-base-v2for Japanese. - Use SemDeDup to remove semantically duplicate images, retaining pairs with higher image–text similarity scores.
Choosing filtering models by language matters. Noetra found that the Japanese model was more accurate at filtering Japanese pairs. A single scoring model should not be assumed to work equally well across all languages.
Add Captions Selectively#
Noetra also generated detailed descriptions for 700 million images using a model identified in the report as gemma-4-31B-it. Following the CLIPS approach, it sampled one sentence from each generated description during training instead of using the entire caption.
Small-scale experiments revealed a clear trade-off. ImageNet-1K accuracy fell from 76.18% to 75.96%, while recall@5 for MSCOCO image-to-text retrieval rose from 86.24% to 88.84%. Synthetic descriptions helped retrieval but slightly reduced classification accuracy.
The team believes the generated descriptions capture the broad content of an image while losing some proper names and fine details found in web captions. This explanation is plausible, but the reported experiments did not isolate the cause. Treat caption regeneration as an option whose effectiveness should be tested on the intended task.
Train Within a Fixed Budget#
The final training run used 224 × 224 images, a global batch size of 128K, a peak learning rate of 0.0005, and 2,000 warmup steps. Under a cosine learning rate schedule, training processed 40 billion samples, equivalent to roughly 16 passes through the image collection.
Training ran for five days on 32 nodes with 256 NVIDIA B200 GPUs. Even with a simple contrastive learning objective, the compute requirements were enormous. At this scale, data augmentation through random resizing, color jitter, and grayscale conversion still proved effective.
Noetra also tried a Warmup-Stable-Decay schedule. It looked promising in small-scale experiments, but training stalled for an extended period at larger scale. The cosine schedule ultimately performed better, demonstrating the need to validate optimization findings from small experiments at the target scale.
Check the Limits on Downstream Tasks#
In the LLM Noetra is developing, replacing SigLIP2 increased the average score across eight vision benchmarks from 0.6050 to 0.6350. DocVQA rose from 0.8706 to 0.9138, and InfoVQA from 0.5866 to 0.6732. However, MMMU and JMMMU declined. Three benchmarks used an LLM as a judge, so the average combines different evaluation methods.
Continued training at higher resolution revealed another caveat. Training at 448 × 448 improved ImageNet accuracy from 82.65% to 83.89%, but did not improve the downstream multimodal LLM. Noetra therefore continued using the 224 × 224 encoder.
“From scratch” refers to weight initialization, not independence from existing models. SigLIP2 was used for caption selection and filtering, while a pretrained VLM generated descriptions. These findings were reported by the developer; broader conclusions about reproducibility and suitability for production still require independent replication and further checks of benchmark data overlap and source licenses.
Next Steps#
Run a controlled experiment in your target multimodal LLM, swapping the encoder while keeping the language model, training data, and evaluation procedure fixed, to determine whether the reported document-understanding gains carry over to your workload.
Source: https://zenn.dev/noetra_techblog/articles/0cea39ce67d2c2
Contents
