A research group calling itself DMIS Lab, the Data Mining and Information Systems Lab, has released UltraBERT, a family of free, open AI models for understanding English text. It also published findings that could change how such models are trained for search.
The group announced the release in a Hugging Face article on 14 October 2026. The models come in two sizes, 150 million and 380 million parameters, and can read up to about 32,000 tokens (roughly 24,000 words) at once. A third model, UltraBERT-Embed-150M, is ready to use for search. All are released under the permissive Apache 2.0 licence. All the benchmark results below are the authors’ own.
What is an encoder, or embedding model?
Chatbots like ChatGPT are decoders: they generate text one word-piece at a time. Encoders do something different. They read a piece of text once, in both directions, and turn it into a list of numbers, called a vector or embedding, that captures its meaning.
Texts with similar meanings get similar vectors. That makes encoders the engine behind:
- Search: finding documents that match a question, even if they use different words.
- RAG (retrieval-augmented generation): chatbots that look up relevant documents before answering. The documents are usually found with an embedding model.
- Classification: sorting text, such as flagging spam or judging quality.
“Encoders are still the most cost-effective way to process text at volume. Most RAG pipelines embed their documents with one,” the authors write. Because there is no word-by-word generation, encoders are cheap to run.
Finding 1: more training stops helping
The team says it ran experiments adding up to more than 70 trillion training tokens. It reports that “Encoders stop improving after 1–2T tokens, even while the loss keeps going down”. In other words, the model keeps getting better at its training exercise, but not at real tasks.
Finding 2: masking decides what the model is good at
Encoders are usually trained by hiding, or “masking”, some words and asking the model to guess them. The authors found the share of hidden words matters a lot:
- Low masking (hiding few words) gives better results at classification.
- High masking (hiding about half) gives better results at retrieval, meaning search. “50% masking gives the best BEIR score at every scale,” they write. BEIR is a standard search benchmark.
“Low masking wins classification, high masking wins retrieval,” the authors say. They add that every existing encoder picks one.
UltraBERT tries to get both. “We mix 5% and 50% masking in every batch, give each ratio its own LM head, and distill from a 1.3B teacher only where it helps,” the team writes. They found learning from a larger “teacher” model helped classification but hurt search at high masking, so they use it only at the low setting.
Why it matters for search and RAG: if you train an encoder for classification, it may be weaker at finding documents. Teams building search or RAG systems may get better results from models trained with heavier masking.
The authors’ results
According to the team:
- UltraBERT scored top on 9 of 10 benchmarks at both sizes, including 91.0 on GLUE and 47.7 on BEIR for the 380M model.
- It kept 99% accuracy on a “passkey” test, finding a hidden item, all the way to 32,000 tokens.
- UltraBERT-Embed-150M scored 67.24 on the MTEB English benchmark. The authors call it “the best dense embedding model under 200M parameters”, trained without copying from a larger embedding model.
The models need a recent version of the Transformers library (version 5 or later) and the trust_remote_code=True setting, according to the model cards.
What this means
- Small, open search models keep improving. A 150M-parameter model can now read long documents whole.
- Training choices matter more than size. The authors say masking ratio, not just more data, decides what an encoder is good at.
- The results are self-reported. Independent tests have not yet confirmed them.
Sources
- DMIS Lab (Data Mining and Information Systems Lab), “UltraBERT”, Hugging Face community article, 14 October 2026. https://huggingface.co/blog/dmis-lab/ultrabert
- Model card, UltraBERT-150M. https://huggingface.co/dmis-lab/UltraBERT-150M
- Model card, UltraBERT-380M. https://huggingface.co/dmis-lab/UltraBERT-380M
- Model card, UltraBERT-Embed-150M. https://huggingface.co/dmis-lab/UltraBERT-Embed-150M
