Learn AI Create the Future — with Pranay
Main site

The BERT Family

Continued pretraining on your own domain

Domain-adaptive pretraining keeps training BERT's original masked language modelling objective on your own unlabelled text, before fine-tuning, so the model already knows your domain's vocabulary and patterns.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Domain-adaptive pretraining keeps training BERT on your own unlabelled text. It uses the same fill-in-the-blank method it already knows, so it picks up your domain's vocabulary before you ever fine-tune it.

Think about a doctor who trained in a general hospital, now moving to a specialist burns unit. They already know medicine broadly — anatomy, general treatment, how to read a chart. What they need is time immersed in this unit's terminology, equipment names and typical cases, before they can make specialist-level decisions.

BERT pretrained on general web text and books already knows English broadly. It has never seen "chargeback", "UPI PIN" or "RRN" used the way a fintech support team uses them. Domain-adaptive pretraining is that immersion period. It is more of the same fill-in-the-blank training BERT already knows how to do, applied only to your text instead of general web text.

Why it exists

Fine-tuning, covered in the previous lesson, teaches BERT a specific labelled task. But fine-tuning data is usually small — a few hundred or thousand labelled examples. BERT's pretrained vocabulary and patterns come from general text that may not resemble your domain's language at all.

Say BERT has never encountered "chargeback" the way your support tickets use it. Fine-tuning alone, on a small labelled dataset, may not teach it the word's role well. Domain-adaptive pretraining bridges this gap. It keeps training with the same self-supervised masked language modelling objective from earlier in this section. This time it trains on your own unlabelled domain text, usually far more plentiful than labelled examples. Only after that does the smaller, labelled fine-tuning step happen.

How it works

  Stage 1 (already done by Google):
    General web text + books  --MLM-->  general-purpose BERT

  Stage 2 (domain-adaptive pretraining, this lesson):
    Your own unlabelled domain text  --same MLM objective-->
      BERT that now also understands your domain's vocabulary and patterns

  Stage 3 (fine-tuning, previous lesson):
    Your smaller labelled dataset  -->
      BERT specialised for your specific task

Stage 2 needs no labels at all, only text. That is why it can use far more domain data than a labelled fine-tuning set typically has available. It sits between general pretraining and task-specific fine-tuning, closing the vocabulary and style gap between the two.

Where you have already seen it

  • Legal AI tools. They continue pretraining a general encoder on large volumes of contracts and case law, before fine-tuning it for a specific legal task.
  • Biomedical NLP models like BioBERT and PubMedBERT, both continued-pretrained on medical literature before being fine-tuned for tasks like clinical entity extraction.
  • Financial services chatbots and document processors, continued-pretrained on regulatory filings and internal documentation before task-specific fine-tuning.

Remember this

  • Domain-adaptive pretraining continues BERT's original masked language modelling on your own unlabelled text.
  • It needs no labels, only domain text, which usually exists in much larger quantity than labelled examples.
  • It sits between general pretraining and task-specific fine-tuning, closing the vocabulary gap between them.

What to learn next

Developer — Code and libraries.

Continued pretraining uses the same AutoModelForMaskedLM class from the masked language modelling lesson, with a training loop instead of a single inference call. This example is a toy, five-sentence domain corpus — real domain-adaptive pretraining uses thousands to millions of unlabelled domain sentences.

Setup

bash
pip install transformers torch

Minimal runnable code

domain_pretrain.py
import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM, DataCollatorForLanguageModeling

tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("distilbert-base-uncased")

# A tiny slice of domain text: UPI/fintech support tickets. Real domain-adaptive
# pretraining uses thousands of such lines; this is a toy stand-in.
domain_text = [
    "The UPI transaction failed due to an NPCI timeout at the acquiring bank.",
    "Please check the beneficiary VPA before retrying the payment.",
    "The chargeback was raised because the merchant did not settle the transaction.",
    "Auto-debit mandates for SIPs sometimes fail when the UPI PIN is entered wrong.",
    "The refund was reversed to the source account after the RRN was verified.",
]

torch.manual_seed(0)
enc = tok(domain_text, truncation=True, padding=True, return_tensors="pt")
collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=True, mlm_probability=0.15)

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)
model.train()

for step in range(30):
    batch = collator([{k: v[i] for k, v in enc.items()} for i in range(len(domain_text))])
    outputs = model(**batch)
    outputs.loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if step % 10 == 0 or step == 29:
        print(f"step {step:2d}  loss {outputs.loss.item():.3f}")

# Check whether the model now expects domain-relevant words in a masked slot.
model.eval()
test = tok("The payment failed because the [MASK] was invalid.", return_tensors="pt")
with torch.no_grad():
    logits = model(**test).logits
mask_pos = (test["input_ids"][0] == tok.mask_token_id).nonzero(as_tuple=True)[0]
top3 = torch.topk(logits[0, mask_pos], 3).indices[0]
print("top predictions after training:", [tok.decode([t]) for t in top3])
Output
step  0  loss 3.217
step 10  loss 2.186
step 20  loss 1.299
step 29  loss 1.294
top predictions after training: ['transaction', 'payment', 'source']

With torch.manual_seed(0) set, this output is reproducible on the same library versions — a different transformers or PyTorch version may shift the exact numbers slightly, though the downward loss trend and the domain-relevant final predictions should hold.

Line by line

DataCollatorForLanguageModeling(mlm=True, mlm_probability=0.15) applies the same 80/10/10 masking recipe from the masked language modelling lesson, automatically, to each training batch — you do not need to reimplement masking by hand for continued pretraining.

The loss drops from 3.217 to 1.294 over 30 steps on five sentences. This is real learning, but on this little data it is also real overfitting — the model can start memorising these exact five sentences rather than learning general domain patterns. A production run uses vastly more data and many fewer passes over any single sentence.

The final prediction is "transaction", "payment", "source" — not the exact word "VPA" from the training text, but recognisably financial-transaction vocabulary, not the kind of word a general-purpose BERT would have guessed before this training. This is the honest, expected outcome of a toy-scale demonstration: real directional movement toward domain vocabulary, without claiming this five-sentence run produced a fully domain-adapted model.

Common mistakes

Treating five sentences and thirty steps as a real domain-adaptive pretraining run. This is a mechanics demonstration. Published domain-adaptive pretraining work (Gururangan et al., 2020, cited in the researcher block) uses corpora in the hundreds of thousands to millions of documents, trained for a small number of epochs over that much larger dataset — not many epochs over a handful of sentences.

Setting mlm_probability far from the standard 0.15 without a reason. Masking too little gives the model too weak a training signal per step; masking too much removes so much context that even the right answer becomes unguessable, which can destabilise training. 0.15 is a well-established, empirically-chosen default from BERT's original paper.

Skipping evaluation on general text after domain-adaptive pretraining. Continued training exclusively on narrow domain text risks eroding some of the model's general language ability — a real trade-off worth measuring, not assuming away, on a held-out general-text sample if the final model still needs to handle non-domain input.

Forgetting this step needs no labels, and reaching for expensive labelled data instead. The entire value of domain-adaptive pretraining is that it runs on plain, unlabelled domain text — support tickets, internal documents, whatever exists in volume — with no annotation effort required.

Try it yourself

Increase domain_text to fifteen or twenty sentences drawn from a domain you know well, and rerun. Check whether the top predictions for a domain-relevant masked slot become noticeably more specific and accurate as the effective corpus size grows, even at this still-tiny scale.

What to learn next

Researcher — Mathematics and papers.

Formal framing

Domain-adaptive pretraining (DAPT) continues the same objective used for general pretraining — masked language modelling, L_MLM as defined in Masked language modelling's researcher block — on an unlabelled domain corpus D_domain, initialising from general-pretrained weights theta_0 rather than random initialisation:

text
theta_DAPT = argmin over theta of  E_{x ~ D_domain} [ L_MLM(x; theta) ],   starting from theta = theta_0

This is distinguished from task-adaptive pretraining (TAPT), which continues pretraining on the unlabelled text of the fine-tuning task itself (not a broader domain corpus), and from fine-tuning, which trains against a labelled objective rather than MLM.

Empirical findings

Gururangan et al. (2020), Don't Stop Pretraining, systematically test DAPT and TAPT across four domains (biomedical, computer science, news, reviews) and eight tasks, finding:

  • DAPT improves downstream task performance in every domain tested, with larger gains when the target domain is more distant from the original pretraining corpus (general web text and books) — biomedical and computer-science text, both further from general web text than news, show the largest gains.
  • TAPT, despite training on much less text than DAPT (only the task's own unlabelled text), provides gains that are often comparable to, and sometimes exceed, DAPT alone.
  • DAPT followed by TAPT (both applied in sequence, before final labelled fine-tuning) gives the best results overall in their experiments, suggesting the two effects are at least partially complementary rather than redundant.

This directly informs practical sequencing: domain corpus first (if available in quantity), then the specific fine-tuning task's own unlabelled text if it exists separately from its labels, then labelled fine-tuning last.

Domain-specific pretrained models as an alternative

Rather than continuing pretraining from a general checkpoint, several widely-used domain models are pretrained from scratch (or from an early general checkpoint, continued for a very large number of steps) entirely on domain text: BioBERT (Lee et al., 2020) on biomedical literature, SciBERT (Beltagy et al., 2019) on scientific papers, PubMedBERT (Gu et al., 2021) on PubMedical abstracts and full text, using a domain-specific vocabulary trained from scratch rather than reusing general BERT's WordPiece vocabulary. Gu et al. report that a domain-specific vocabulary, not only domain-specific text, contributes measurably to downstream performance, since general BERT's vocabulary fragments many domain-specific terms into less useful subword pieces — a tokenizer-level effect, distinct from and additive to the training-data effect DAPT targets.

Catastrophic forgetting during continued pretraining

Continued pretraining on a narrow domain corpus risks degrading performance on general text, an instance of catastrophic forgetting analogous to the risk discussed in Fine-tuning BERT for classification's researcher block, but at the pretraining-objective level rather than the task-fine-tuning level. Gururangan et al. do not report this as a dominant concern for the domain-then-general-task pipelines they study, since downstream tasks in their evaluation are themselves domain-specific, but it becomes a real consideration for a model expected to serve both domain-specific and general-purpose text after continued pretraining.

Key references

  • Gururangan, S. et al. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. arXiv:2004.10964
  • Lee, J. et al. (2020). BioBERT: a pre-trained biomedical language representation model. arXiv:1901.08746
  • Beltagy, I., Lo, K. & Cohan, A. (2019). SciBERT: A Pretrained Language Model for Scientific Text. arXiv:1903.10676
  • Gu, Y. et al. (2021). Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. arXiv:2007.15779 — PubMedBERT.

Current state and open problems

Domain-adaptive pretraining remains a standard, low-cost technique whenever unlabelled domain text exists in quantity and general encoders underperform on domain-specific vocabulary — a common situation in specialised fields like law, medicine and finance. Whether to continue-pretrain a general checkpoint or train a domain-specific vocabulary from scratch is a real, unresolved cost-benefit decision: from-scratch domain vocabularies (BioBERT-style) measurably help but require much larger domain corpora and compute budgets than continued pretraining from a general checkpoint does. With large language models increasingly used zero-shot or few-shot for tasks that once required task-specific fine-tuned encoders, the relative value of DAPT versus prompting a much larger general model directly with domain context in-context is an actively debated, task-dependent trade-off rather than a settled answer either way.

What to learn next