Continued Pre-Training: Domain Adaptation for Large Language Models

Continued pre-training (CPT) extends a pre-trained model by training it further on a large corpus of domain-specific text, using the same next-token prediction objective as the original pre-training. Where SFT teaches instruction-following with structured input-output pairs, CPT injects raw knowledge — large volumes of unstructured domain text absorbed directly.

Picture it as lengthening the model's pre-training phase with your own data. It learns the domain's vocabulary, the relationships between its concepts, and the texture of its writing. The result is a stronger foundation in the domain, on which subsequent SFT and alignment stages build more effectively.

Reach for CPT when the domain knowledge your model needs was thin in the original pre-training data. General-purpose LLMs are trained chiefly on web text, which leaves shallow coverage of clinical medicine, patent law, quantitative finance, or proprietary internal documentation.

If the model already grasps the domain and merely needs to follow particular instructions or emit particular formats, SFT alone is enough. But where fundamental knowledge is missing — confused medical terminology, misapplied legal precedent, factually wrong domain content — CPT is the opening move.

Four domains account for most CPT work:

• Medical: clinical notes, medical literature, drug databases, and treatment guidelines — building models fluent in medical terminology and clinical reasoning patterns.
• Legal: case law, statutes, contracts, and regulatory filings — models that can analyse legal documents and reason about legal concepts.
• Financial: earnings reports, SEC filings, market analysis, and financial news — models that understand instruments, valuation methods, and market dynamics.
• Code: specialised codebases, internal libraries, and documentation — models that know your stack and its conventions.

CPT datasets are raw text files — no instruction-response formatting required. The text should be clean, well-structured, and representative of the knowledge you want absorbed. Typical sources: research papers, technical documentation, internal wikis, textbooks, standards documents, and curated web crawls filtered for domain relevance.

Quality outweighs quantity here too. A curated 1GB set of high-grade domain text routinely outperforms 10GB of noisy scrapes. Deduplicate, strip boilerplate (headers, footers, navigation), and confirm clean decoding — no encoding artifacts, no broken characters.

Pick Continued Pre-Training as the method in the configurator. Upload the domain text as a plain text or JSONL file (one document per line). The hyperparameters that matter: learning rate (typically 1e-5 to 5e-5 for CPT — below SFT's), context length, and epochs.

The wizard carries the CPT-specific settings:

CPT jobs run longer than SFT jobs, given the corpora they process. Watch perplexity on a held-out validation set in the dashboard: falling perplexity means the model is absorbing the domain's language patterns. Stop when it plateaus — further training only courts overfitting.

The recommended pipeline: base model, then CPT over the domain text, then SFT over instruction-response pairs, then optional alignment (DPO, SimPO, or a sibling) over preferences. Each stage stands on the last.

After CPT the model knows the domain but does not yet take instructions; SFT teaches it to answer queries with structured output; alignment then tunes quality to human preference. On Vidman AI the stages chain as separate jobs — each run's checkpoint becomes the starting model for the next.

When the gap is behaviour, not knowledge. CPT teaches the model your domain's text — vocabulary, facts, conventions — but not how to follow your instructions. Failures of format, tone, or task discipline belong to SFT, and more pre-training documents will not fix them.

When the corpus is small. Continued pre-training is the hungriest method in the catalog; a few megabytes of domain text is not a pre-training dataset, it is context. Small, high-value knowledge bases are better served by retrieval — where an update is an edited document, not a training run.

And when the domain knowledge moves weekly. Pre-training bakes knowledge into weights, and weights freeze at the training date. Fast-moving facts — prices, inventories, policies — belong in a retrieval layer, where an update is an edit, not a retrain.

CPT outcomes hinge on preparation and pacing. First: curate the corpus with intent. The model absorbs patterns from every document in the set, so admit only high-quality, well-written domain text — strip headers, footers, navigation menus, copyright notices, and duplicated passages; render tables as readable text, not raw HTML or fragmented columns.

Second: keep the learning rate below SFT's. CPT reaches deeper into the model than instruction tuning, and aggressive rates overwrite useful pre-trained knowledge. Open at 1e-5, dropping to 5e-6 if general-purpose benchmarks degrade after CPT. The aim is added domain knowledge, not lost general capability.

Third: validate the CPT model before SFT. Generate answers to domain questions and compare against the base — the CPT model should show richer domain vocabulary, more accurate domain reasoning, surer command of the concepts. No visible improvement suggests a dataset too small, too noisy, or too close to what the model already knew.

Fourth: never skip SFT after CPT. The CPT model holds domain knowledge but has not been taught to take instructions; SFT after CPT beats either stage alone. On Vidman AI the chain is literal — the CPT checkpoint becomes the starting model for the SFT run.

CPT begins with the corpus. Collect high-quality text from the domain's sources — research papers, technical documentation, internal wikis, textbooks, curated crawls — then clean it: boilerplate out, duplicates out, encoding verified. Shape it as a plain text or JSONL file, one document per line.

Upload through the Datasets page; for CPT the platform takes plain text and validates encoding and formatting. Create a new run, select Continued Pre-Training, and choose the base model. Set learning rate 1e-5, match context length to the documents (2048 or 4096 tokens typical), and set epochs by corpus size — 1 to 2 for large corpora, 3 to 5 for smaller ones.

Launch and watch the perplexity metric. It should fall steadily as the model absorbs the domain's language; when it plateaus, the corpus has given what it has and further training wastes compute — stop there to dodge overfitting.

With CPT complete, feed the checkpoint into an SFT run. The two-stage pipeline — CPT, then SFT — yields models that both understand the domain deeply and follow instructions reliably.