VLM Fine-Tuning: How to Train Vision-Language Models

Vision-language models (VLMs) are multimodal networks that read images and text together. A vision encoder — typically CLIP, SigLIP, or a sibling contrastive vision model — feeds a projection layer that maps visual features into a large language model’s embedding space.

That assembly lets a VLM answer questions about images, describe what it sees, pull text out of documents, read charts, and reason over scenes. Fine-tuning specialises those abilities for your visual domain and your output format.

The major open families are on the platform: InternVL 2 and 2.5 (the strongest open VLMs for OCR and visual reasoning), Qwen-VL (Alibaba’s multilingual multimodal line), LLaVA (the visual instruction-tuning architecture from Microsoft Research), DeepSeek-VL (efficient multimodal), and MiniCPM-V (compact, for constrained deployments).

Strengths split cleanly: InternVL for document understanding and complex visual reasoning, Qwen-VL for multilingual visual tasks, LLaVA for visual instruction following. Size and deployment constraints decide the rest.

Images paired with text annotations. For SFT, each example carries an image (path or URL), an instruction or question about it, and the expected response. Preference alignment adds chosen and rejected responses for the same image-question pair.

Images arrive as file paths (uploaded with the dataset), URLs, or base64 strings in the JSONL. Loading, resizing to the model’s native resolution, and encoding are handled by the platform — and image accessibility and format are validated at upload.

The same set as text-only models: SFT, LoRA, QLoRA, and full fine-tuning. The adapter lands on the language-model side, while the vision encoder can stay frozen or be trained, per your configuration.

Freeze it for most tasks — the pre-trained visual features are the expensive part of what you bought. Unfreeze only when your images live far outside natural-image territory: medical imaging, satellite imagery, specialised scientific visualisations.

Teams fine-tune VLMs for a spread of visual work:

• Document understanding: structured extraction from invoices, receipts, forms, contracts.
• Visual question answering: questions over product photos, property listings, technical diagrams.
• OCR and extraction: domain-specific text pulled from images, formatted your way.
• Medical imaging: X-rays, pathology slides, dermatological images (with the data handling those demand).
• Quality inspection: defect spotting in manufacturing imagery.
• Chart analysis: reading business charts and graphs.

Pick a VLM family in the model picker (filtered by the VLM tag) and upload the image-text dataset. The configurator handles VLM-specific settings itself: image resolution, vision-encoder treatment, multi-image support where it applies.

The wizard adapts to VLM configuration:

VLM runs want more VRAM than text-only ones — the vision encoder and the image token sequences both spend it. The platform shows accurate VRAM estimates for VLM jobs and recommends the GPU tier; QLoRA earns its keep here, letting larger VLMs train on a single GPU.

Loss curves and metrics stream live:

When the information already exists as text. If the “visual” content in the pipeline is available structured — a CSV behind the chart, HTML behind the page, OCR text behind the scan — a text-only model reads it cheaper, trains simpler, deploys lighter. Train a VLM when the pixels carry information the text does not.

When the dataset is smaller than the task. Vision-language training must learn the visual pattern and its text association; a few hundred pairs rarely cover both. Where the dataset cannot grow, a strong base VLM with careful prompting is the honest alternative.

And when the image itself is incidental. If the same content pasted as text loses nothing, the VLM is solving an input-format problem — an OCR or extraction step in front of a text model is usually the simpler pipeline to own.

Image quality and consistency come first. Training images should be clear, correctly oriented, and representative of production: blurry, low-resolution, or rotated inputs inject noise straight into the model. Document work should include the same layouts, fonts, and quality levels the live system will see.

Freeze the vision encoder for most runs. It was pre-trained on a large, diverse image corpus and its general features are the asset; unfreezing risks overwriting them with narrow patterns from your smaller set. Unfreeze only for far-from-natural images — medical scans, satellite imagery, specialised visualisations — where the pre-trained features may not transfer.

Budget more data than text-only work: the model is learning visual patterns and their text associations together. Aim for at least 5,000 image-text pairs for LoRA, 20,000-plus for full fine-tuning; below that, aggressive augmentation (flips, rotation, colour jitter) stretches a small set.

Make QLoRA the default adapter. The vision encoder’s memory overhead is real, and quantising the language-model side frees room for the visual pipeline — typically one GPU tier larger than standard LoRA would allow on the same hardware.

Start with the dataset: image (path, URL, or base64), instruction or question, expected response — formatted as JSONL and uploaded through the Datasets page. Image references and text fields are validated at upload.

Create a new training run and filter the model list to VLM architectures. InternVL for document understanding, Qwen-VL for multilingual visual work, LLaVA for visual instruction following, MiniCPM-V for constrained deployments — and QLoRA as the adapter for the best memory-quality balance.

Open with the platform defaults, launch, and watch the loss curves. VLM training often starts with a steeper loss drop than text-only work — the model is learning to associate visual features with text. When it finishes, generate answers for test images and measure against the base VLM.