Your LLM Is Blind By Default: Fine-Tune a Vision Model on a Free Colab GPU With SmolVLM + QLoRA

SmolVLM + QLoRA lets you fine-tune a vision-language model on a free Colab T4 to analyze your domain images — receipts, charts, screenshots, PDFs — without a proprietary vision API.

What you'll be able to do after this:

• Fine-tune a compact VLM (SmolVLM-500M or 2B) to answer questions about images in your domain using only a consumer GPU or free Colab T4
• Structure image+text conversation datasets the way VLMs expect — a different format than text-only SFT, but straightforward once you see it
• Evaluate visual accuracy and serve the result through the same vLLM or Ollama stack you already run for text models

Walk-through

Install:

pip install transformers trl datasets pillow torch

Dataset format — each example is a conversation where user messages include both an image reference and a text question:

# Dataset row structure
{
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "image"},   # the image object lives in "images" list
                {"type": "text", "text": "What does this chart show?"}
            ]
        },
        {
            "role": "assistant",
            "content": "Q3 revenue grew 40% YoY driven by enterprise deals..."
        }
    ],
    "images": [pil_image_object]   # PIL Image or path
}

LoRA + SFTTrainer:

from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
from transformers import AutoProcessor, AutoModelForVision2Seq

model = AutoModelForVision2Seq.from_pretrained(
    "HuggingFaceTB/SmolVLM-500M-Instruct",
    load_in_4bit=True,    # fits on a T4 with 15GB VRAM
)
processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM-500M-Instruct")

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
)

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(output_dir="smolvlm-finetuned", num_train_epochs=3),
    train_dataset=dataset,
    peft_config=lora_config,
    processing_class=processor,
)
trainer.train()

For larger VLMs on the same free Colab hardware, Unsloth's Qwen3-VL (8B) notebooks train 1.7× faster with 60% less VRAM — the free Colab T4 handles it comfortably with 4-bit quantization. The key insight: VLM fine-tuning is structurally identical to text SFT. You're adding image tokens to the user message; the rest of your pipeline transfers.

Sources: Fine-tuning SmolVLM on a Consumer GPU — HuggingFace Cookbook · Fine-Tuning VLM with TRL (Qwen2-VL-7B) — HuggingFace Cookbook · Vision Fine-tuning — Unsloth Docs · Qwen3-VL Run and Fine-Tune — Unsloth