Feed Claude a Screenshot: How to Use the Vision API for Visual Automation

TL;DR: Pass any screenshot or local image to Claude as a base64-encoded content block — the same messages.create call you already use for text, just with a different content type.

Most AI workflows stop at text. But Claude Sonnet 5 and Opus 4.8 accept images natively in the same API your text calls use — no separate endpoint, no special SDK. That makes visual automation (dashboard scraping, UI testing, document OCR, diagram extraction) a drop-in addition to any existing pipeline.

The pattern

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

def ask_about_image(image_path: str, prompt: str) -> str:
    image_bytes = Path(image_path).read_bytes()
    image_data = base64.standard_b64encode(image_bytes).decode("utf-8")

    ext = Path(image_path).suffix.lower()
    media_type = {
        ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
        ".png": "image/png",
        ".gif": "image/gif",
        ".webp": "image/webp",
    }.get(ext, "image/png")

    message = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": media_type,
                        "data": image_data,
                    },
                },
                {"type": "text", "text": prompt},
            ],
        }],
    )
    return message.content[0].text

# Parse a monitoring dashboard screenshot
print(ask_about_image(
    "screenshots/dashboard.png",
    "What is the current p99 latency shown? Return only the value and unit."
))

Or use a URL directly

For public images (e.g., a CI badge, a chart hosted on S3):

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=512,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "url", "url": "https://your-bucket.s3.amazonaws.com/chart.png"}},
        {"type": "text", "text": "List the five highest bars in this chart."},
    ]}],
)

Practical use cases

• UI test assertions: take a Playwright screenshot, ask "does this page show a success banner?" — no selector brittleness
• Dashboard parsing: extract metric values from monitoring screenshots without brittle HTML scraping
• Document OCR: rasterize a PDF page to PNG, extract tables, form fields, or signatures
• Diagram review: pass architecture diagrams to Claude in a code-review pipeline, ask it to flag missing error paths

Cost and limits

One full 1920×1080 PNG screenshot costs roughly 2,000–5,000 input tokens depending on image complexity. At Sonnet 5 pricing, that's a few tenths of a cent per screenshot. Max image size is 5 MB per image; you can send up to 20 images per request. For repeated analysis of the same document, use the Files API — upload once, reference by file_id.

Sources: Vision — Claude Platform Docs · Claude Vision API Image Analysis Guide — AI for Anything