Vision Models: Analyzing Images With GPT-5 and Claude

If you are building a feature that reads screenshots, scanned invoices, product photos, or chart images, vision models are now the fastest path from pixels to structured data. This guide is for backend and full-stack developers who already call an LLM API and want to add image input without guessing at the request format. You will learn how to send images to both GPT-5 and Claude, how to get JSON back instead of prose, how to control the token cost that images quietly add, and which failure modes break vision pipelines in production.

First, a naming correction worth making up front. There is no separate “GPT-5V” model. The V suffix belonged to the GPT-4V era, when vision was a distinct variant. Vision is now built into the main GPT-5 model, exactly as it is built into Claude. You do not opt into a vision model; you simply add an image block to a request you were already making.

What Are Vision Models (and What Changed)?

Vision models are large language models that accept images as input alongside text, encoding each image into tokens the model reasons over in the same context window as your prompt. Unlike older OCR tools, they do not just transcribe characters. They interpret layout, follow references between a chart and its legend, and answer questions about what the image means.

That distinction matters for architecture. Traditional document pipelines chained three services: OCR to get text, a layout parser to recover structure, then an LLM to interpret the result. Errors compounded at every hop, and a misread table header poisoned everything downstream. Modern vision models collapse that chain into one call. Consequently, the engineering problem shifts from stitching services together to controlling cost, validating output, and handling the images that models genuinely cannot read.

Both providers converged on similar mechanics. You attach an image as a content block inside a user message, mix it with text in the same message, and read the response as usual. However, the block shapes differ, the resolution limits differ, and the cost models differ in ways that matter at volume.

Prerequisites

Before starting, make sure you have the following in place:

1. Python 3.10 or later, with a virtual environment activated
2. Both SDKs installed: pip install anthropic openai pydantic
3. API keys exported as ANTHROPIC_API_KEY and OPENAI_API_KEY
4. A test image on disk — a screenshot or a scanned receipt works well
5. Familiarity with basic chat completion calls (if not, start with getting started with the Claude API or building apps with the OpenAI API)

Both APIs accept JPEG, PNG, GIF, and WebP. Notably, neither accepts PDFs as images — PDFs use a separate document block type, which is a common source of confusion when teams migrate a scanning pipeline.

Step 1: Send Your First Image to Claude

Claude takes images as an image content block inside the user message. The block sits alongside a text block in the same content array, and order matters: put the image first when the question refers to it, because the model reads content blocks sequentially.

import base64
import anthropic

client = anthropic.Anthropic()

with open("receipt.png", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": image_data,
                },
            },
            {
                "type": "text",
                "text": "List every line item on this receipt with its price.",
            },
        ],
    }],
)

for block in response.content:
    if block.type == "text":
        print(block.text)

Why this works the way it does: the media_type must match the actual file bytes, not the file extension. A .png file that was really a JPEG renamed by a user upload flow will fail validation, which is why production code should sniff the type rather than trust the filename.

Also note the loop over response.content. On current Claude models, thinking is enabled by default, so the response can contain thinking blocks before the text block. Code that reads response.content[0].text unconditionally will break the moment the model thinks first. Always filter by block type.

One more detail worth internalizing: max_tokens caps thinking plus visible output together. A vision request that worked with max_tokens=1024 on an older model may truncate mid-answer now, because reasoning consumes part of that budget. Give image-heavy calls room.

Step 2: Send the Same Image to GPT-5

GPT-5 uses the Responses API, where images arrive as input_image parts and text as input_text parts. The image is passed as a data URI rather than a separate base64 field, which is the single biggest shape difference between the two providers.

import base64
from openai import OpenAI

client = OpenAI()

with open("receipt.png", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.responses.create(
    model="gpt-5",
    input=[{
        "role": "user",
        "content": [
            {
                "type": "input_text",
                "text": "List every line item on this receipt with its price.",
            },
            {
                "type": "input_image",
                "image_url": f"data:image/png;base64,{image_data}",
                "detail": "high",
            },
        ],
    }],
)

print(response.output_text)

Why the detail parameter matters: it is the main cost lever on the OpenAI side. Setting detail: "low" processes the image at a fixed small token cost regardless of dimensions, which suits classification and yes-or-no questions. Setting detail: "high" tiles the image and costs proportionally more, which you need for dense text, small labels, and fine chart gridlines. Leaving it at "auto" lets the model choose, which is convenient in development and unpredictable in a budget forecast.

Because the two request shapes differ this much, teams supporting both providers usually write a thin adapter rather than branching inline. If you are running several providers behind one interface, a gateway handles this normalization for you.

Step 3: Handle Image Sources Beyond Base64

Base64 is fine for a demo and wasteful in production. Encoding inflates payload size by roughly a third, and you re-upload the same bytes on every request. Three better options exist depending on your access pattern.

Public URLs work when the image already lives somewhere the provider can fetch, such as a CDN or a signed S3 link. Claude accepts a url source type directly:

{
    "type": "image",
    "source": {"type": "url", "url": "https://cdn.example.com/receipt.png"},
}

OpenAI accepts the URL in the same image_url field you would use for a data URI, so the switch is a one-line change. Keep in mind that the provider fetches the URL server-side, so it must be reachable from the public internet and must not require your application’s auth headers.

The Files API suits the case where you ask several questions about one image. Upload once, then reference the returned ID across many requests, which avoids re-transferring the bytes each time:

uploaded = client.beta.files.upload(
    file=("receipt.png", open("receipt.png", "rb"), "image/png"),
)

response = client.beta.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    betas=["files-api-2025-04-14"],
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "file", "file_id": uploaded.id}},
            {"type": "text", "text": "What is the total, including tax?"},
        ],
    }],
)

Prompt caching helps when the image is stable and the questions vary. Because caching is a prefix match, place the image block first and the varying question last, then set a cache breakpoint after the image. Subsequent questions about the same image then read the encoded image from cache at a fraction of the input price. This pattern pays off quickly in document-review interfaces where a user asks five or six follow-up questions about one page.

Step 4: Get Structured Data Instead of Prose

Prose output is a trap in vision pipelines. The model describes the receipt beautifully, and then your parser has to guess whether “Total: $47.30” appeared before or after the tax line. Structured outputs remove the guesswork by constraining the response to a schema you define.

On the Claude side, define a Pydantic model and use messages.parse:

from pydantic import BaseModel
import anthropic

class LineItem(BaseModel):
    description: str
    quantity: int
    unit_price: float

class Receipt(BaseModel):
    merchant: str
    purchase_date: str
    line_items: list[LineItem]
    subtotal: float
    tax: float
    total: float

client = anthropic.Anthropic()

response = client.messages.parse(
    model="claude-opus-5",
    max_tokens=4096,
    output_format=Receipt,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64",
                                         "media_type": "image/png",
                                         "data": image_data}},
            {"type": "text", "text": "Extract this receipt."},
        ],
    }],
)

receipt: Receipt = response.parsed_output
print(receipt.total, len(receipt.line_items))

The OpenAI equivalent uses responses.parse with text_format, returning the validated object on response.output_parsed. Either way, you get a typed object instead of a string, and validation failures surface immediately rather than three functions downstream.

Why this is the single highest-value step: a schema does double duty. It shapes the output, and it also communicates intent to the model. A field named purchase_date with a description saying “ISO 8601 date the transaction occurred, not the print date” measurably reduces the wrong-date failure that plagues receipt extraction. Treat field descriptions as prompt real estate, not documentation.

Do add a sanity check that the schema cannot express. Verify that subtotal + tax equals total within a cent, and flag the record for review when it does not. Vision models read numbers well but occasionally transpose digits on low-contrast thermal paper, and arithmetic consistency catches most of those cases cheaply. For deeper patterns here, see structured LLM outputs with Instructor and Pydantic.

Step 5: Control Resolution and Token Cost

Images are expensive in a way that surprises teams on their first invoice. A single high-resolution screenshot can consume more input tokens than several pages of text, and a pipeline processing thousands of images per day feels that immediately.

Current Claude Opus models accept images up to 2576 pixels on the long edge, consuming up to roughly 4,784 tokens for a full-resolution image. Earlier models capped at 1568 pixels and around 1,600 tokens, so the newer high-resolution tier costs meaningfully more per image in exchange for reading small text and dense diagrams that previously failed. Measure before assuming you need it:

count = client.messages.count_tokens(
    model="claude-opus-5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64",
                                         "media_type": "image/png",
                                         "data": image_data}},
            {"type": "text", "text": "Extract this receipt."},
        ],
    }],
)
print(count.input_tokens)

Run that against a representative sample of your real images rather than one test file. Screenshots, photos, and scans tokenize very differently, and an average drawn from a single example will mislead your capacity planning.

Three levers reduce cost without hurting accuracy. First, downsample client-side when the fidelity is not needed — a 4000-pixel phone photo of a printed page rarely reads better than the same image at 1500 pixels. Second, crop to the region of interest when you know where the data lives; a cropped invoice header beats a full-page scan on both cost and accuracy. Third, use the low-detail setting on OpenAI for classification-style questions where you only need to know what kind of document this is.

For a broader treatment of measuring and budgeting spend across an LLM application, see token counting and budget management for LLM apps.

GPT-5 vs Claude for Vision: Practical Differences

Dimension	GPT-5	Claude (Opus 5)
Image block	input_image with data URI or URL	image block with base64, url, or file source
Resolution control	detail parameter: low, high, auto	Automatic; downsample client-side to control cost
Max long edge	Tiled above a base size	2576 pixels
Reusable upload	Files API	Files API (beta header required)
Structured output	responses.parse with text_format	messages.parse with output_format
Multiple images	Multiple input_image parts per message	Multiple image blocks per message
Coordinate output	Approximate	Maps 1:1 to pixels on high-resolution tier

The coordinate row deserves emphasis if you are building anything that draws boxes on an image. Because the current Claude high-resolution tier returns coordinates that map directly to actual image pixels, you can render a bounding box without applying a scale factor. Pipelines written against older models often carry scale-factor math that is now wrong; delete it rather than adjusting it.

Beyond that, treat capability claims skeptically and test on your own data. Both models handle clean documents well. Divergence appears on the hard cases: rotated scans, handwriting, low-contrast thermal receipts, and screenshots with heavy anti-aliasing. Build a fixture set of twenty genuinely difficult images from your domain and evaluate against that, because published benchmarks rarely resemble the photos your users actually upload.

If you are also evaluating a third option, the Gemini API for multimodal vision and video covers a provider with a different pricing shape for high-volume image work.

Real-World Scenario: A Claims-Intake Backlog

Consider a small insurance-tech team with a two-person backend group, replacing a manual claims-intake step over the course of a quarter. Adjusters were receiving photos of damage estimates as phone snapshots, then retyping fourteen fields into an internal system. Volume sat in the low thousands per week, and the retyping was both the bottleneck and the main source of data-entry errors.

The first version worked in an afternoon and looked like a success: one vision call per image, a Pydantic schema with fourteen fields, and a queue worker. Accuracy on the team’s own test images was high enough to demo. Then production traffic arrived, and three problems surfaced that the test set had not.

Where the First Version Broke

Rotation was the first. Adjusters photographed documents sideways more often than anyone expected, and while the model often coped, it silently degraded on the sideways images rather than failing loudly. The fix was cheap: detect orientation from EXIF metadata and rotate before sending. That is ordinary image preprocessing, not model work, and it is the sort of step teams skip because the model “usually handles it.”

Cost was the second. Phone photos arrive at full sensor resolution, and the team was sending every one at full size. Downsampling to roughly 1500 pixels on the long edge before encoding cut image token consumption substantially with no measurable accuracy loss on their documents, because the text on a printed estimate was large relative to the frame. The saving came from doing nothing clever at all.

Why Confident Errors Were the Real Problem

The third issue proved hardest to solve. A confidently extracted claim number that was off by one digit is worse than a failed extraction, because it flows into the system unchallenged. The team added two guards: an arithmetic consistency check on the monetary fields, and a rule that routes any claim above a value threshold to human review regardless of apparent confidence. The trade-off is explicit — they accepted a review queue on high-value claims in exchange for never auto-approving a misread figure.

The pattern generalizes. Vision models remove the transcription work, but they do not remove the need for validation, and the images your users send are consistently worse than the ones you tested with.

Common Errors and How to Fix Them

Could not process image on a valid-looking file. Almost always a media_type mismatch, a corrupted upload, or an unsupported format such as HEIC arriving from an iPhone. Sniff the actual bytes with Pillow or python-magic instead of trusting the extension, and convert HEIC to JPEG at the ingest boundary.

Request too large. Base64 encoding inflates payload size by about a third, and several images in one request add up fast against request size limits. Switch to URL or Files API references, or downsample before encoding.

Truncated output mid-sentence. Check stop_reason — a value of max_tokens means you ran out of budget. On models where thinking is on by default, reasoning shares that budget with visible output, so raise max_tokens rather than shortening the prompt.

Empty or wrong field on a specific document type. Before adding prompt instructions, look at the image at the resolution the model actually received. If you downsampled to 800 pixels and the field is 6-point print, the information is simply not there. No prompt recovers deleted pixels.

Hallucinated values on blank or ambiguous regions. Models tend to fill in plausible-looking data rather than reporting absence. Make optional fields genuinely optional in your schema and say so explicitly: “return null when the field is not present in the image, and do not infer it from context.”

Rate limits under burst load. Image requests consume input tokens quickly, so a batch job hits token-per-minute limits sooner than a text workload at the same request rate. Queue the work and apply backoff rather than firing the whole batch at once.

When to Use Vision Models

• Documents with layout that carries meaning, such as tables, forms, and invoices where position determines what a value refers to
• Screenshots for support tooling, QA triage, or automated bug-report enrichment
• Charts and diagrams where the question is about the trend rather than the pixel values
• Mixed-content pages that combine printed text, handwriting, stamps, and logos
• Any workflow where a human currently looks at an image and types what they see

When NOT to Use Vision Models

• High-volume plain-text OCR at scale, where a dedicated OCR engine costs far less per page for the same result
• Precise measurement tasks that need exact pixel coordinates or sub-pixel accuracy, which belong to classical computer vision
• Real-time video frame analysis, where per-frame latency and cost make the approach impractical
• Cases where you already have the underlying structured data and are only reaching for the image out of habit
• Regulated decisions that must be auditable to a deterministic rule, since a model’s reasoning is not a reproducible audit trail

Common Mistakes with Vision Models

• Trusting the output without validation, which turns a confident misread into corrupted downstream data
• Sending images at maximum resolution by default, inflating cost with no accuracy benefit on documents with large print
• Skipping orientation and format normalization at ingest, then blaming the model for degraded results on sideways photos
• Asking for prose and regex-parsing the answer instead of defining a schema
• Testing only on clean images the team curated, which hides the failure modes that dominate real traffic
• Reading response.content[0] on Claude without filtering by block type, which breaks when a thinking block comes first
• Batching many images into one request to save calls, which makes it impossible to tell which image produced which error

Conclusion

Vision models turn a multi-service OCR pipeline into a single API call, and the mechanics are genuinely simple: attach an image block, define a schema, read a typed object back. The engineering work is not in the request — it is in normalizing input at ingest, measuring token cost against real images rather than test files, and validating output that arrives confidently wrong just often enough to matter.

Start with one narrow document type, build a fixture set of twenty difficult real examples, and measure accuracy and token cost against that before expanding scope. Next, add structured outputs with the OpenAI API to lock down the response shape, and explore Claude tool use when your image workflow needs to trigger actions rather than just return data.