Qwen2.5-VL-32B vision · Docs

What it is

Folder: Qwen2.5-VL-32B-AWQ_vision_tensor_parallel_min_36gb_vram. A vision-language model for hard documents: degraded scans, handwriting, dense tables — the inputs where smaller vision models start guessing. The workhorse use case is extraction: send a page image with a prompt, get structured JSON back. It is a general tool; invoices, lab reports, and field notes are all the same problem to it.

The model is Qwen2.5-VL-32B-Instruct in 4-bit AWQ (~19 GB of weights, Apache-2.0), served in plain mode: one worker, one GPU, tensor_parallel_size: 1. Clients use the standard OpenAI vision protocol, so any OpenAI client that can send image content parts works unmodified.

The config

inference.yaml

engine: vllm_openai

model:
  module: "serve_module"

vllm:
  tensor_parallel_size: 1
  pipeline_parallel_size: 1

cluster:
  num_workers: 1
  device: "gpu"
  gpus_per_worker: 1
  cpus_per_worker: 8
  min_vram_gb_per_worker: 36   # ~19 GB AWQ weights + KV pool + overhead
  mode: "data_parallel"
  max_ongoing_requests: 16

scripts/serve_module.py (core)

llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="vision",
        model_source="Qwen/Qwen2.5-VL-32B-Instruct-AWQ",
    ),
    accelerator_type=None,
    deployment_config=dict(
        autoscaling_config=dict(min_replicas=1, max_replicas=1),
        max_ongoing_requests=16,
    ),
    engine_kwargs=dict(
        tensor_parallel_size=1,
        pipeline_parallel_size=1,
        max_model_len=32768,
        max_num_seqs=8,
        kv_cache_dtype="fp8",
        enable_prefix_caching=True,
        enable_chunked_prefill=True,
        gpu_memory_utilization=0.90,
        dtype="auto",
        limit_mm_per_prompt={"image": 6},   # pages per request
    ),
)

app = build_openai_app({"llm_configs": [llm_config]})

note

limit_mm_per_prompt caps images per request at 6. With the 32k context and fp8 KV there is headroom for multi-page batches; raise it if your documents run longer.

Submit and call

Shell

cd Qwen2.5-VL-32B-AWQ_vision_tensor_parallel_min_36gb_vram
cp .env.example .env       # RESON_API_KEY + S3 keys
python submit.py           # prints endpoint + predict key
python predict.py                                  # sample invoice → JSON
python predict.py ./hard_scan.png "Transcribe the handwritten notes"

Images travel as OpenAI content parts: a data URL for local files, or a plain URL the runtime can fetch.

client.py

import base64
from openai import OpenAI

client = OpenAI(
    base_url=f"{ENDPOINT}/v1",
    api_key=KEY,
    default_headers={"X-API-Key": KEY},
)

b64 = base64.b64encode(open("invoice.png", "rb").read()).decode()
resp = client.chat.completions.create(
    model="vision",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url",
             "image_url": {"url": f"data:image/png;base64,{b64}"}},
            {"type": "text",
             "text": "Extract vendor, invoice_number, date, line_items and total. "
                     "Reply with JSON only."},
        ],
    }],
)
print(resp.choices[0].message.content)

Hardware

VRAM min → rec	CPU	RAM	Free disk
36 → 48 GB	8 cores	96 GB	~46 GB (image + ~19 GB checkpoint × download headroom)

In plain mode this wants one big card: an L40S or A40 (48 GB) runs it with room to spare, and an A100-80G fits even the fp16 checkpoint. At 32B parameters, AWQ is the right trade: the larger model more than compensates the 4-bit loss that hurts smaller models on fine print. Tier guidance: Choosing hardware.

PreviousQwen3-8B guarded RAG chat Next How it works