inference.yaml: Inference · Docs

The full file

The yaml is the platform's side of the deal: it describes the hardware your job needs so the scheduler can pick matching workers. Runtime behavior (which model, context, batching) stays in serve_module.py. A typical single-GPU job:

inference.yaml

engine: vllm_openai

model:
  module: "serve_module"        # scripts/serve_module.py — must export `app`

vllm:                           # must mirror engine_kwargs in the module
  tensor_parallel_size: 1
  pipeline_parallel_size: 1

cluster:
  num_workers: 1                # worker machines to reserve
  device: "gpu"
  gpus_per_worker: 1
  cpus_per_worker: 8
  min_vram_gb_per_worker: 22    # free VRAM each selected worker must have
  # Advisory — dashboard labels only:
  mode: "data_parallel"
  max_ongoing_requests: 32

# No `requirements:` — the prebuilt image owns the vLLM stack.

Field reference

Field	Required?	What it does
engine	yes	must be `vllm_openai`. Omitting it selects the custom /predict contract instead.
model.module	yes	the Python module under `scripts/` (no .py) that exports `app`. Checked against your uploaded files at submit.
vllm.tensor_parallel_size	defaults to 1	GPUs jointly holding one copy of the weights. All on one worker.
vllm.pipeline_parallel_size	defaults to 1	layer stages, each on its own worker. One replica consumes this many workers.
cluster.num_workers	yes	worker machines to reserve. With pipeline parallelism it must be a whole multiple of pipeline_parallel_size.
cluster.device	yes	`"gpu"` (or `"auto"`). vLLM needs CUDA; `"cpu"` is rejected.
cluster.gpus_per_worker	defaults to 1	GPUs each worker must expose. Must be ≥ tensor_parallel_size.
cluster.cpus_per_worker	defaults to 8	CPU cores each worker must expose.
cluster.min_vram_gb_per_worker	defaults to 8	free VRAM each selected worker must contribute. The single most important sizing field; see below.
cluster.mode	advisory	a dashboard label (data_parallel / tensor_parallel / pipeline_parallel). Never forwarded to vLLM.
cluster.max_ongoing_requests	advisory	dashboard accounting only. The real concurrency cap lives in the module's deployment_config.
requirements	leave empty	the prebuilt image ships Ray + vLLM + torch as one tested set. Adding ray or vllm lines is rejected at submit; only genuine extras belong here.

note

model.class_name and model.init_args belong to the custom /predict contract. In vLLM mode they are ignored, so leave them out.

Sizing min_vram_gb_per_worker

The scheduler filters workers by this number, so set it from what your model actually needs: weights + KV cache + about 1 GB of overhead, rounded up. Weights for a 4-bit AWQ checkpoint are roughly 0.55 bytes × parameter count (an 8B model ≈ 5.5 GB); fp16 is 2 bytes × parameters. The examples spell the arithmetic out in a comment next to the field, per job.

warn

Set it too low and the scheduler places you on a card the model can't boot on. Too high and no worker qualifies. When adapting an example to a different model, this field and max_model_len in the module are the two numbers to re-derive. The napkin math lives in LLMConfig options.

With tensor_parallel_size above 1 the check is per GPU: every card on the worker needs min_vram_gb_per_worker ÷ gpus_per_worker free, because vLLM shards evenly and the smallest card gates the whole group.

The sync rule

The vllm: block exists because the scheduler reads only the yaml. It cannot execute your Python, yet the parallel sizes decide which hardware qualifies: TP=2 needs a two-GPU worker, PP=2 needs two workers per replica. So the two values are declared here and enforced in the module, and submit fails fast if they disagree:

vllm.tensor_parallel_size   == engine_kwargs.tensor_parallel_size
vllm.pipeline_parallel_size == engine_kwargs.pipeline_parallel_size

Both are whole numbers ≥ 1; they count physical GPUs. Most jobs run 1/1 and can omit the block entirely.

Previousserve_module.py Next LLMConfig options