Sign inStart now ›
Inference /Calling your model

Calling your model

One endpoint, one key: OpenAI calls for vLLM runtimes, bytes or JSON for custom ones, plus limits, errors and callbacks.

What you need

Where to get itNotes
Endpoint URLThe runtime’s page, or job.endpoint after rt_submit_inference()Generated per runtime. Read it; don’t build it from a pattern
Predict API keyShown once at submit: the wizard’s last panel, or job.api_keyRotate API key on the runtime’s Overview tab mints a new one; the old key stops working within seconds
HeaderX-API-Key: <key>On every call except the health checks

Every request enters through the job’s central server. It checks the key, so a missing or wrong key gets 401 before any GPU sees the request, then routes the call to a worker running your model.

RuntimeCallHealth checks, no key
vLLM<endpoint>/v1/…, the OpenAI protocolGET /health
CustomPOST <endpoint>, which ends in /predictGET /health, answered by your health() if you define one

vLLM runtimes: the OpenAI protocol

Point any OpenAI client at <endpoint>/v1. The model field is the model_id from your serve_module.py.

client.py
from openai import OpenAI

client = OpenAI(
    base_url=f"{ENDPOINT}/v1",
    api_key=PREDICT_KEY,                          # the client requires a value
    default_headers={"X-API-Key": PREDICT_KEY},   # the platform reads this one
)

reply = client.chat.completions.create(
    model="assistant",
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
    max_tokens=256,
)
print(reply.choices[0].message.content)

for chunk in client.chat.completions.create(model="assistant", messages=[...], stream=True):
    print(chunk.choices[0].delta.content or "", end="")
Shell
curl "$ENDPOINT/v1/chat/completions" \
  -H "X-API-Key: $PREDICT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "assistant", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 64}'
RouteFor
POST /v1/chat/completionsChat and vision-language models
POST /v1/completionsPlain text completion
POST /v1/embeddingsEmbedding models
POST /v1/scoreRerankers (see the RAG example)
  • One endpoint can serve several models: each LLMConfig gets its own model_id.
  • An answer that runs past 300 s is cut. Cap max_tokens for long generations.
  • A prompt longer than max_model_len gets 400; other requests are unaffected.

Custom runtimes: /predict

Your class’s predict(data: bytes) always receives bytes. The Content-Type header tells the platform how to get them from your request:

Content-TypeRequest bodypredict(data) gets
image/*audio/*video/*A fileThe file’s bytes
application/octet-streamAny bytes, including your own JSON. Also used when the header is missingThe body as is
multipart/form-dataA form with a file fieldThe first file’s bytes
application/json{"url": "https://…"}The file at that URL, downloaded for you. Presigned S3 links work
application/json{"data_b64": "…"}The decoded bytes

Any other Content-Type gets 400 before your code runs.

warn
application/json is an envelope for url or data_b64 only; any other field is rejected. To send your own JSON (an image plus parameters, say), post it as application/octet-stream and json.loads(data) inside predict. The SDK’s predict_json() does this for you.
Shell
# a file
curl -X POST "$ENDPOINT" -H "X-API-Key: $PREDICT_KEY" \
  -H "Content-Type: image/jpeg" --data-binary @cat.jpg

# your own JSON: octet-stream, not application/json
curl -X POST "$ENDPOINT" -H "X-API-Key: $PREDICT_KEY" \
  -H "Content-Type: application/octet-stream" \
  -d '{"prompt": "a red bicycle", "steps": 30}'

# let the platform fetch a large input
curl -X POST "$ENDPOINT" -H "X-API-Key: $PREDICT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/scan.tiff"}'

From Python

client.py
from resontech import InferenceClient

client = InferenceClient(url=ENDPOINT, api_key=PREDICT_KEY)

client.predict_file("cat.jpg")                          # reads the file, guesses the type
client.predict(img_bytes, content_type="image/png")     # bytes in memory
client.predict_url("https://example.com/scan.tiff")     # the platform fetches it
client.predict_json({"image_b64": b64, "threshold": 0.7})
client.predict_many(files, concurrency=8)               # N requests in flight

Keep concurrency at or below max_ongoing_requests × workers; higher values only wait in the central server’s queue. Each response carries an X-Request-Id; quote it when you ask Support about a call.

Limits and errors

StatusWhenWhat to do
401AuthErrorThe X-API-Key header is missing, wrong, or holds a rotated keyUse the current key in X-API-Key, or rotate it on the job’s page
400ValidationErrorThe Content-Type isn’t supported, or the JSON has fields other than url and data_b64Fix the request; see the table above
400ValidationErrorYour predict raised ValueErrorFix the input your model rejected; the error message is in the response
400ValidationErrorThe prompt is longer than max_model_len (vLLM)Shorten the prompt, or raise max_model_len
413PredictErrorThe body, or the file behind the URL, is over 200 MBSplit the input, or send a smaller file
500ServerErrorYour predict raised another exception (its class name is in error.type), or returned something that isn’t JSONReturn plain lists, numbers and strings; convert tensors and numpy values
503ServerErrorEvery copy of the model is busy and the queue is fullRetry with backoff, or add workers
504PredictErrorYour predict ran longer than 60 sSpeed it up or split the input. For a model that needs longer, contact Support

The second line in each Status cell is the exception the Python client raises.

The Python client retries 502, 503 and 504 with backoff and never retries a 4xx. What happens as load grows, stage by stage: Choosing hardware → Under load.

Callbacks

A custom runtime can also post each result to your server after it answers. Set a job-wide callback URL in the wizard, or send X-Callback-Url on a single request.

Value
Body{"request_id": "…", "result": {…}}
X-Reson-Signaturesha256=<HMAC of the body>
X-Reson-DeliveryA unique id per delivery, for deduplication
RetriesWith backoff, whenever your server answers anything but 2xx

Next

  • Request shapes for real models: vLLM examples and custom model examples, where each folder ships a predict.py.
  • The runtime page has a playground that calls your endpoint from the browser.
PreviousChoosing hardwareNext Benchmarks