Sim runs: evaluate a policy on Kite's machines

A sim run evaluates a policy. You send a spec: the policy, the world it acts in, the task, how many episodes, and the evaluators that score them. Kite runs it on its own machines, on an NVIDIA L4 when the policy runs on cuda and on a CPU machine otherwise. You get a scorecard, every evaluator’s value with its confidence interval, and the run directory as one archive. A sim run is reproducible. Its manifest hash identifies it: the same spec, files and Kite version give the same hash. Its run hash covers every episode’s trace, so two runs that behaved the same have the same run hash.

Beta. Sim runs are open to every account and charged by the minute (see Pricing). Their shapes can still change before they are stable, without a new Kite-Version. Changes are announced before they ship.

Check a spec first

POST /v1/sim_runs/validate runs every check a run would, for free, and creates nothing. It plans the spec, confirms every file it names is accounted for, and quotes the price. A spec is YAML. This one runs a scripted pick on the demo scene, three episodes, scored by the evaluators Kite ships:

smoke.yaml

kite_spec_version: "2"
name: smoke
policy: {kind: scripted_pick, ref: scripted_pick}
world: {kind: mujoco, scene: training_match, render: {shadow: true}}
task: {id: pick_cube}
control: {rate_hz: 50, physics_dt_ns: 2000000, max_steps: 400}
gate: {deadline_ns: 20000000, request_interval_steps: 10, max_reuse_steps: 100, safe_stop_after: 5,
       chunk_blend: temporal_ensemble, ensemble_m: 0.01}
episodes: {count: 3, seed_base: 1000}
compute: null
trace: {image_every_n: 10}
eval: {evals_dir: ../evals}

Send its text with the files it names. ../evals is a folder Kite ships, so you name it instead of uploading it:

jq -n --rawfile spec smoke.yaml '{spec_yaml: $spec, spec_path: "specs/smoke.yaml",
  files: {"../evals": {kind: "directory", shipped: "evals"}}}' > request.json

curl https://api.kiteml.com/v1/sim_runs/validate \
  -H "Authorization: Bearer $KITE_API_KEY" \
  -H "Kite-Version: 2026-10-12" \
  -H "Content-Type: application/json" \
  -d @request.json

{
  "object": "sim_run_validation",
  "ok": true,
  "kite_version": "0.2.0",
  "plan": {"name": "smoke", "episodes": {"count": 3, "seed_base": 1000, "seed_last": 1002}, "ok": true},
  "price": {"tier": "cpu_basic", "tokens_per_hour": 5000, "max_minutes": 60, "reserved": 5000, "billing": true}
}

A spec Kite refuses keeps its own diagnosis: 400 spec_invalid with the field in param, and details.kite_code, details.remediation (the fix) and details.location (the file and line).

Files

files maps each path the spec names, as it names it, to where its bytes come from. A path with no entry is 400 files_required, which lists every missing path in details.files.

Source	Example
`shipped`	`{"kind": "directory", "shipped": "evals"}`	A file or folder Kite ships: `evals`, `tasks`, `profiles`, `specs`, `templates/robots`.
`sha256`	`{"kind": "directory", "sha256": "…"}`	A file or folder Kite ships, by its hash. A copy of a shipped file uploads nothing.
`text`, `base64`	`{"text": "…"}`, `{"kind": "directory", "base64": "<a .tar>"}`	The bytes inline, up to about 15 MB in all.
`hf`	`{"hf": {"repo": "owner/name", "revision": "<40-character commit>", "path": "model.safetensors"}}`	One file of a public Hugging Face repository, at a commit. Kite pins its sha256 when the run is created.

Only Python that Kite ships runs on Kite’s machines. Evaluators and task files must be ones Kite ships, or the request is refused with code_not_allowed.

Create a run

POST /v1/sim_runs takes the same body, plus optional options and label. It answers 202 with a queued run. Send an Idempotency-Key header so a retry can’t run twice.

curl https://api.kiteml.com/v1/sim_runs \
  -H "Authorization: Bearer $KITE_API_KEY" \
  -H "Kite-Version: 2026-10-12" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d @request.json

Option	Type	Description
`options.max_minutes`	integer	The run’s cap, 1 to 720: what it reserves and the most it can cost. Default 60 on CPU, 120 on an L4.
`options.no_eval`	boolean	Run the episodes only: no evaluators, no scorecard.
`options.verify_determinism`	boolean	Run episode 0 twice and fail the run if the two differ.
`overrides`	array of strings	`FIELD=VALUE` changes to the spec, such as `episodes.count=20`.
`label`	string	Up to 120 characters, for your own list.

You can have two runs building at once (429 concurrency_limit_exceeded past that).

Track progress

GET /v1/sim_runs/{id} returns the run, live. status goes queued → processing → succeeded, failed or canceled. queued includes waiting for a machine: an L4 can take minutes to free up. While processing, phase says what it is doing (materializing, planning, fetching, running, verifying, packaging) and episodes how many are done. Poll every few seconds, or subscribe to the sim_run.succeeded, sim_run.failed and sim_run.canceled webhooks. Once it has succeeded, result holds the run’s identity and what it measured:

"result": {
  "manifest_hash": "0cb1b0439dbf9acb484e58e356cc6e5915ad8168a9a0b4aaaca2f999ac2aa5be",
  "run_hash": "d40d870edb0cf91afdcdf377ee22c0673590631d7a243901deaa2a7b8cb4a986",
  "verified": true,
  "metrics": [{"id": "task_success", "version": "1.0.0", "value": 1.0, "interval": [0.44, 1.0], "n": 3}]
}

GET /v1/sim_runs/{id}/report returns the full scorecard: every evaluator’s value, interval and count, and what was not measured. GET /v1/sim_runs/{id}/logs?tail=200 returns what the run printed, while it runs and after.

Download the run

Once succeeded, download links the run directory as one archive:

"download": {
  "url": "https://storage.googleapis.com/…/srun_01M4KQ89FTB4XA37ZW68ATCYBX.tar.gz?X-Goog-Algorithm=GOOG4-RSA-SHA256&…",
  "requires_auth": false,
  "expires_at": "2026-10-10T21:16:21Z",
  "filename": "srun_01M4KQ89FTB4XA37ZW68ATCYBX.tar.gz",
  "bytes": 4022175,
  "sha256": "9de320aae85e9ecc7cb129548d8fea8ae970c20f5b5a441a4585da59b076dc8d"
}

url is a signed link, valid for an hour. Fetch it without your API key, check the checksum, and extract. The archive unpacks to <run id>/:

curl -fL "$URL" -o srun.tar.gz
echo "$SHA256  srun.tar.gz" | shasum -a 256 -c
tar xzf srun.tar.gz

GET /v1/sim_runs/{id}/archive redirects to a fresh link, and GET /v1/sim_runs/{id}/files lists every file with its sha256.

Without writing HTTP

The CLI finds the files a spec names on your disk and sends each one: by hash when Kite ships it, inline otherwise.

pip install kiteml-cli
kite auth login
kite sim validate specs/smoke.yaml
kite sim run specs/smoke.yaml --wait --out ./runs

--wait --out waits for the run, downloads it, verifies its checksum, and unpacks it to ./runs/<run id>/. A checkpoint on the Hugging Face Hub goes as a reference: --hf PATH=OWNER/NAME@COMMIT:FILE. AI agents get the same through the kite_sim_* tools of the MCP server.

Cancel and resume

POST /v1/sim_runs/{id}/cancel stops a queued or processing run. It reads canceling until its machine has stopped, then canceled. POST /v1/sim_runs/{id}/resume runs a failed or canceled run again from its request, as a new attempt.

Pricing

A run costs the minutes its machine runs: 2,000 tokens an hour on an L4 and 5,000 tokens an hour on a CPU machine, counted in whole minutes. A create reserves the run’s cap, and when the run ends the minutes it didn’t use come back. A run that succeeded pays its minutes. One you canceled pays the minutes it ran. One that failed, or never got a machine, costs nothing. Validation is free. A balance short of the cap is 402 insufficient_tokens, with details.tokens_required; lower options.max_minutes or top up.

Errors

Code	Where	Meaning
`spec_invalid`	validate, create	Kite refused the spec: `param`, `details.kite_code`, `details.remediation`, `details.location`.
`files_required`	validate, create	The spec names paths the request doesn’t carry: `details.files`.
`code_not_allowed`	validate, create	A Python file Kite doesn’t ship.
`file_unavailable`, `file_hash_mismatch`, `file_invalid`	validate, create	A file’s source is missing, or its bytes aren’t what it says.
`device_unavailable`	validate, create	`policy.device` is one Kite’s machines don’t have: use `cuda` or `cpu`.
`insufficient_tokens`	create, resume	`402`: the balance can’t cover the cap.
`concurrency_limit_exceeded`	create	`429`: two runs are already building.
`policy_failed`, `world_failed`, `evaluator_failed`, `run_failed`, `input_unavailable`	the run	The run’s own error while it ran, with its `kite_code` in `details`.
`launch_failed`	the run	No machine could be had. Resume tries again.
`interrupted`	the run	The run’s machine stopped before it finished. Resume runs it again.

The sim run object

Field	Type	Description
`id`	string	`srun_` and a time-ordered id.
`status`	string	`queued`, `processing`, `canceling`, `succeeded`, `failed` or `canceled`.
`phase`	string or null	While `processing`: `materializing`, `planning`, `fetching`, `running`, `verifying` or `packaging`.
`progress`	number	0 to 1, mostly the episodes done.
`episodes`	object or null	`done` and `total`, once the run is planned.
`error`	object or null	When `failed`: `code`, `message`, `param`, `details`, and `resumable`.
`spec`	object	The spec’s `path` as you sent it, and its `name`.
`options`	object	`no_eval`, `verify_determinism`, and `max_minutes` when you set it.
`result`	object or null	Once `succeeded`: `manifest_hash`, `run_hash`, `verified`, `determinism_verified`, and every evaluator’s `metrics`.
`download`	object or null	Once `succeeded`: `url`, `requires_auth`, `expires_at`, `filename`, `bytes`, `sha256`.
`tokens`	object	`charged` (the cap reserved at create, and again at each resume) and `refunded`.
`label`, `webhook_metadata`		What you sent at create.
`created_at`, `started_at`, `completed_at`	string	RFC 3339. `started_at` is when the run’s machine started.

Every endpoint is in the API reference.

Get started

Platform API

Tools

Sim runs: evaluate a policy on Kite's machines

Check a spec first

Files

Create a run

Track progress

Download the run

Without writing HTTP

Cancel and resume

Pricing

Errors

The sim run object

​Check a spec first

​Files

​Create a run

​Track progress

​Download the run

​Without writing HTTP

​Cancel and resume

​Pricing

​Errors

​The sim run object

Check a spec first

Files

Create a run

Track progress

Download the run

Without writing HTTP

Cancel and resume

Pricing

Errors

The sim run object