> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kiteml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sim runs: evaluate a policy on Kite's machines

> Send a Kite SDK spec and get your policy evaluated in simulation on Kite's machines: every episode traced, every evaluator scored, a scorecard, and the whole run directory as one verified download.

A **sim run** evaluates a policy. You send a spec: the policy, the world it acts in, the task, how many episodes, and the evaluators that score them. Kite runs it on its own machines, on an NVIDIA L4 when the policy runs on `cuda` and on a CPU machine otherwise. You get a scorecard, every evaluator's value with its confidence interval, and the run directory as one archive.

A sim run is reproducible. Its **manifest hash** identifies it: the same spec, files and Kite version give the same hash. Its **run hash** covers every episode's trace, so two runs that behaved the same have the same run hash.

<Note>
  **Beta.** Sim runs are open to every account and charged by the minute (see [Pricing](#pricing)). Their shapes can still change before they are stable, without a new `Kite-Version`. Changes are announced before they ship.
</Note>

## Check a spec first

`POST /v1/sim_runs/validate` runs every check a run would, for free, and creates nothing. It plans the spec, confirms every file it names is accounted for, and quotes the price.

A spec is YAML. This one runs a scripted pick on the demo scene, three episodes, scored by the evaluators Kite ships:

```yaml smoke.yaml theme={"system"}
kite_spec_version: "2"
name: smoke
policy: {kind: scripted_pick, ref: scripted_pick}
world: {kind: mujoco, scene: training_match, render: {shadow: true}}
task: {id: pick_cube}
control: {rate_hz: 50, physics_dt_ns: 2000000, max_steps: 400}
gate: {deadline_ns: 20000000, request_interval_steps: 10, max_reuse_steps: 100, safe_stop_after: 5,
       chunk_blend: temporal_ensemble, ensemble_m: 0.01}
episodes: {count: 3, seed_base: 1000}
compute: null
trace: {image_every_n: 10}
eval: {evals_dir: ../evals}
```

Send its text with the files it names. `../evals` is a folder Kite ships, so you name it instead of uploading it:

```bash theme={"system"}
jq -n --rawfile spec smoke.yaml '{spec_yaml: $spec, spec_path: "specs/smoke.yaml",
  files: {"../evals": {kind: "directory", shipped: "evals"}}}' > request.json

curl https://api.kiteml.com/v1/sim_runs/validate \
  -H "Authorization: Bearer $KITE_API_KEY" \
  -H "Kite-Version: 2026-10-12" \
  -H "Content-Type: application/json" \
  -d @request.json
```

```json theme={"system"}
{
  "object": "sim_run_validation",
  "ok": true,
  "kite_version": "0.2.0",
  "plan": {"name": "smoke", "episodes": {"count": 3, "seed_base": 1000, "seed_last": 1002}, "ok": true},
  "price": {"tier": "cpu_basic", "tokens_per_hour": 5000, "max_minutes": 60, "reserved": 5000, "billing": true}
}
```

A spec Kite refuses keeps its own diagnosis: `400 spec_invalid` with the field in `param`, and `details.kite_code`, `details.remediation` (the fix) and `details.location` (the file and line).

### Files

`files` maps each path the spec names, as it names it, to where its bytes come from. A path with no entry is `400 files_required`, which lists every missing path in `details.files`.

| Source | Example | |
| - | - | - |
| `shipped` | `{"kind": "directory", "shipped": "evals"}` | A file or folder Kite ships: `evals`, `tasks`, `profiles`, `specs`, `templates/robots`. |
| `sha256` | `{"kind": "directory", "sha256": "…"}` | A file or folder Kite ships, by its hash. A copy of a shipped file uploads nothing. |
| `text`, `base64` | `{"text": "…"}`, `{"kind": "directory", "base64": "<a .tar>"}` | The bytes inline, up to about 15 MB in all. |
| `hf` | `{"hf": {"repo": "owner/name", "revision": "<40-character commit>", "path": "model.safetensors"}}` | One file of a public Hugging Face repository, at a commit. Kite pins its sha256 when the run is created. |

Only Python that Kite ships runs on Kite's machines. Evaluators and task files must be ones Kite ships, or the request is refused with `code_not_allowed`.

## Create a run

`POST /v1/sim_runs` takes the same body, plus optional `options` and `label`. It answers `202` with a `queued` run. Send an `Idempotency-Key` header so a retry can't run twice.

```bash theme={"system"}
curl https://api.kiteml.com/v1/sim_runs \
  -H "Authorization: Bearer $KITE_API_KEY" \
  -H "Kite-Version: 2026-10-12" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d @request.json
```

| Option | Type | Description |
| - | - | - |
| `options.max_minutes` | integer | The run's cap, 1 to 720: what it reserves and the most it can cost. Default 60 on CPU, 120 on an L4. |
| `options.no_eval` | boolean | Run the episodes only: no evaluators, no scorecard. |
| `options.verify_determinism` | boolean | Run episode 0 twice and fail the run if the two differ. |
| `overrides` | array of strings | `FIELD=VALUE` changes to the spec, such as `episodes.count=20`. |
| `label` | string | Up to 120 characters, for your own list. |

You can have two runs building at once (`429 concurrency_limit_exceeded` past that).

## Track progress

`GET /v1/sim_runs/{id}` returns the run, live. `status` goes `queued` → `processing` → `succeeded`, `failed` or `canceled`. `queued` includes waiting for a machine: an L4 can take minutes to free up. While `processing`, `phase` says what it is doing (`materializing`, `planning`, `fetching`, `running`, `verifying`, `packaging`) and `episodes` how many are done. Poll every few seconds, or subscribe to the `sim_run.succeeded`, `sim_run.failed` and `sim_run.canceled` [webhooks](/platform-api/api-reference#operations-and-webhooks).

Once it has succeeded, `result` holds the run's identity and what it measured:

```json theme={"system"}
"result": {
  "manifest_hash": "0cb1b0439dbf9acb484e58e356cc6e5915ad8168a9a0b4aaaca2f999ac2aa5be",
  "run_hash": "d40d870edb0cf91afdcdf377ee22c0673590631d7a243901deaa2a7b8cb4a986",
  "verified": true,
  "metrics": [{"id": "task_success", "version": "1.0.0", "value": 1.0, "interval": [0.44, 1.0], "n": 3}]
}
```

`GET /v1/sim_runs/{id}/report` returns the full scorecard: every evaluator's value, interval and count, and what was not measured. `GET /v1/sim_runs/{id}/logs?tail=200` returns what the run printed, while it runs and after.

## Download the run

Once `succeeded`, `download` links the run directory as one archive:

```json theme={"system"}
"download": {
  "url": "https://storage.googleapis.com/…/srun_01M4KQ89FTB4XA37ZW68ATCYBX.tar.gz?X-Goog-Algorithm=GOOG4-RSA-SHA256&…",
  "requires_auth": false,
  "expires_at": "2026-10-10T21:16:21Z",
  "filename": "srun_01M4KQ89FTB4XA37ZW68ATCYBX.tar.gz",
  "bytes": 4022175,
  "sha256": "9de320aae85e9ecc7cb129548d8fea8ae970c20f5b5a441a4585da59b076dc8d"
}
```

`url` is a signed link, valid for an hour. Fetch it **without** your API key, check the checksum, and extract. The archive unpacks to `<run id>/`:

```bash theme={"system"}
curl -fL "$URL" -o srun.tar.gz
echo "$SHA256  srun.tar.gz" | shasum -a 256 -c
tar xzf srun.tar.gz
```

`GET /v1/sim_runs/{id}/archive` redirects to a fresh link, and `GET /v1/sim_runs/{id}/files` lists every file with its sha256.

## Without writing HTTP

The [CLI](/tools/cli-and-mcp) finds the files a spec names on your disk and sends each one: by hash when Kite ships it, inline otherwise.

```bash theme={"system"}
pip install kiteml-cli
kite auth login
kite sim validate specs/smoke.yaml
kite sim run specs/smoke.yaml --wait --out ./runs
```

`--wait --out` waits for the run, downloads it, verifies its checksum, and unpacks it to `./runs/<run id>/`. A checkpoint on the Hugging Face Hub goes as a reference: `--hf PATH=OWNER/NAME@COMMIT:FILE`. AI agents get the same through the `kite_sim_*` tools of the [MCP server](/tools/cli-and-mcp).

## Cancel and resume

`POST /v1/sim_runs/{id}/cancel` stops a `queued` or `processing` run. It reads `canceling` until its machine has stopped, then `canceled`. `POST /v1/sim_runs/{id}/resume` runs a `failed` or `canceled` run again from its request, as a new attempt.

## Pricing

A run costs the minutes its machine runs: **2,000 tokens an hour on an L4** and **5,000 tokens an hour on a CPU machine**, counted in whole minutes. A create reserves the run's cap, and when the run ends the minutes it didn't use come back. A run that succeeded pays its minutes. One you canceled pays the minutes it ran. One that failed, or never got a machine, costs nothing. Validation is free. A balance short of the cap is `402 insufficient_tokens`, with `details.tokens_required`; lower `options.max_minutes` or top up.

## Errors

| Code | Where | Meaning |
| - | - | - |
| `spec_invalid` | validate, create | Kite refused the spec: `param`, `details.kite_code`, `details.remediation`, `details.location`. |
| `files_required` | validate, create | The spec names paths the request doesn't carry: `details.files`. |
| `code_not_allowed` | validate, create | A Python file Kite doesn't ship. |
| `file_unavailable`, `file_hash_mismatch`, `file_invalid` | validate, create | A file's source is missing, or its bytes aren't what it says. |
| `device_unavailable` | validate, create | `policy.device` is one Kite's machines don't have: use `cuda` or `cpu`. |
| `insufficient_tokens` | create, resume | `402`: the balance can't cover the cap. |
| `concurrency_limit_exceeded` | create | `429`: two runs are already building. |
| `policy_failed`, `world_failed`, `evaluator_failed`, `run_failed`, `input_unavailable` | the run | The run's own error while it ran, with its `kite_code` in `details`. |
| `launch_failed` | the run | No machine could be had. Resume tries again. |
| `interrupted` | the run | The run's machine stopped before it finished. Resume runs it again. |

## The sim run object

| Field | Type | Description |
| - | - | - |
| `id` | string | `srun_` and a time-ordered id. |
| `status` | string | `queued`, `processing`, `canceling`, `succeeded`, `failed` or `canceled`. |
| `phase` | string or null | While `processing`: `materializing`, `planning`, `fetching`, `running`, `verifying` or `packaging`. |
| `progress` | number | 0 to 1, mostly the episodes done. |
| `episodes` | object or null | `done` and `total`, once the run is planned. |
| `error` | object or null | When `failed`: `code`, `message`, `param`, `details`, and `resumable`. |
| `spec` | object | The spec's `path` as you sent it, and its `name`. |
| `options` | object | `no_eval`, `verify_determinism`, and `max_minutes` when you set it. |
| `result` | object or null | Once `succeeded`: `manifest_hash`, `run_hash`, `verified`, `determinism_verified`, and every evaluator's `metrics`. |
| `download` | object or null | Once `succeeded`: `url`, `requires_auth`, `expires_at`, `filename`, `bytes`, `sha256`. |
| `tokens` | object | `charged` (the cap reserved at create, and again at each resume) and `refunded`. |
| `label`, `webhook_metadata` | | What you sent at create. |
| `created_at`, `started_at`, `completed_at` | string | RFC 3339. `started_at` is when the run's machine started. |

Every endpoint is in the [API reference](/platform-api/api-reference).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.