> ## Documentation Index
> Fetch the complete documentation index at: https://allhandsai-docs-provider-connections.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuring Custom Sandbox Images

> Configure custom sandbox images through the Runtime API, each kept ready in its own warm pool and independently selectable by users.

**Requirements:**

* OpenHands Enterprise **0.64.0 or later**
* Custom images built and pushed as described in [Building a Custom Image](/enterprise/custom-sandbox-images/building-custom-images)

## How It Works

Custom sandbox images are registered through the **Runtime API** — a
management interface built into OpenHands Enterprise. The process has three
steps:

1. **Set an admin password** in the install admin UI. This secures the Runtime
   API so only authorized administrators can register or remove images.
   On Helm installs, use the `admin-password` secret created during installation.

2. **Register images via the API.** Use the helper script below to give each
   image a name and tell OpenHands where to pull it from. OpenHands pulls the
   image from the registry you specify and keeps a pool of ready sandboxes for
   it. No restarts or redeployments are needed — new images become available
   within about a minute.

3. **Users choose their environment.** Each registered image appears in the
   user's **Settings → Application → Default Sandbox** dropdown. Users pick
   their default and all their new conversations start in that environment.

***

## Step 1: Confirm the Admin Password

The Runtime API admin endpoints require an admin password.

<Tabs>
  <Tab title="VM Install">
    The password is **auto-generated at install** (`{{repl RandomString 32}}`)
    and stored in the `admin-password` Kubernetes secret. The helper script in
    Step 2 reads it from the pod environment automatically — no action
    required for a standard installation.

    To set a memorable password or rotate the generated one:

    1. Open the **Admin Console** at `https://admin.<your-base-domain>:30000`.
    2. Navigate to **Config → Sandbox Configuration → Runtime API Admin Password**.
    3. Enter your new password and click **Save config**, then **Deploy**.

    The Admin Console updates the secret and rolls out the runtime-api
    automatically. The password persists across all future Admin Console
    deploys.

    <Warning>
      Do not use `kubectl patch` to set the password. The Admin Console manages
      the `admin-password` secret and overwrites it on every deploy, so a
      patched value is lost the next time you save any config change. Always
      use the Admin Console field.
    </Warning>
  </Tab>

  <Tab title="Helm">
    The password was set when you created the `admin-password` secret during
    installation:

    ```bash theme={null}
    kubectl -n openhands create secret generic admin-password \
      --from-literal=admin-password=<your-password>
    ```

    The Helm commands in Step 2 read this secret with `kubectl`.

    To rotate the password:

    ```bash theme={null}
    # Store the new value somewhere secure before running this
    kubectl -n openhands delete secret admin-password
    kubectl -n openhands create secret generic admin-password \
      --from-literal=admin-password=$(openssl rand -base64 24)
    kubectl -n openhands rollout restart deployment \
      -l app.kubernetes.io/name=runtime-api
    kubectl -n openhands rollout status deployment \
      -l app.kubernetes.io/name=runtime-api
    ```
  </Tab>
</Tabs>

***

## Enable Overlay Mode (Helm Installs Only)

By default on Helm installs, saving any configuration via the API takes over
warm pool management and the installer-managed `v1_current` default is
ignored. Enable overlay mode so API-saved configurations sit alongside
`v1_current` rather than replacing it. VM installs have overlay mode enabled
by default and can skip this section.

Merge these settings into the same `values.yaml` used for the
[Kubernetes installation](/enterprise/k8s-install/installation). Keep the
other values for your release:

```yaml theme={null}
runtime-api:
  warmRuntimes:
    enabled: true
  env:
    WARM_RUNTIME_CONFIG_OVERLAY: "1"
```

Upgrade with the licensed chart and wait for Runtime API to roll out:

```bash theme={null}
helm upgrade openhands oci://registry.replicated.com/openhands/openhands \
  --namespace openhands --values values.yaml
kubectl -n openhands rollout status deployment \
  -l 'app.kubernetes.io/name=runtime-api,app.kubernetes.io/instance=openhands'
```

You can confirm overlay mode is active after Step 2 by running
`python3 scripts/warm_runtime_configs.py list` — `v1_current` should appear
with `"source": "file"`.

***

## Step 2: Install the Management CLI

We publish a small Python CLI — the **`runtime-api-configs`** plugin — that
wraps the runtime-api admin endpoints. It uses only the Python standard
library and runs from any host that can reach the runtime-api (or from
inside the pod on Helm installs). Install it once with OpenHands
[extensions](https://github.com/OpenHands/extensions):

```bash theme={null}
git clone --depth 1 https://github.com/OpenHands/extensions
cd extensions/plugins/runtime-api-configs
```

<Note>
  Prefer to run everything as raw HTTP calls? The [API Reference](#api-reference)
  section at the bottom of this page documents the endpoints so you can drive
  them directly from `curl` or any HTTP client. All following steps show the
  CLI form because it is shorter and handles the PBKDF2 handshake for you.
</Note>

<Tabs>
  <Tab title="VM Install">
    The runtime-api is exposed externally at
    `https://runtime-api.<your-base-domain>`. Export the two env vars the
    CLI needs — the URL, and the admin password you confirmed in Step 1:

    ```bash theme={null}
    export RUNTIME_API_URL=https://runtime-api.<your-base-domain>
    export ADMIN_PASSWORD=<the-password-from-Step-1>
    python3 scripts/warm_runtime_configs.py list
    ```

    That is the full setup. No `kubectl`, no SSH, no cluster access. The
    CLI uses the admin password directly for `save` and `delete` (via the
    PBKDF2 handshake), and for `list` and `template` it logs in as admin
    and fetches the read-only API key over HTTPS from
    `/api/admin/api-keys`.

    Prefer to pull the credentials straight from Kubernetes secrets in one
    shot? That is the `bootstrap` subcommand — kept as an optional
    convenience for cluster operators and CI, and documented in
    [Advanced: bootstrap from Kubernetes](#advanced-bootstrap-from-kubernetes)
    at the bottom of this page.
  </Tab>

  <Tab title="Helm">
    The [Kubernetes installation guide](/enterprise/k8s-install/installation#step-3-configure-values)
    enables Runtime API ingress. Use its configured hostname and read the
    admin password from the secret created in Step 1:

    ```bash theme={null}
    export RUNTIME_API_URL=https://runtime-api.openhands.example.com
    export ADMIN_PASSWORD=$(kubectl -n openhands get secret admin-password \
      -o jsonpath='{.data.admin-password}' | base64 -d)
    python3 scripts/warm_runtime_configs.py list
    ```

    Replace the example URL with your `runtime-api.ingress.host`. If you
    disabled ingress, port-forward the release-named Service instead:

    ```bash theme={null}
    RUNTIME_API_SERVICE=$(kubectl -n openhands get svc \
      -l 'app.kubernetes.io/name=runtime-api,app.kubernetes.io/instance=openhands' \
      -o jsonpath='{.items[0].metadata.name}')
    test -n "$RUNTIME_API_SERVICE"
    kubectl -n openhands port-forward "svc/$RUNTIME_API_SERVICE" 5000:5000 &
    export RUNTIME_API_URL=http://localhost:5000
    python3 scripts/warm_runtime_configs.py list
    ```

    If your Helm release has another name, change the
    `app.kubernetes.io/instance` selector to match it.

    As on VM installs, `list` and `template` fetch the read-only API key
    through the admin login. The port-forward path uses local HTTP;
    `ADMIN_PASSWORD` is the only credential you need to export.
  </Tab>
</Tabs>

<Note>
  For `save` and `delete`, the CLI runs a PBKDF2 challenge-response login
  with `ADMIN_PASSWORD` to obtain a 24-hour JWT and calls the admin routes
  as `Authorization: Bearer <jwt>`. For `list` and `template`, it uses that
  same admin login to fetch the `default` read-only API key from
  `/api/admin/api-keys` and sends it as `X-API-Key` on
  `/api/warm-runtime-configs`. Export `API_KEY` explicitly if you would
  rather skip the extra login round trip on reads. See the plugin's
  [`SKILL.md`](https://github.com/OpenHands/extensions/blob/main/plugins/runtime-api-configs/SKILL.md)
  for the full subcommand reference.
</Note>

***

## Step 3: Save Your First Configuration

Do not write configurations from scratch. The default configuration contains
install-specific values (callback URLs, CA bundles, workspace paths) that
sandboxes need to function. The CLI's `template` subcommand fetches an
existing configuration, strips the identity fields, and lets you override the
image and pool size in one step.

<Tabs>
  <Tab title="VM Install">
    The default `v1_current` pool keeps running while you add configurations.
    Derive your custom configuration from the template and save it:

    ```bash theme={null}
    python3 scripts/warm_runtime_configs.py template v1_current \
      --image ghcr.io/your-org/openhands-php:8.4-v1 --count 1 \
      --output php-web.json
    python3 scripts/warm_runtime_configs.py save php-web --file php-web.json
    ```

    Piping directly into `save` works too:

    ```bash theme={null}
    python3 scripts/warm_runtime_configs.py template v1_current \
      --image ghcr.io/your-org/openhands-php:8.4-v1 --count 1 \
      | python3 scripts/warm_runtime_configs.py save php-web --file -
    ```
  </Tab>

  <Tab title="Helm">
    With overlay mode enabled (see [Enable Overlay Mode](#enable-overlay-mode-helm-installs-only)
    above), the default `v1_current` pool keeps running while you add
    configurations. Derive your custom configuration from the template and
    save it:

    ```bash theme={null}
    python3 scripts/warm_runtime_configs.py template v1_current \
      --image ghcr.io/your-org/openhands-php:8.4-v1 --count 1 \
      --output php-web.json
    python3 scripts/warm_runtime_configs.py save php-web --file php-web.json
    ```
  </Tab>
</Tabs>

***

## Configuration Format

| Field | Type | Required | Description |
| - | - | - | - |
| `image` | string | Yes | Full image reference (e.g. `ghcr.io/your-org/openhands-php:8.4-v1`) |
| `working_dir` | string | Yes | Working directory inside the sandbox — copy from the default |
| `command` | array | Yes | Agent-server start command — copy from the default |
| `environment` | object | Yes | Environment variables the sandbox boots with — copy from the default |
| `count` | integer | No | Warm pods to keep ready. Falls back to the installer-wide **Warm Runtime Count** setting — on Replicated installs this defaults to **1** (adjustable in **Config → Sandbox Configuration**); the code-level fallback when nothing is configured is **3** |
| `run_as_user` | integer | No | Copy from the installer default so warm pods match application start requests |
| `run_as_group` | integer | No | Copy from the installer default so warm pods match application start requests |
| `fs_group` | integer | No | Copy from the installer default so warm pods match application start requests |
| `fuse_s3_mount` | boolean | No | Use the fusey S3 workspace instead of a PVC. Copy from the installer default when unsure — most installs do not set it. |

The configuration name comes from the URL path (the `save <name>` argument),
not the body. A `source` field appears in list responses (`file` for
installer-managed entries, `db` for API-managed entries) but must not be
included in saved configurations.

The application uses the image reference as the sandbox spec ID. Give every
selectable configuration a distinct image reference; configurations that share
an image reference cannot be selected independently.

<Tip>
  Set `count` explicitly. Every warm pod reserves the full sandbox resource
  envelope (25 Gi of ephemeral storage by default) whether or not it is in
  use, so the sum of all pool sizes must fit your node capacity. Pools that
  exceed capacity show up as `Pending` pods. Start with `count: 1` per image
  and grow the pools that see real traffic.
</Tip>

***

## Step 4: Verify

Confirm your configurations were saved:

```bash theme={null}
python3 scripts/warm_runtime_configs.py list
```

The response shows each saved configuration with its name, image, pool size,
and source. Within about a minute the pool is ready. Open
**Settings → Application → Default Sandbox** — your image name appears in the
dropdown. Select it and start a conversation to confirm it loads in a few
seconds rather than 20 or more.

On Helm, test both the default and custom images with a simple tool check. The
dropdown shows image references rather than configuration names. If you test
concurrent conversations, leave node capacity for active sandboxes and the
replacement warm pods; a `Pending` replacement means the pool is not ready
for the next conversation.

If the image does not appear or conversations cold-start, see
[Troubleshooting](#troubleshooting) below.

***

## Updating and Deleting Configurations

Update by re-deriving from the current default and saving under the same
name:

```bash theme={null}
python3 scripts/warm_runtime_configs.py template v1_current \
  --image ghcr.io/your-org/openhands-php:8.4-v2 --count 1 \
  | python3 scripts/warm_runtime_configs.py save php-web --file -
```

Within a minute the reconciler stops the old pods and starts pods on the new
image. Delete a configuration to remove its pool:

```bash theme={null}
python3 scripts/warm_runtime_configs.py delete php-web
```

If the deleted name overrides an installer-managed entry, the underlying
installer entry becomes effective again. Confirm with
`python3 scripts/warm_runtime_configs.py list` — its `source` changes from `db` to `file`.

Keep superseded image tags available in your registry while conversations that
used them can still resume: a paused conversation resumes on its **original**
image. Delete old tags only after the conversations that used them are gone
(stopped sandboxes are cleaned up after 10 days by default).

***

## After Upgrading OpenHands Enterprise

<Warning>
  API-managed configurations are **frozen snapshots** — upgrades do not touch
  them. The installer-managed `v1_current` entry updates automatically unless
  a database entry with that name overrides it. Each release expects a
  specific agent-server version and may add or change sandbox environment
  variables. After every OHE upgrade:

  1. Rebuild your custom images on the release's new agent-server base version.
  2. Re-export the default template (Step 3) from the refreshed ConfigMap.
  3. Re-derive and save each API-managed custom configuration from the new template.
  4. If you intentionally override `v1_current`, refresh or delete that override
     so the new installer-managed entry can take effect.

  Skipping this leaves configurations pinned to the previous agent-server
  version. Existing features keep working - agent-server is largely forward
  and backward compatible - but any newer feature with a declared minimum
  version (Hooks, MCP test, MCP OAuth, and future additions) fails on those
  sandboxes with an `AGENT_SERVER_VERSION_TOO_OLD` error until the
  configurations are refreshed. See
  [Version Compatibility](/enterprise/custom-sandbox-images/building-custom-images#version-compatibility)
  for details.
</Warning>

***

## Returning an Entry to Installer Management

Delete a same-named database override to restore the installer-managed entry
on the next reconciler cycle:

```bash theme={null}
python3 scripts/warm_runtime_configs.py delete v1_current
python3 scripts/warm_runtime_configs.py list   # v1_current now reports "source": "file"
```

Other API-managed configurations continue running. Delete them individually
when you no longer want their pools or images in the application's selector.

***

## Troubleshooting

| Symptom | Cause and fix |
| - | - |
| `HTTP 403: Admin functionality is disabled` | The runtime-api deployment has no admin password configured. On Replicated installs, set **Runtime API Admin Password** per Step 1 and deploy. |
| `HTTP 401` on login | Wrong password, or the challenge expired (challenges are single-use and expire after 5 minutes; the script fetches a fresh one per call). To verify the current password value: `kubectl get secret admin-password -n openhands -o jsonpath='{.data.admin-password}' \| base64 -d`. On Replicated installs, the password only changes if you update it in the Admin Console and deploy. |
| `HTTP 401: ...provide a valid API key...` on list | The list endpoint authenticates with `X-API-Key`, not the admin JWT. Use the helper script. |
| Saved a config but the dropdown does not show it | The app server caches the config list for 60 seconds; the UI may cache it for up to 5 minutes. Wait, then navigate away from and back to the Settings page to prompt a fresh fetch. Confirm the config was saved with `python3 scripts/warm_runtime_configs.py list`. |
| No warm pods appear | Check the reconciler log: `JOB=$(kubectl -n openhands get jobs --sort-by=.metadata.creationTimestamp -o name \| grep warm-runtimes \| tail -1) && kubectl -n openhands logs "$JOB"`. Look for image pull errors or scheduling failures. |
| Warm pods `Pending` | Insufficient node resources. Check with `kubectl -n openhands get deploy -l 'runtime_id,!session_id'`. Every warm pod reserves the full sandbox resource envelope; lower the pool `count`s or add capacity. |
| Conversations cold-start despite warm pods | Pool exhausted or configuration recently changed. See [How Warm Pods Are Claimed](/enterprise/custom-sandbox-images/using-custom-images#how-warm-pods-are-claimed). |
| Sandbox fails with an agent-server version error | The custom image's base version does not match the release. Rebuild on the expected agent-server version. See [Version Compatibility](/enterprise/custom-sandbox-images/building-custom-images#version-compatibility). |
| Conversations start but never show agent output | The configuration's `environment` is missing install-specific values. Rebuild the configuration from the default template (Step 3). |

***

## API Reference

The endpoints below are served by the runtime-api service.

**Admin authentication** (required for save, delete, and — if you skip the
`X-API-Key` header on reads — for `GET /api/admin/api-keys`):

1. `GET /api/admin/challenge` returns `{challenge, salt, iterations}`. Challenges
   are single-use and expire after 5 minutes. `salt` is returned as an ASCII
   hex string.
2. Compute `PBKDF2-HMAC-SHA256(password_utf8, (salt_hex + challenge).utf8, iterations, dklen=32)`
   and hex-encode the result. The `salt` value returned above goes into PBKDF2
   as **its ASCII hex string**, not decoded to raw bytes first — concatenate
   `salt` and `challenge` as strings, then UTF-8 encode.
3. `POST /api/admin/login` with `{"challenge": ..., "hash": ...}` returns
   `{"token": ...}`, a JWT valid for 24 hours.
4. Send `Authorization: Bearer <token>` on admin requests.

**Fetch the read-only API key over HTTPS** (admin — lets you drive
`/api/warm-runtime-configs` without any cluster access):

```http theme={null}
GET /api/admin/api-keys
Authorization: Bearer {admin-jwt}
```

Returns `200` with `[{"id": ..., "name": "default", "key_value": "...", ...}, ...]`.
Use the `key_value` of the `name: "default"` entry as your `X-API-Key`.

**List configurations** (regular API key, not admin):

```http theme={null}
GET /api/warm-runtime-configs
X-API-Key: {api-key}
```

Returns `200` with the effective configuration set:

```json theme={null}
{
  "configs": [
    {
      "name": "v1_current",
      "image": "ghcr.io/openhands/agent-server:1.46.0-python",
      "source": "file",
      "count": 1
    },
    {
      "name": "php-web",
      "image": "ghcr.io/your-org/openhands-php:8.4-v1",
      "source": "db",
      "count": 1
    }
  ]
}
```

`source: "file"` — installer-managed entry. `source: "db"` — API-managed
entry. The list is the full effective set: ConfigMap entries merged with
same-named API entries overriding them.

**Create or update a configuration** (admin):

```http theme={null}
PUT /api/admin/warm-runtime-configs/{name}
Authorization: Bearer {admin-jwt}
Content-Type: application/json

{"image": "...", "working_dir": "...", "command": [...], "environment": {...}, "count": 1}
```

Returns `200` with the saved configuration. Creates or overwrites; the name
in the URL is the identity.

**Delete a configuration** (admin):

```http theme={null}
DELETE /api/admin/warm-runtime-configs/{name}
Authorization: Bearer {admin-jwt}
```

Returns `200` with a confirmation message, or `404` if no database
configuration has that name. When the deleted name also exists in the
installer-managed ConfigMap, that ConfigMap entry becomes effective again.

***

## Advanced: bootstrap from Kubernetes

The CLI's `bootstrap` subcommand pulls `RUNTIME_API_URL`, `API_KEY`, and
`ADMIN_PASSWORD` from Kubernetes secrets in one step. It is optional — the
HTTPS-only flow in Step 2 is preferred for interactive administration. Use
`bootstrap` when you have `kubectl` access anyway and want a single
one-liner for a CI job or an operator runbook.

<Tabs>
  <Tab title="VM Install">
    The Replicated embedded k0s cluster stores its kubeconfig at
    `/var/lib/k0s/pki/admin.conf`, which is root-owned. Run under `sudo -E`
    so the CLI's `kubectl` calls can read it:

    ```bash theme={null}
    eval "$(sudo -E python3 scripts/warm_runtime_configs.py bootstrap \
      --namespace openhands)"
    python3 scripts/warm_runtime_configs.py list
    ```

    `bootstrap` prints three `export` lines. After the `eval`, subsequent
    commands run from any host with network access to the ingress — no
    further cluster access needed.
  </Tab>

  <Tab title="Helm">
    Run against your own `kubectl` context (no `sudo` needed on typical
    Helm-managed clusters). Port-forward first, then bootstrap with
    `--skip-url` so your `port-forward` target is not overwritten:

    ```bash theme={null}
    RUNTIME_API_SERVICE=$(kubectl -n openhands get svc \
      -l 'app.kubernetes.io/name=runtime-api,app.kubernetes.io/instance=openhands' \
      -o jsonpath='{.items[0].metadata.name}')
    test -n "$RUNTIME_API_SERVICE"
    kubectl -n openhands port-forward "svc/$RUNTIME_API_SERVICE" 5000:5000 &
    export RUNTIME_API_URL=http://localhost:5000
    eval "$(python3 scripts/warm_runtime_configs.py bootstrap \
      --namespace openhands --skip-url)"
    python3 scripts/warm_runtime_configs.py list
    ```

    If your Helm release has another name, change the
    `app.kubernetes.io/instance` selector to match it.
  </Tab>
</Tabs>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.