Skip to Content
OperateDeployArcade Deploy on your own cluster (Helm)

Arcade Deploy on your own cluster (Helm)

This page is for platform operators who run Arcade with the Helm chart and want developers to ship custom servers into that installation with arcade deploy. Once you turn deployments on, the runs each deployed server as a workload in your own cluster. Bundles go to an object store you own, and runner images come from a registry you choose.

Bundles stay in your environment: the chart requires an artifact store (deployments.artifacts.url), and the publishes bundles only there. Usage-event tracking (usageTracking.enabled) is off by default in the chart. Leave it off so your installation sends no usage events to Arcade.

The deployments.* values first shipped in chart 1.11.0. Use chart 1.11.1 or later: in 1.11.0 you must set deployments.image.repository yourself, or the chart fails to render, and there are no deployments.runtime.* values.

How it works

  • A developer runs arcade deploy against your installation. The publishes the server’s bundle to your artifact store (S3-compatible, GCS, or Azure Blob).
  • The creates the server in the workers namespace (arcade-workers by default). Each deployed server runs as a Kubernetes Deployment of the runner image (arcadedev/runner), one pod by default, and the pod fetches its bundle from the artifact store.
  • Every deployed server gets its own NetworkPolicy. It accepts traffic only from the . Outbound, it reaches the public internet and cluster DNS, but not private address ranges, link-local addresses (including cloud metadata), or the cluster ranges you list, unless you allowlist a range.

The ’s service gets a namespaced Role in the workers namespace. That Role covers Deployments, Services, Secrets, ServiceAccounts, NetworkPolicies, ResourceQuotas, Leases, read access to pods and pod logs, and read access to pod metrics. Developers never need cluster access.

Prerequisites

  • A working Arcade installation from the Helm chart, with gateway.hostname, ingress.hostname, or deployments.domain set
  • A bucket in an S3-compatible store, Google Cloud Storage, or Azure Blob Storage for release bundles
  • Credentials for that bucket that the can pick up from its environment. The Arcade Engine reads them from the ambient cloud credential chain, never from the bucket URL and never from a deploy request. See Bucket credentials.
  • Access from your cluster to the runner image, either arcadedev/runner on Docker Hub or a copy in your own registry
  • A CNI plugin that enforces Kubernetes NetworkPolicies, so the per-server isolation takes effect

metrics-server is optional. The ’s Role may read pod metrics, but the Arcade Engine doesn’t query them yet, so nothing breaks without it.

Bucket credentials

The opens the artifact store with each cloud SDK’s default credential lookup. Give credentials to the Arcade Engine’s ServiceAccount, <release>-engine in the release namespace, or put them in its environment. Pick one method:

MethodStoreChart
EKS Pod IdentityS31.11.1 or later
Static keys in a SecretS3, Azure Blob1.11.1 or later
EKS IRSAS3The first chart release after 1.11.1
Azure Workload IdentityAzure BlobThe first chart release after 1.11.1
GKE Workload IdentityGCSThe first chart release after 1.11.1
GCP service account key fileGCSThe first chart release after 1.11.1

The identity you give the needs to read and write objects in the bucket. Deployed servers don’t need bucket credentials as long as the Arcade Engine can sign download URLs. See Artifact store.

EKS Pod Identity

EKS Pod Identity needs no chart values. With the EKS Pod Identity Agent  installed, create an IAM role that EKS Pod Identity can assume and associate it with the ’s ServiceAccount. For a release named arcade in the arcade namespace:

Terminal
aws eks create-pod-identity-association \ --cluster-name my-cluster \ --namespace arcade \ --service-account arcade-engine \ --role-arn arn:aws:iam::111122223333:role/arcade-engine

See EKS Pod Identity  for the role’s trust policy.

Static keys

Put the keys in a Secret in the release namespace and pass it to the with engine.extraEnvFrom:

  • S3: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  • Azure Blob: AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY. A shared key also lets the sign download URLs.
Terminal
kubectl -n arcade create secret generic arcade-bundle-store \ --from-literal=AWS_ACCESS_KEY_ID=... \ --from-literal=AWS_SECRET_ACCESS_KEY=...
YAML
values.yaml
engine: extraEnvFrom: - secretRef: name: arcade-bundle-store

Workload identity and key files

These methods need the first chart release after 1.11.1. Chart 1.11.1 has no values for ServiceAccount annotations, pod labels, or extra volumes.

Set up the identity on the cloud side first, following EKS IRSA , Azure Workload Identity , or GKE Workload Identity  for the ’s ServiceAccount. Then set the matching values:

ValueDefaultDescription
engine.serviceAccount.annotations{}Annotations on the Arcade Engine’s ServiceAccount.
engine.podLabels{}Labels on the Arcade Engine’s pods. Keys the chart already sets are rejected.
engine.podAnnotations{}Annotations on the Arcade Engine’s pods. checksum/* keys are rejected.
engine.extraVolumes[]Extra volumes for the Arcade Engine’s pods.
engine.extraVolumeMounts[]Extra volume mounts for the Arcade Engine’s container.

The identity applies to the ’s Deployment only. The Arcade Engine’s Jobs (migration, gateway bootstrap, worker reconcile) use their own ServiceAccounts. Extra volumes and mounts also go into those Jobs.

EKS IRSA:

YAML
values.yaml
engine: serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/arcade-engine

Azure Workload Identity. Also name the storage , with AZURE_STORAGE_ACCOUNT in engine.extraEnv or storage_account= in deployments.artifacts.url:

YAML
values.yaml
engine: serviceAccount: annotations: azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000 podLabels: azure.workload.identity/use: "true"

GKE Workload Identity:

YAML
values.yaml
engine: serviceAccount: annotations: iam.gke.io/gcp-service-account: arcade-engine@my-project.iam.gserviceaccount.com

GCP service key file, mounted from a Secret named gcp-key with a key.json entry:

YAML
values.yaml
engine: extraVolumes: - name: gcp-key secret: secretName: gcp-key extraVolumeMounts: - name: gcp-key mountPath: /var/secrets/gcp readOnly: true extraEnv: - name: GOOGLE_APPLICATION_CREDENTIALS value: /var/secrets/gcp/key.json

Enable deployments

Add the deployments values

Deployments are off by default. Turn them on and name your artifact store:

YAML
values.yaml
deployments: enabled: true artifacts: # s3://, gs:// or azblob:// url: s3://arcade-bundles?region=us-east-1

Use an s3://, gs://, or azblob:// URL. The chart refuses to render when deployments.enabled is true and deployments.artifacts.url is empty. The refuses to start when the URL scheme isn’t one it supports, and names the supported schemes.

The also accepts mem:// and file://, but only its tests use them. They don’t work in a cluster, because a deployed server’s pod can’t read the Arcade Engine’s memory or local disk.

Upgrade the release

Apply the new values to your existing release:

Terminal
helm upgrade arcade \ oci://public.ecr.aws/s5i6x9d1/charts/arcade \ --namespace arcade \ -f values.yaml

With the defaults, this creates the arcade-workers namespace, a default-deny NetworkPolicy for it, a ResourceQuota and LimitRange, and the ’s Role in that namespace.

Deploy a test server

Before the first deploy, make sure the developer has:

  • An in your installation, and a in it they can deploy to.
  • Signed in with arcade login --url https://arcade.example.com, and selected that with arcade project set <project-id> if it isn’t the default.

Then ask them to deploy a server and check that it reaches Running. If the deploy fails, check the logs and the pod events in the workers namespace.

Configuration

All settings live under deployments.* in the chart values. Leave a value at its default unless you need to change it.

Core settings

ValueDefaultDescription
deployments.enabledfalseTurns deployments on.
deployments.providerkubernetesThe deployment provider. kubernetes runs deployed servers in your cluster.
deployments.mutationsAllowedtrueWhen false, the Arcade Engine blocks creating and updating deployments.
deployments.domain""Root domain for deployments, passed to the Arcade Engine. Falls back to gateway.hostname, then ingress.hostname. The chart refuses to render if none of them is set. In the platform’s own cluster, the Arcade Engine reaches deployed servers on their in-cluster Service address.
deployments.timeout5mHow long a deploy may take to become healthy before it’s marked failed.
deployments.autoRolltrueWhen true, the reconciler moves deployed servers onto a new runner image one at a time (see Upgrades). When false, the move is deferred and deployed servers stay on the runner image they already run until they’re redeployed. A redeploy picks up the configured runner image.
deployments.reconciler.enabledtrueRuns the deployment reconciler. It moves deployed servers onto a new runner image and removes NetworkPolicies left behind by removed servers.
deployments.readinessProberolloutHow the Arcade Engine decides a deployed server is ready. rollout waits for the Kubernetes rollout to finish, which needs the pod to accept TCP connections on the server’s port. Any other value, such as health, also waits until the server’s /worker/health endpoint answers HTTP 200 to the Arcade Engine.

Artifact store

ValueDefaultDescription
deployments.artifacts.url""Bucket URL for release bundles, for example s3://bundles?region=us-east-1, gs://bundles, or azblob://bundles. Required when deployments are on.

Deployed servers’ pods get no bucket credentials: their ServiceAccount mounts no token, and their NetworkPolicy blocks the cloud metadata endpoint. Instead, the gives each pod a presigned download URL for its bundle when it can sign one:

StoreSigns whenOtherwise
S3Any credential. With STS session credentials, the URL expires with the session.The pod falls back to the store URL, which it has no credentials for.
GCSA key file is configured, or the Arcade Engine’s service account holds iam.serviceAccounts.signBlob.The pod falls back to the store URL, which it has no credentials for.
Azure BlobA shared key is configured (AZURE_STORAGE_KEY).Workload and managed identities can’t sign, so the pod falls back to the store URL, which it has no credentials for.

A signing failure doesn’t fail the deploy, but when no valid signed URL remains, the pod can’t fetch its bundle. With Azure Workload Identity or GKE Workload Identity, make sure signing works as described above.

Runner image and registry

ValueDefaultDescription
deployments.image.repositoryarcadedev/runnerRunner image that deployed servers run in.
deployments.image.tag""Runner image tag. Empty uses the chart’s appVersion.
deployments.registry.server""Registry host to pull the runner image from. When set, it’s prefixed to deployments.image.repository.
deployments.registry.imagePullSecrets[]Names of image pull Secrets for deployed servers’ pods. When dockerRegistry.enabled is true and the chart creates that Secret, the chart adds it to this list.

For a private mirror, push arcadedev/runner to your registry and point the chart at it:

YAML
values.yaml
deployments: registry: server: registry.example.internal imagePullSecrets: - my-registry-pull-secret

Sizing

Every deployed server gets the same size.

ValueDefaultDescription
deployments.sizing.instances1Instances per deployed server. Keep it at 1: Arcade Deploy runs one serving instance per server, without high availability or autoscaling. A higher value makes the Arcade Engine run that many pods per server.
deployments.sizing.cpuLimit1.0CPU limit per deployed server, in cores.
deployments.sizing.memoryLimit512Memory limit per deployed server, in mebibytes (an integer, not a quantity string).
deployments.runtime.className""RuntimeClass for deployed servers, for example a sandboxed runtime. Empty uses the cluster default.
deployments.runtime.nodeSelector{}Node selector for deployed servers’ pods.
deployments.runtime.tolerations[]Tolerations for deployed servers’ pods.

With the chart, each pod’s requests equal its limits, so pods run in the Guaranteed QoS class. Each pod also has a fixed ephemeral-storage request and limit of 6Gi, which covers a 5Gi volume for the bundle and its virtual environment.

Workers namespace

ValueDefaultDescription
deployments.namespacearcade-workersNamespace deployed servers run in.
deployments.createNamespacetrueCreates the namespace with the restricted Pod Security Standard (enforce, audit, and warn) and a default-deny NetworkPolicy. Helm keeps both on uninstall.
deployments.resourceQuota.enabledtrueRenders a ResourceQuota named <release>-workers-quota from deployments.resourceQuota.hard.
deployments.resourceQuota.hardrequests.cpu: 10, requests.memory: 20Gi, limits.cpu: 20, limits.memory: 40Gi, pods: 50Hard limits for that quota.
deployments.limitRange.enabledtrueRenders a LimitRange for containers in the namespace.
deployments.limitRange.defaultcpu: 1, memory: 512MiDefault container limits.
deployments.limitRange.defaultRequestcpu: 250m, memory: 256MiDefault container requests.

To use a namespace you manage yourself, set deployments.createNamespace: false. The chart then adds no default-deny NetworkPolicy, unless you also set deployments.network.defaultDenyExistingNamespace: true. Only do that when the namespace holds nothing but deployed servers.

When the reconciler is on, the also applies its own ResourceQuota, arcade-deployments-budget, in the workers namespace. It limits pods, limits.cpu, and limits.memory, and the reconciler re-applies it on every pass, so manual edits to it don’t last. A pod has to fit every quota in the namespace.

Size deployments.resourceQuota.hard for the number of servers you expect, at deployments.sizing.cpuLimit and deployments.sizing.memoryLimit each, and make sure your nodes can hold them. If deploys stop creating pods while the chart quota still has room, check arcade-deployments-budget with kubectl -n arcade-workers get resourcequota.

If the chart already rendered its quota, only a helm upgrade with deployments.resourceQuota.enabled: false removes it. The ’s Role can’t delete ResourceQuotas.

Network

ValueDefaultDescription
deployments.egressAllowlist[]CIDRs deployed servers may reach in addition to the public internet, for example an internal API.
deployments.network.clusterCIDRs[]Your cluster’s pod and service CIDRs. Deployed servers can’t reach them. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, and 169.254.0.0/16 stay unreachable as well. Ranges in deployments.egressAllowlist are reachable, and so is DNS (TCP and UDP port 53) to the resolvers below.
deployments.network.dns.modeclustercluster uses cluster DNS. external replaces pod DNS with deployments.network.dns.nameservers and removes every cluster-DNS exception.
deployments.network.dns.nameservers[]Public IPv4 nameservers. Required in external mode.
deployments.network.dns.namespacekube-systemNamespace of the cluster DNS pods (cluster mode only).
deployments.network.dns.podSelector{}Labels of the cluster DNS pods, for example k8s-app: kube-dns (cluster mode only). Empty allows every pod in that namespace.
deployments.network.dns.cidrs[]Extra resolver IP ranges, such as a NodeLocal DNS /32. These allow TCP and UDP port 53 only (cluster mode only).

external DNS mode also needs a bundle endpoint the pod can reach without cluster DNS. Existing deployed servers pick up a DNS mode change only after a redeploy.

Deploy a server

Developers use the same CLI as on Arcade Cloud. Each developer needs an and a in your installation first. They sign in to your installation once, select the project if needed, then deploy from the project directory:

Terminal
arcade login --url https://arcade.example.com arcade deploy -e src/my_server/server.py

arcade login --url signs the developer in and saves a for your installation, and arcade deploy targets that context with no --host or other target argument. arcade project set <project-id> switches the the context selects. An arcade deploy without a saved context exits with an error that names the arcade login command to run first, and uploads nothing, unless ARCADE_URL and ARCADE_API_KEY are set (see the CI deploys below). See Arcade Deploy for the full developer guide and the CLI cheat sheet for every flag.

  • Deploy servers built with the Arcade MCP framework for Python. The refuses a bundle without a Python definition or with a missing entrypoint.
  • The refuses a changed redeploy while the previous one is still rolling out.
  • The also refuses a bundle over its 16 MiB limit or one that unpacks past 256 MiB, and discards a virtual environment that the bundle ships. The 16 MiB limit caps the upload request, with a little headroom for encoding. Both limits are built into the Arcade Engine; no chart or configuration value changes them.
  • A redeploy that changes the server’s code or secrets replaces the release under the same name and gateway URL. A redeploy of a running release that changes nothing is a no-op. Deploying again after a failed release rolls it out again, even if nothing changed.

For non-interactive CI deploys, set ARCADE_URL and ARCADE_API_KEY (a ) in the CI environment, then run arcade deploy. With both set, the CLI skips the saved and browser sign-in and uses ARCADE_URL directly as the ’s URL, without discovery. This needs arcade-mcp 1.16.0 or later.

Check status and logs

Dashboard and CLI

The Servers page in the dashboard lists deployed servers. A removed server disappears from the list. The tracks each deployed server in one of these states, and arcade deploy waits for a final one:

StateMeaning
pendingA new server is rolling out. The dashboard shows Deploying.
updatingA redeploy is rolling out. The Arcade Engine refuses another redeploy of the server until it finishes. The dashboard shows Updating.
runningThe current release is ready and serving.
degradedNo instance of the current release is serving, for example because it keeps crashing, or a rollout failed while the previous instance still serves. The status message gives the reason.
failedThe release didn’t become ready, because it failed to start or deployments.timeout ran out, and nothing is serving it.

From the CLI:

Terminal
arcade server list arcade server get <name> arcade server logs <name> -f

Logs cover the current release. After a crash, they also include the immediately previous instance. Arcade doesn’t keep logs beyond that, so ship pod logs to your own logging stack if you need them longer.

kubectl

Deployed servers are ordinary Kubernetes objects in the workers namespace, so your existing tooling works:

Terminal
kubectl -n arcade-workers get deployments,pods,services kubectl -n arcade-workers get networkpolicies -l app.kubernetes.io/managed-by=arcade-engine kubectl -n arcade-workers get resourcequota

Upgrades

Upgrade the chart with helm upgrade as described in Self-host with Helm. Keep these points in mind:

  • Runner image. With deployments.image.tag empty, the runner tag follows the chart’s appVersion. Pin deployments.image.tag to control which runner tag the configures.
  • Private registry. Mirror the new runner tag into your registry before you upgrade.
  • Namespace. With deployments.createNamespace: true, Helm keeps the workers namespace and its default-deny NetworkPolicy on uninstall, so deployed servers’ objects aren’t deleted with the release.

Moving deployed servers onto a new runner image

When an upgrade changes the runner image tag, the reconciler moves running deployed servers onto it one at a time. This needs deployments.reconciler.enabled: true.

  • On each pass (every 30 seconds), the reconciler starts at most one move, and only when no other move is still rolling out.
  • If a server can’t start on the new runner image, the rolls it back to the previous one and marks it degraded with the message “could not start on the upgraded runtime; the previous runtime is still serving”. If no instance of the server is serving when it’s marked degraded, the reason is “release artifact unavailable” instead. The move then halts: no other server moves onto that image.
  • Set deployments.autoRoll: false to defer the move. Deployed servers then stay on the runner image they already run until they’re redeployed.

Troubleshooting

SymptomCause and fix
helm upgrade fails with deployments.artifacts.url must name an S3-compatible artifact store when deployments.enabled is trueSet deployments.artifacts.url.
helm upgrade fails with deployments.domain (or gateway.hostname / ingress.hostname) must be set when deployments.enabled is trueSet deployments.domain, gateway.hostname, or ingress.hostname.
helm upgrade fails with engine.extraConfig sets deployments:The chart owns the deployments: block in the Arcade Engine’s configuration. Use deployments.* chart values instead of engine.extraConfig.
The Arcade Engine doesn’t start and names an unsupported schemeUse an s3://, gs://, or azblob:// URL in deployments.artifacts.url.
A deploy fails because the server never became healthyThe pod started but didn’t answer its health check within deployments.timeout. Read the server’s logs, and check that it binds the port the platform probes. If many servers fail at once, suspect the runner image. If start-up is just slow, raise deployments.timeout.
Deploys stop creating podsA ResourceQuota in the workers namespace is full. Run kubectl -n arcade-workers get resourcequota to see which one. For <release>-workers-quota, raise deployments.resourceQuota.hard and make sure your nodes can hold the extra servers. If arcade-deployments-budget is full, the Arcade Engine’s own cap is reached; remove deployed servers you no longer need.
A deployed server can’t reach an internal serviceAdd the service’s CIDR to deployments.egressAllowlist.
NetworkPolicies stay behind after you remove serversWith deployments.reconciler.enabled: false, nothing removes them. Delete NetworkPolicies labeled app.kubernetes.io/managed-by=arcade-engine by hand, or turn the reconciler back on.

Next steps

Last updated on