Building a Multi-Tenant Control-Plane for Ephemeral Agent Workloads

"Give every team a way to let their agents run code safely" sounds like an infrastructure request. In practice it decomposes into three separate, reusable problems — and building the wrong one first is the expensive mistake.

Three problems that get conflated into one ticket

A platform team asked to support agent code execution across the org usually gets a single request that's actually three distinct concerns wearing a trenchcoat: who's allowed to run what and how much of it (accounts, auth, fair-use limits), how a pod actually gets created and torn down fast enough that a user isn't staring at a ten-second cold start (lifecycle and warm-pool management), and what a single running pod is actually allowed to do (the isolation boundary itself). Building all three as one tangled service is how you end up rewriting the whole thing when the second team asks for a slightly different quota policy.

The rest of this post is a walk through how boxxkite (v0.2.2, Apache-2.0) draws those three lines, and — more usefully — which parts of that design are generic enough to copy versus which parts are load-bearing details you have to get right for your own cluster. The concrete claims here are grounded in the shipped source: the control-plane's UsagePolicy layer, the WarmPoolManager, the per-pod auth/TLS modules, and the reaper.

Layer one: the control-plane decides, and knows nothing about pods

The control-plane's entire job is deciding whether a request should be allowed to proceed — API key validation, per-account fair-use enforcement, and routing. It has no opinion about Kubernetes at all. In boxxkite that separation is literal: the fair-use rules live in a UsagePolicy wrapper, deliberately kept out of SandboxManager, which "only knows pods and sessions." That is what lets a fair-use policy change — a new tier, a different sandbox-hours cap — happen without anyone touching pod-provisioning code, and it's what lets the same control-plane sit in front of a self-hosted cluster or a hosted one without a rewrite.

The enforcement itself is not one check, it's three, and the order matters. On a create-session request the policy layer counts the globalactive-session total first (the node-capacity ceiling — enough accounts each sitting under their own small cap can still collectively exhaust the cluster), then the account's own concurrent sessions, then the account's cumulative sandbox-hours this calendar month. Only if all three pass does it reserve the slot and provision anything.

There's a subtle concurrency bug lurking in a naive version of this: two requests both read "account is one under its cap," both decide they're fine, and both provision — busting the cap by one. boxxkite closes that by running the count-check-and-reserve step inside a critical section that inserts a placeholder sandbox_sessions row (with pod_name=None) to claim the slot before the real, slow SandboxManager.create_session() call. Crucially, the critical section is exitedbefore that K8s round trip — which can take tens of seconds — so one stuck pod-create doesn't serialize sandbox creation for every other account. Only the cheap check-then-reserve is exclusive.

This is also the layer where the single-replica-versus-many distinction bites, and it's the kind of thing that passes every local test and then quietly misbehaves in production. Both the rate limiter and the usage lock ship with an in-memory default that is completely correct for one control-plane replica — and silently enforces limit × replica_countonce you scale out, because each replica keeps its own independent copy of the state. boxxkite documents this explicitly and gives you a shared Postgres backend for each, reusing the database you already run rather than adding Redis. It is not auto-detected, because a single process has no way to know how many replicas exist — so it's on the operator to set it.

# Fair-use limits — set from your own capacity plan, not these placeholders.
# Enforced by the control-plane's UsagePolicy layer, never inside SandboxManager.
BOXXKITE_GLOBAL_MAX_CONCURRENT_SANDBOXES=<cluster-wide ceiling>
BOXXKITE_MAX_CONCURRENT_SANDBOXES=<per account>
BOXXKITE_FREE_MONTHLY_SANDBOX_HOURS=<per account, per calendar month>
BOXXKITE_MAX_SESSION_MINUTES=<hard stop enforced by the reaper>
BOXXKITE_SESSION_REAPER_INTERVAL_SECONDS=<how often the reaper scans>

# Multi-replica correctness: the in-memory defaults enforce limit * replicas.
# Switch to the shared Postgres backends when you run more than one replica.
BOXXKITE_RATE_LIMIT_BACKEND=postgres
BOXXKITE_USAGE_LOCK_BACKEND=postgres

Layer two: the warm pool is how "fast" stops fighting "isolated"

The naive version of pod-per-session isolation is genuinely correct but slow: schedule a new pod, wait for image pull and container start, then hand it to the user. A warm-pool manager breaks that tradeoff by keeping a small number of pre-provisioned, not-yet-assigned pods on hand, so "create a session" usually means "claim an already-running pod" instead of "wait for the scheduler." The interesting design decision in boxxkite's WarmPoolManager is that it keeps no in-memory pod state at all— Kubernetes labels are the single source of truth for whether a pod is warm or claimed. That's what makes the whole thing survive a control-plane restart and lets a second process claim pods a first one created without any shared cache.

Because labels are the source of truth, "claim a warm pod" is a compare-and-swap, not a read followed by a write. The manager patches the pod's pool label fromwarm to claimedconditionally on the pod'sresourceVersionand labels still matching what it read — so if two claimants race for the same pod, exactly one patch succeeds and the loser transparently moves on to the next candidate. There's an optional fast path that pops a pod off an in-process ready index to skip a per-request list call, but it is strictly an optimization: on any miss (stale index entry, already-claimed, moved-on resourceVersion) it falls back to the list-based claim, byte-identical to the slow path.

Two edge cases separate a toy warm pool from one you'd trust in production. The first is size classes: a small and a large sandbox need different resource requests, so the pool is really several sub-pools keyed by size, and a claim only ever pulls from the matching one. The second is stale pods. A warm pod isn't free of a clock — every sandbox pod carries a KubernetesactiveDeadlineSecondsbackstop (24h by default) that hard-kills it regardless of activity. Claim a pod that's 23h59m old and the user's first tool call fails seconds later. boxxkite guards this with an explicit age gate (compute_max_claimable_age_seconds) shared by both the claim path and the pool scan: a pod within a configurable buffer of its deadline is skipped, clamped so the buffer can never eat the whole deadline and leave zero usable window.

This is also the layer worth actually load-testing before rollout — pool size versus average session duration versus how fast the pool refills determines whether your p99 session-start latency is fine or embarrassing. boxxkite ships an opt-in adaptive sizing mode (a rolling claims-per-second signal driving each sub-pool's target) that is off by default and, per its own module docstring, has notbeen validated against real production claim-rate logs — only synthetic sequences. That honesty is the point: even the project that wrote the mechanism won't claim its default constants are optimal for your traffic. When it is enabled, your configured sizes become a floor and the pool's overall budget a ceiling, so it can never scale to zero or without bound. The specific numbers are yours to find.

Layer three: isolation is a property of the pod spec, not the application code

Non-root, all Linux capabilities dropped, a read-only root filesystem, a default-denyNetworkPolicy, seccompProfile: RuntimeDefault on both containers, automountServiceAccountToken: falseso the sandbox can't reach the Kubernetes API, and base images pinned by digest rather than a mutable :latest— none of this lives in a service's business logic. It lives in the pod template and the cluster's RBAC and NetworkPolicy manifests. That's the whole payoff of putting isolation in the spec: a platform team can audit and change the posture for every tenant at once, in one place, instead of trusting every product team to configure their own pod specs correctly.

The auth boundary a NetworkPolicy alone doesn't give you

Here's the detail that separates "we set a NetworkPolicy" from actual defense in depth, and it is the part most first designs get wrong. The sidecar — the process the manager talks to over HTTP to run commands and manipulate files inside the pod — historically had no authentication of its own, relying entirely on network isolation to keep it unreachable. That assumption doesn't hold universally: NetworkPolicy enforcementis CNI-dependent, and several common managed setups (GKE Autopilot without Dataplane V2 explicitly enabled, EKS's default VPC CNI) don't enforce it at all without extra configuration. Even where it is enforced, a too-broad egress rule on the sandbox (an allow-all-443 for object storage, say) also governs what can reach the sidecar's own port, because both containers in a pod share one network namespace.

boxxkite's answer is a second, independent layer: a secret generated per pod at creation time — never a static repo-wide value — stored in a per-pod Kubernetes Secret and injected into the sidecar viasecretKeyRef (never a literal env value), sent back on every call asX-Sidecar-Auth-Token. The manager's RBAC for those Secrets is deliberatelyget/create/delete only — notlist/watch— so a compromised manager can't enumerate every live tenant's token. On top of that, the transport is per-pod self-signed TLS pinned by exact certificate identity (the SSH known_hoststrust model, minus the first-use ambiguity, since the manager generated the cert itself moments earlier) — so the token and every command body aren't crossing the pod network in cleartext for a compromised co-located pod or node to read.

Pod recycling is where cross-tenant leakage actually hides

Warm pods are an efficiency win, but they introduce a failure mode a pod-per-session design doesn't have: a pod that served one tenant can be recycled and handed to the next. If anything from the first tenant's session survives — a background process still running, or even its buffered output sitting in the sidecar — that's a cross-tenant leak, and it's explicitly in boxxkite's security scope. The mitigation is a_kill_all_processes() invariant enforced in two places on purpose: manager-side before a pod is recycled, and again sidecar-side on every /configurecall as a pod is reassigned. Two independent enforcement points, so a bug in either path still can't let a process or its output cross a tenant boundary. It's the kind of invariant that looks like belt-and-suspenders until you realize the whole warm-pool optimization is unsafe without it.

The cleanup problem that gets skipped in the first design

Ephemeral workloads create an ongoing garbage-collection problem that's easy to miss in a first design: a session whose caller crashed, disconnected, or simply forgot to call "destroy" still has a pod consuming cluster capacity — and a per-account concurrent slot — indefinitely. Without a server-side reaper,BOXXKITE_MAX_SESSION_MINUTES would be a number the API returns asexpires_atwith nothing behind it. boxxkite runs the reaper as an asyncio task off the control-plane's lifespan, scanning on a fixed interval for active sessions past the cutoff and tearing them down.

A nice detail worth stealing: the reaper doesn't call Kubernetes directly, it goes through the sameUsagePolicy.destroy_session() path a normal DELETE takes. That means thesandbox.destroyed webhook and the usage bookkeeping fire once, from one call site, whether teardown was user-initiated or reaped — instead of two divergent code paths that drift apart over time. The same layer also reaps the public demo playground on its own much shorter cutoff, because a demo pod already killed by its shorter activeDeadlineSeconds would otherwise hold a concurrency slot until the far longer global cutoff elapsed.

What to verify before you trust any of this

A design being correct on paper and a deployment being correct are different claims, and the gap is where the incidents live. Three verification steps are worth building in from the start. First, load-test the claim pathagainst your own session-duration distribution, not someone else's — the warm-pool sizing math is entirely workload-specific and the shipped adaptive defaults are, by the project's own admission, unvalidated against real traffic. Second, if you run more than one control-plane replica, prove the Postgres backends are actually selected — the in-memory defaults will pass every functional test and then enforce your limits times your replica count in production. Third, verify the isolation posture directly: confirm your CNI actually enforces the NetworkPolicy (it may not), and that the sandbox can't reach cloud metadata (IMDS) or the Kubernetes API.