{"id":90991,"date":"2026-05-17T16:34:57","date_gmt":"2026-05-17T16:34:57","guid":{"rendered":"https:\/\/youzum.net\/meet-litellm-agent-platform-a-kubernetes-based-self-hosted-infrastructure-layer-for-isolated-agent-sandboxes-and-persistent-session-management-in-production\/"},"modified":"2026-05-17T16:34:57","modified_gmt":"2026-05-17T16:34:57","slug":"meet-litellm-agent-platform-a-kubernetes-based-self-hosted-infrastructure-layer-for-isolated-agent-sandboxes-and-persistent-session-management-in-production","status":"publish","type":"post","link":"https:\/\/youzum.net\/ja\/meet-litellm-agent-platform-a-kubernetes-based-self-hosted-infrastructure-layer-for-isolated-agent-sandboxes-and-persistent-session-management-in-production\/","title":{"rendered":"Meet LiteLLM Agent Platform: A Kubernetes-Based, Self-Hosted Infrastructure Layer for Isolated Agent Sandboxes and Persistent Session Management in Production"},"content":{"rendered":"

Running AI agents in a local script is straightforward. Running them reliably in production across teams, across restarts, with isolated environments per context is a different problem entirely. BerriAI, the company behind the LiteLLM AI Gateway, is now open-sourcing a purpose-built answer to that problem: the LiteLLM Agent Platform<\/a><\/strong>. The platform is described as a simple, self-hosted infrastructure platform for running multiple agents in production. <\/p>\n

What Problem Does it Solve?<\/strong><\/h2>\n

It helps to understand what happens when you try to scale agents beyond a single process. Agents are stateful: they carry session history, tool call results, and intermediate reasoning across turns. If the container running your agent crashes, restarts, or gets replaced during a deployment, that session state is gone unless something is explicitly managing it. At the same time, different teams often need different runtime environments, different tools, different secrets, different access scopes which means you cannot throw all agents into one shared container.<\/p>\n

The platform manages two things: per-team and per-context sandboxes, and session continuity across pod restarts and upgrades. These two capabilities are the core infrastructure primitives the platform provides.<\/p>\n

Architecture and Technical Stack<\/strong><\/h2>\n

The platform is a standalone Next.js dashboard for LiteLLM v2 managed agents, covering sessions chat, agent CRUD, and live status. The codebase is primarily TypeScript (92.8%), with Shell scripts for provisioning, a Dockerfile for containerization, and CSS for the dashboard UI.<\/p>\n

The architecture separates concerns cleanly. A web process<\/strong> runs on port 3000 and serves the Next.js dashboard. A worker process<\/strong> handles async agent tasks. Postgres<\/strong> is used as the persistent backing store, and a schema migration runs as an init container on startup \u2014 so the database is always in the correct state before the application boots.<\/p>\n

For the sandbox layer \u2014 the isolated runtime environment where agents actually execute \u2014 sandboxes run on Kubernetes via the kubernetes-sigs\/agent-sandbox<\/code> CRD. Local development uses kind. If you are not already familiar with it: kind<\/code> (Kubernetes in Docker) lets you spin up a full Kubernetes cluster locally using Docker containers as nodes, without needing a cloud provider. The agent-sandbox<\/code> CRD (Custom Resource Definition) is a Kubernetes extension from kubernetes-sigs<\/code> that the platform installs to manage the lifecycle of individual sandbox environments.<\/p>\n

The platform also includes a harness system under harnesses\/opencode<\/code>, which contains the configuration for running coding agents \u2014 such as Claude Code or OpenAI Codex \u2014 inside isolated sandboxes with a vault proxy for credential management. BerriAI team also maintains a separate litellm-agent-runtime<\/code> repository, described as a coding-agent runtime that runs inside per-session VMs provisioned by a LiteLLM proxy, generic by design, with customization happening via harness configuration or a hydrate payload.<\/p>\n

One practical detail worth noting is how environment variables are handled across sandbox containers. Anything in .env<\/code> prefixed with CONTAINER_ENV_<\/code> is injected into every sandbox container with the prefix stripped \u2014 for example, CONTAINER_ENV_GITHUB_TOKEN=ghp_...<\/code> means the container sees GITHUB_TOKEN=ghp_...<\/code> This gives teams a clean way to pass secrets into sandboxed agent sessions without modifying container images.<\/p>\n

\n
\"\"
https:\/\/github.com\/BerriAI\/litellm-agent-platform<\/figcaption><\/figure>\n<\/div>\n

Getting Started<\/strong><\/h2>\n

The prerequisites for local development are Docker Desktop, kind<\/code>, kubectl<\/code>, helm<\/code>, and a LiteLLM gateway. No cloud credentials are required to get started locally. The quickstart is two commands:<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
bin\/kind-up.sh\ndocker compose up<\/code><\/pre>\n<\/div>\n<\/div>\n
bin\/kind-up.sh<\/code> is idempotent \u2014 it provisions a kind cluster named agent-sbx<\/code>, installs the agent-sandbox controller, and loads the harness image. docker compose up<\/code> boots Postgres, runs the schema migration, and starts the web process on port 3000 along with the worker.<\/p>\n
For production deployment, the recommended path is AWS EKS for the sandbox cluster and Render for the web and worker processes. bin\/eks-up.sh<\/code> provisions the EKS cluster, and a Render Blueprint provides a one-click deployment option. <\/p>\n
Relationship to the LiteLLM Gateway<\/strong><\/h2>\n
The Agent Platform is a layer on top of the existing LiteLLM ecosystem, not a replacement for it. LiteLLM\u2019s core is a Python SDK and Proxy Server \u2014 an AI Gateway \u2014 that calls 100+ LLM APIs in OpenAI format, with cost tracking, guardrails, load balancing, and logging, supporting providers including Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, SageMaker, HuggingFace, vLLM, and NVIDIA NIM. The Agent Platform consumes a running LiteLLM gateway as a dependency and builds agent orchestration and session management infrastructure on top of it. Model routing, cost tracking, and rate limiting remain in the gateway layer. Sandbox isolation, session continuity, and the management dashboard are handled by the Agent Platform.<\/p>\n
Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n
\n
  <\/p>\n
\n
\n      \n        <\/path><\/path><\/path>\n      <\/svg>\n    <\/div>\n
\n
LiteLLM Agent Platform<\/div>\n
Self-Hosted Agent Infrastructure Guide<\/div>\n<\/div>\n
Alpha<\/div>\n<\/div>\n
  <\/p>\n
\n
<\/div>\n<\/div>\n
  <\/p>\n
\n
Overview<\/div>\n
Concepts<\/div>\n
Architecture<\/div>\n
Prerequisites<\/div>\n
Quickstart<\/div>\n
Production<\/div>\n<\/div>\n
  <\/p>\n
\n
    <\/p>\n
\n
01 \/ 06<\/div>\n
What is LiteLLM Agent Platform?<\/div>\n
BerriAI open-sourced this platform on May 8, 2026. It is a self-hosted infrastructure layer for running multiple AI agents in production, built on top of the LiteLLM AI Gateway.<\/div>\n
\n
\n
\"\ud83e\uddf1\"<\/div>\n
Self-Hosted<\/div>\n
Runs entirely on your own infrastructure. No data leaves your environment. Suited for regulated industries and teams with data residency requirements.<\/div>\n<\/div>\n
\n
\"\ud83e\udd16\"<\/div>\n
Multi-Agent<\/div>\n
Designed to run multiple agents in parallel, with full isolation between teams and contexts using per-session sandboxes.<\/div>\n<\/div>\n
\n
\"\ud83d\udd01\"<\/div>\n
Session Continuity<\/div>\n
Agent sessions persist across pod restarts and upgrades, so stateful work is not lost when containers are replaced.<\/div>\n<\/div>\n
\n
\"\u26a1\"<\/div>\n
Open Source (MIT)<\/div>\n
Fully open source under the MIT license. Repo: github.com\/BerriAI\/litellm-agent-platform. File issues and contribute directly.<\/div>\n<\/div>\n<\/div>\n
\n
Prerequisite Knowledge<\/div>\n
This guide assumes familiarity with Docker, basic command-line usage, and a general understanding of what an AI agent is (a model that calls tools and runs multi-step tasks). Kubernetes experience helps but is not required to follow along.<\/div>\n<\/div>\n<\/div>\n
    <\/p>\n
\n
02 \/ 06<\/div>\n
Key Concepts to Know First<\/div>\n
Before running the platform, understand these four building blocks. They appear throughout the setup and configuration.<\/div>\n