modelsphere

ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuously optimized.

The models swiss can deploy, and how to serve each one: engine, image, flags, GPUs and tuned variants

Deploy control plane for the ModelSphere sglang and vllm charts, driven by the model catalog

Kubernetes operator that keeps routing-layer config (OpenResty / cache-aware-router / monitoring) in sync with discovered model backends

Kubernetes operator that autoscales LLM inference workloads on LLM-specific signals — KV-cache utilization, queue depth, TPM — rather than CPU and memory

ModelSphere community portal — identity (users/roles/login) + federation gateway to swiss and other backends

Helm charts for the ModelSphere LLM inference stack (sglang, vllm, cart, rdma-injector)

Default community health files and organization profile for ModelSphere

HTTP API for reading and changing the SLO of an LLM inference service, backed by LLMSLORequirement CRs

Hang-detection sidecar for LLM inference engines (vLLM / sglang)

Tunes LLM serving configurations overnight on your own GPUs: search spaces, pluggable search policies, benchmarked by LLMBench, winners promoted as merge requests

Search policies for LLM AutoTune: the policy SDK, a reference random search, and a chaos policy

OpenResty-based session-affinity router for LLM inference backends

An open benchmark platform for LLM serving endpoints: throughput sweeps, functional acceptance, real-traffic replay, scored leaderboards

Turns SLO requirements and live signals into replica decisions for LLM inference services

Cache-aware router for LLM inference backends: routes each request to the replica that already holds the longest matching prompt prefix

Continuation gateway: when a streaming completion stalls or drops mid-generation, resumes it once from what the client already received, so the client sees one continuous, complete stream