Scale idle LLMs to zero.Wake them on demand.

Noctaya is a minimal Kubernetes control plane for bursty and long-tail models. It keeps one stable OpenAI-compatible endpoint while releasing accelerators between requests.

Get started See how it works

Kubernetes-native API NVIDIA and Ascend Apache-2.0

llmservice / qwen-longtailendpoint online

PhaseScaledToZeroGateway1 / 1 readyBackend0 / 2 active

next requestactivation lease

idle0accelerators used

demand

serve1..Nready backends

quiet

release0accelerators used

Gatewayadmit + holdKEDAscale backendModelforward when ready

$kubectl get llmservice qwen-longtail -w0 → 1..N → 0

Physical validation recorded for

NVIDIA A10Atlas 300I DuoAtlas 300I ProAscend 910B3

Evidence

The cold path is the product

Zero is one state. Recovery is a pipeline.

Scheduling, image pull, weight load, and readiness each have a different tail. Noctaya keeps that entire path bounded, observable, and safe for the gateway.

Schedule

Place the backend on a node with the requested accelerator.

Pull

Fetch runtime layers when the selected node has no warm image.

Load

Mount the model cache, load weights, and initialize the runtime.

Ready

Release admitted traffic only after model-aware probes succeed.

Preserved activation

An activation lease keeps demand visible while the backend starts, even if the first client disconnects.

Bounded admission

The gateway returns a clear 429 before queued connections can become the next failure.

Streaming-safe waits

SSE heartbeats protect long cold starts, while reject mode gives clients an explicit retry path.

Graceful return to zero

Readiness gates new traffic and drain hooks protect in-flight streams during scale-down.

Small by design

Use Kubernetes as the contract.

Noctaya translates portable serving intent into the workloads and lifecycle resources your cluster already understands. It does not replace your inference engine, device plugin, scheduler, or monitoring stack.

Application

Serving intentModel, runtime selector, resources, cache, scaling, and endpoint behavior.

Noctaya

Model lifecycleReconciliation, cold activation, admission, readiness, and graceful drain.

Cluster

InfrastructureInference engines, device plugins, schedulers, KEDA, and monitoring.

Read the architecture guide

Namespaced intentLLMService+Reusable profileInferenceRuntime

reconcile

Noctaya operatorKubernetes lifecycle translation

Always onGatewayScale 0..NModel backendOptionalCache + prewarmRendered APIScaledObject

KEDArequired · installed independently

Composable, not all-encompassing

Fit Noctaya beside the stack you already operate.

Components keep clear ownership. KEDA scales. Runtime images infer. Device plugins expose accelerators. Schedulers place workloads. Noctaya coordinates the per-model lifecycle.

Required peerKEDAInstalled and managed independently

RuntimevLLM + vendor pluginsExisting inference images stay in place

Cluster-ownedDevice plugins + schedulersIncluding optional Volcano placement

On demandPrometheus + GrafanaObservability remains outside reconciliation

Evidence before claims

Validation tied to recorded stacks.

Hardware reports name the device, topology, software versions, commands, and observed lifecycle. Rendering tests never stand in for a physical result.

NVIDIA A10 report Ascend 310P report Ascend 910B3 report

Start with one model

Build the complete 0 → 1 → N → 0 lifecycle.

Install Noctaya and KEDA independently, select a hardware profile, and follow the backend from idle through inference and back to zero.

Open Getting Started Contribute