CRD reference
Noctaya exposes two resources in serving.noctaya.io/v1alpha1.
| Resource | Scope | Purpose |
|---|---|---|
LLMService | Namespaced | Declares a model endpoint and its resources, scaling, cache, and cold-start behavior |
InferenceRuntime | Cluster | Defines a reusable serving image and its accelerator, scheduling, health, and lifecycle integration |
The API is alpha. Kubernetes may accept reserved fields that the controller cannot yet reconcile; those fields are marked not implemented below. The Go types in api/v1alpha1 and the generated CRDs remain authoritative.
Durations use Go/Kubernetes syntax such as 10s, 2m, and 5m. Resource quantities use Kubernetes syntax such as 500m, 32Gi, and 60Gi.
LLMService
The schema requires spec.model and spec.runtime. Set exactly one model source (source or the reserved catalogRef) and exactly one runtime selection (name or selector.vendor).
Model and runtime
| Field | Type | Default / constraint | Behavior |
|---|---|---|---|
spec.model.source.uri | string | Required for reconciliation | Supports hf:// (huggingface:// alias), modelscope://, digest-pinned oci://<registry>/<repository>@sha256:<digest>, and pvc://<claim>[/<subpath>] |
spec.model.catalogRef | string | Not implemented | Reserved for model-catalog lookup |
spec.model.source.secretRef | LocalObjectReference | Optional for oci:// only | References a kubernetes.io/dockerconfigjson Secret in the service namespace |
spec.runtime.name | string | Exactly one runtime selection | Pins one cluster-scoped InferenceRuntime |
spec.runtime.selector.vendor | string array | Exactly one runtime selection; nvidia or ascend | Selects the first available vendor in order, then the highest-priority runtime |
spec.runtime.argsOverride | string array | Optional | Appends arguments after the runtime's templated arguments |
A pvc:// source mounts existing weights read-only at /models and performs no download. Use cache.strategy: None to avoid creating a separate cache PVC. An oci:// source must use an immutable SHA-256 digest and a persistent cache; Noctaya pulls its files with ORAS, rejects traversal and symbolic links, and exposes the cache path only after atomic promotion. The artifact filesystem root must contain the exact model directory expected by the runtime. Tags and s3:// are not supported.
Resources and scaling
| Field | Type | Default / constraint | Behavior |
|---|---|---|---|
spec.resources.accelerators | integer | 1; minimum 1 | Whole devices requested per backend replica |
spec.resources.cpu | quantity | Optional | CPU request per backend replica |
spec.resources.memory | quantity | Optional | Equal memory request and limit per backend replica |
spec.resources.fraction | object | Not implemented | Reserved for sub-device sharing |
spec.scaling.min | integer | 0; minimum 0 | Minimum backend replicas; 0 enables scale-to-zero |
spec.scaling.max | integer | 1; minimum 1 | Maximum backend replicas; must be at least min |
spec.scaling.metric | string | queueDepth; queueDepth or kvCacheUtil | Only queueDepth is implemented |
spec.scaling.target | integer | 10; minimum 1 | Queue-depth target per backend replica |
spec.scaling.activationTimeout | positive duration | 5m | Bounds a cold request wait or reject-mode activation lease |
spec.scaling.scaleDownStabilization | duration | 5m; whole seconds, 0s..1h | HPA stabilization and KEDA cooldown before scale-to-zero |
spec.scaling.drainTimeout | non-negative duration | 2m | Pre-stop drain time when the runtime enables preStopDrain |
spec.scaling.externalScaler.tls.serverSecretName | string | Required when tls is set | Mounts an existing Secret containing tls.crt, tls.key, and the client-verifying ca.crt into the ExternalScaler Pod |
spec.scaling.externalScaler.tls.authenticationRef.name | string | Required when tls is set | References the KEDA authentication object that supplies caCert, tlsClientCert, and tlsClientKey |
spec.scaling.externalScaler.tls.authenticationRef.kind | string | TriggerAuthentication; or ClusterTriggerAuthentication | Selects the KEDA authentication-object scope |
KEDA owns backend replicas; min and max do not control gateway replicas.
Noctaya consumes existing mTLS credentials but does not issue or rotate them. The server certificate must be valid for <llmservice>-scaler.<namespace>.svc.
Cache and endpoint
| Field | Type | Default / constraint | Behavior |
|---|---|---|---|
spec.cache.strategy | string | NodeLocalPVC | Implements NodeLocalPVC, SharedPVC, HostPath, and None; BakedImage is not implemented |
spec.cache.size | quantity | 50Gi controller default | PVC request for NodeLocalPVC or SharedPVC |
spec.cache.storageClassName | string | Cluster default | Selects the cache PVC StorageClass; SharedPVC requires ReadWriteMany support |
spec.cache.prewarm | boolean | false | Creates one download Job for a persistent hub cache; OCI sources always create a staging Job |
spec.endpoint.openAICompatible | boolean | true | Informational; the gateway currently always serves the OpenAI-compatible API |
spec.endpoint.maxQueue | integer | 100; minimum 1 | Maximum admitted requests per gateway replica; overflow returns 429 with Retry-After |
spec.endpoint.resources.requests.cpu / memory | quantity | Optional | CPU and memory requested by each gateway replica |
spec.endpoint.resources.limits.cpu / memory | quantity | Optional | CPU and memory limits for each gateway replica |
spec.endpoint.authentication.secretRef.name | string | Required when authentication is set | References a Secret in the LLMService namespace |
spec.endpoint.authentication.secretRef.key | string | Required when authentication is set | Selects the Secret data key used as the accepted Bearer token |
spec.endpoint.coldStart.mode | string | keepalive; keepalive or reject | Holds streaming requests with SSE heartbeats or returns 503 with Retry-After |
spec.endpoint.coldStart.heartbeatInterval | positive duration | 10s | Keepalive heartbeat interval during activation |
spec.imagePullSecrets | LocalObjectReference array | Optional | Applied to backend, gateway, and prewarm Pods; Secrets must be in the service namespace |
Cache PVCs and prewarm Jobs are create-once resources carrying an immutable-spec hash. A changed size, StorageClass, source, or prewarm specification fails reconciliation instead of silently reusing incompatible state. Delete the Job to retry or prewarm again; preserve needed data before replacing a PVC. SharedPVC uses ReadWriteMany; its prewarm Job writes the cache while serving containers mount a completed cache read-only. Without prewarm, serving containers retain write access.
Client authentication is opt-in. When configured, missing or invalid Authorization: Bearer <key> credentials on proxied inference routes return 401. The gateway rereads the mounted Secret file per request, so projected Secret rotation does not recreate the LLMService or gateway Pods. /healthz, /metrics, and /noctaya/queue remain unauthenticated and require network isolation.
LLMService status
| Field | Description |
|---|---|
status.phase | Summary state: Pending, Loading, Ready, ScaledToZero, or Degraded. |
status.resolvedRuntime | Name of the InferenceRuntime selected by the controller. |
status.replicas | Number of ready backend replicas. Gateway replicas are not included. |
status.endpointURL | In-cluster OpenAI-compatible base URL, ending in /v1. |
status.conditions | Kubernetes conditions. Ready reports serving availability, Degraded reports a Kubernetes-observed backend activation failure, and AutoscalingReady reports whether KEDA External Push is configured. Every condition uses the reconciled ObservedGeneration. |
Ready=False uses progress reasons such as Activating, Starting, ModelLoading, and SchedulingDelayed, or the active failure reason. Degraded=True uses stable failure reasons: ImagePullFailed, OOMKilled, CrashLoopBackOff, ContainerRestarting, or ProgressDeadlineExceeded. See Troubleshoot Noctaya for diagnosis and the distinction between these observations and the gateway's bounded activation_timeout.
InferenceRuntime
InferenceRuntime is reusable configuration, not a workload. Its controller is passive; the LLMService controller consumes it and requeues matching services when it changes.
Runtime and container
| Field | Type | Default / constraint | Behavior |
|---|---|---|---|
spec.family | string | vllm; only vllm | Serving-engine family |
spec.vendor | string | Required; nvidia or ascend | Selects the registered backend adapter |
spec.priority | integer | 0 | Higher value wins within a selected vendor; equal top values are ambiguous |
spec.container.image | string | Required | Serving image exposing an OpenAI-compatible API |
spec.container.args | string array | Optional | Templates using .Model.Path, .Service.Name, and .Service.Namespace |
spec.container.env | Kubernetes EnvVar array | Optional | Copied to the container; literal values support the same templates |
spec.container.port.name | string | http | Port name used by Services and probes |
spec.container.port.containerPort | integer | 8000; 1..65535 | Serving API port |
spec.container and spec.container.port are required objects. Runtime arguments are rendered first; LLMService.spec.runtime.argsOverride is appended.
Accelerator and scheduling
| Field | Type | Default / constraint | Behavior |
|---|---|---|---|
spec.accelerator.resourceName | string | Required | Device-plugin resource, for example nvidia.com/gpu |
spec.accelerator.sharing.supported | boolean | false; not implemented | Capability placeholder; it does not enable fractional allocation |
spec.accelerator.nodeSelector | string map | Optional | Copied to backend and prewarm Pods |
spec.accelerator.tolerations | Kubernetes Toleration array | Optional | Copied to backend and prewarm Pods |
spec.accelerator.scheduler.name | string | Kubernetes default scheduler | Sets Pod schedulerName, for example volcano |
spec.accelerator.scheduler.queue | string | Requires scheduler.name: volcano | Adds the Volcano queue annotation |
The runtime references existing drivers, device plugins, and schedulers; it does not install or manage them.
Health, lifecycle, and metrics
| Field | Default / constraint | Behavior |
|---|---|---|
spec.health.readiness | Controller default: HTTP GET /health | Gates backend traffic until the model is loaded |
spec.health.startup | Controller default: GET /health, 10-second period, 60 failures | Allows about 10 minutes for startup |
spec.health.liveness | None | Optional post-start failure detection |
spec.lifecycle.terminationGracePeriodSeconds | Optional; minimum 1 | Base Pod shutdown budget |
spec.lifecycle.preStopDrain | false | Sleeps for the service drainTimeout; requires /bin/sh in the image |
spec.metrics.path / port | /metrics / http | External scrape metadata |
spec.metrics.queueDepth | Required when metrics is set | Runtime queue metric name |
spec.metrics.kvCacheUtil / running / ttft | Optional | Additional external metric names |
When drain is enabled, Noctaya widens the termination grace period to at least drainTimeout plus 10 seconds. Runtime metric names are metadata only; autoscaling uses gateway demand.
InferenceRuntime.status.conditions is declared but not populated because the controller is passive. Operational state is reported on each consuming LLMService. See the roadmap for planned features.