llm-d project contributors announced the release of llm-d 1.0, a major milestone for the open source project designed to help organizations deploy and scale large language model inference more efficiently across modern infrastructure. Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, and now governed by the Cloud Native Computing Foundation (CNCF), llm-d plays a key role in the open foundation for enterprises deploying AI at scale in production.
As organizations race to put AI into production, many are discovering that running AI at scale can be as challenging and expensive as building the models themselves. Organizations need a way to deliver responsive experiences for users, support agentic workloads, and maximize utilization of increasingly scarce compute resources. llm-d 1.0 is built to address these challenges through intelligent distributed inference capabilities to facilitate improvements in performance, increases in throughput and help organizations gain greater control over the economics of AI serving.
Turning Innovation into Production Reality
With the rise of agentic AI, new types of workloads are placing a different kind of stress on infrastructure systems. Agentic workloads continually access and reuse vast amounts of context while supporting parallel agents that can generate unpredictable bursts of activity. Managing that context often consumes more resources than generating the final output itself, making latency, throughput, memory efficiency and infrastructure costs critical factors in realizing AI ROI. llm-d already delivers hardware heterogeneity and advanced inference capabilities out of the box for complex infrastructure environments, as well as prefix-cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch and async inference across multiple model servers, including vLLM and SGLang.
Also Read: RapidCanvas Raises $29M to Bring Enterprise-Grade AI to All
llm-d 1.0 is engineered to bring that performance to the realities of Day 2 operations and the ongoing work of running inference in production: serving multiple tenants, balancing competing priorities, providing with zero downtime features and supporting emerging agentic workloads, all within one suite of co-designed components that compose into a platform. New capabilities in 1.0 include:
- Multi-tenant quality of service – Priority and fairness-aware flow control helps latency-sensitive requests, and bursty agent workloads share a fleet without starving one another.
- Zero-downtime operations – Router high availability with hot standby and rolling updates keep traffic flowing through upgrades and planned maintenance. Management of prefill/decode replicas as one logical unit, with coordinated rollouts and lockstep upgrades making disaggregated serving practical in production.
- A stable foundation to build on – A v1 configuration API and a hardened request path, validated across clouds and accelerators, give adopters a release they can standardize on.
- Agent-aware serving – Session-aware routing keeps multi-turn conversations close to their cached context, while turn-priority scheduling and native Responses API state support help agents reuse context across turns and reduce recomputation.
The result is infrastructure that can support diverse AI workloads, ranging from latency-sensitive user interactions to large-scale autonomous agent workflows, without forcing organizations to choose between responsiveness and efficiency.
Growing Ecosystem Momentum
llm-d is already being used across a growing ecosystem of model developers, cloud providers, infrastructure vendors and enterprise users. Today, the project includes more than 500 contributors from over 25 organizations collaborating on the future of distributed AI serving and helping build common foundations across the AI infrastructure stack.
Core users including organizations running large-scale distributed inference, frontier model developers optimizing inference stacks and cloud providers serving multi-tenant AI endpoints have all seen measurable benefit from deploying llm-d.
In one recent use case, open frontier models and llm-d enabled IBM Research to explore quantum error correction hardware at 11.5 times lower inference cost than an all-proprietary configuration. Researchers reported a prefix-cache hit rate of approximately 98% and a blended serving cost of about $0.11 per million tokens.
In a separate recent demonstration using benchmark workloads, researchers showed that llm-d could efficiently serve large mixture-of-experts models for agentic workloads. Researchers used llm-d to deploy GLM 5.2, an approximately 753-billion-parameter open-weight model, on widely deployed NVIDIA GPUs, demonstrating how enterprises can use existing GPU fleets to serve frontier-class AI at scale. On a workload with hundreds of agentic sessions running in parallel, the deployment delivered over 6.6 million output tokens per minute at peak on agentic benchmarks, serving up to 3,000 concurrent coding agents with zero preemptions. At current cloud rental rates, this deployment led to costs that would have been 5 to 10 times less per token than equivalent commercial API pricing, with the largest savings on the input-heavy traffic patterns typical of agentic workloads, highlighting how llm-d can save resources and costs for enterprises deploying AI in production at scale.
Through its CNCF governance model and broad industry participation, llm-d aims to accelerate innovation while providing organizations with an open alternative for managing increasingly complex AI infrastructure. Looking to the future, the community will continue adapting to new hardware and models as they emerge, evolving the software stack to achieve peak performance across new architectures. To learn more, read the announcement blogs from IBM and Red Hat.
Source: IBM