Driven by the surge in applications built to utilize large language models (LLMs), the open source AI gateway space is heating up. In response, engineers from Bloomberg and Tetrate have partnered to develop an innovative, community-led set of core AI gateway features for enterprise AI integration. This effort will expand the capabilities of the CNCF’s Envoy Gateway project, one of the Kubernetes Gateway API implementations.
As the emerging open standard for handling Kubernetes ingress traffic, Envoy Gateway is designed for at-scale operation and is extensible, making it a solid choice to underpin this new set of features, as well as future innovation in the AI API gateway space. Furthermore, Envoy Gateway is a community-led open source project with no commercially licensed features, and where community members drive decisions about upstream feature development.
This sets it apart from both existing vendor-led open source AI gateway offerings and fully proprietary, commercial AI gateway solutions that have sought to address this problem. Both of these approaches can create greater complexity for some enterprises and hamper innovation, which is why the Envoy community is developing an option without vendor lock-in or features that can only be accessed through additional, paid enterprise licenses.
“Historically, when shared problems arise in the software industry, the open source community rallies to solve them, accelerating innovation,” said Varun Talwar, founder of Tetrate. “Our collaboration with Bloomberg and the CNCF aims to achieve precisely that: designing and delivering a community-led, fully open source AI gateway, powered by the leading contender to replace legacy models for Kubernetes ingress. It’s a solution the market is asking for, and we’re excited to be part of the team of maintainers and contributors creating it.”
AI Gateways enable organizations to integrate AI functionality into workflows and applications. They route requests to multiple AI service providers and models through a single reverse proxy layer (often referred to as a gateway). AI Gateways simplify AI integration by providing a single unified API layer with which developers interact, and can provide additional functionality, such as rate limiting, caching and observability.
The initial idea for this project arose when Dan Sun, engineering team lead for Bloomberg’s Cloud Native Compute Services – AI Inference team and co-founder/maintainer of the KServe project, came to the Envoy community and outlined his views of the problem space and a potential path forward for solving it. Tetrate, a major upstream contributor to the Envoy project, stepped forward to express interest in helping Sun and Bloomberg turn their vision for the Envoy AI Gateway API into reality.
“Bloomberg has more than 15 years of experience delivering value to our customers by incorporating artificial intelligence (AI) – in particular, machine learning and natural language processing – in enterprise applications,” said Steven Bower, engineering lead for Bloomberg’s Cloud Native Compute Services group. “When we looked to the community for someone to collaborate with to start building gateway features that accelerate AI integration in our products, we immediately identified the engineering team at Tetrate. Their people are leading contributors to Envoy Gateway, and they bring significant expertise to the project around handling cloud-native, scalable traffic. Beyond that, as an ‘open source first’ company, Bloomberg believes in the power and collaborative nature of the open source community to develop web scale solutions, and that important difference makes this project a valuable alternative to other ongoing efforts.”
Envoy Gateway and KServe can be used together to allow traffic routing to both self-hosted and vendor-hosted LLMs. In this case, the AI gateway sits on the top and routes open source LLM model traffic to self-hosted endpoints using KServe, and vendor-hosted model traffic is routed to AWS Bedrock or other, similar cloud-based services.
The first features to be included in Envoy AI Gateway will provide:
- application traffic management to LLM providers with high-availability routing strategies;
- LLM usage monitoring and control at the application, organization and enterprise levels, to help users manage costs; and
- a unified interface for LLM requests through which the gateway handles back-end connectivity to various LLM providers.
The open source Envoy Gateway extensions and enhancements will offer usage control for applications that are integrated with multiple LLM providers and models, robust authorization mechanisms, and intelligent fallback options to ensure continued operation even when cloud providers are unavailable or too expensive.
This open source initiative, part of the Cloud Native Computing Foundation (CNCF), is more than just a tool; it’s a strategic response to challenges enterprises face in adopting and integrating AI in their applications at scale. By laying the groundwork for scalable AI platforms, Tetrate and Bloomberg engineers are addressing the immediate needs of today’s enterprises and setting the stage for the future of AI applications within cloud-native environments.
SOURCE: PRWeb