Yilialinn commented on code in PR #2120: URL: https://github.com/apache/apisix-website/pull/2120#discussion_r4002989392
########## blog/en/blog/2025/02/24/apisix-ai-gateway-features.md: ########## @@ -4,136 +4,93 @@ keywords: - APISIX - AI gateway - LLM API - - security protection - token rate limiting - - smart traffic management + - model routing - multi-LLM proxy - - cloud-native API -description: "Learn how Apache APISIX AI Gateway handles LLM traffic with model routing, token rate limiting, retry and fallback, prompt controls, and observability." + - prompt controls + - AI response caching + - semantic routing + - AI observability +description: "Learn how Apache APISIX AI Gateway handles LLM traffic with model proxying, configured routing, caching, token limits, prompt controls, RAG, and observability." tags: [Ecosystem] image: https://static.api7.ai/uploads/2025/03/07/Qs4WrU0I_apisix-ai-gateway.webp --- ->This article will provide an in-depth look at the AI gateway features of the current and upcoming versions of APISIX. As a multifunctional API and AI gateway, Apache APISIX offers efficient and secure LLM API calls for AI applications. +The [Apache APISIX AI Gateway](/ai-gateway/) applies model proxying, configured routing, response caching, token limits, prompt controls, retrieval, and gateway-level observability to LLM traffic through open-source plugins. This article maps those capabilities to the APISIX plugins that implement them. <!--truncate--> -## Introduction: The Rise of AI Agents and the Evolution of AI Gateway +## Why AI Traffic Needs Additional Gateway Controls -In recent years, AI agents such as AutoGPT, Chatbots, and AI Assistants have seen rapid development. These applications rely heavily on API calls to large language models (LLMs), which have brought about challenges considering high concurrency, cost control, and security. +LLM requests share many requirements with ordinary API traffic, including authentication, routing, rate limiting, resilience, and observability. They also introduce provider-specific request formats, token-based consumption, long-running streamed responses, and prompt-processing requirements. -Traditional API gateways primarily serve Web APIs and microservices and are not optimized for the unique needs of AI applications. This has led to the emergence of the concept of AI gateway. An AI gateway needs to provide enhanced capabilities in the following areas: +Apache APISIX handles these concerns on the network path between an authorized application and configured model endpoints. It does not select tools, orchestrate agent workflows, evaluate answer quality, or replace application-level authorization. Those responsibilities remain in the application and AI platform layers. -- **Multi-LLM Proxy**: Support for multiple LLM providers to avoid vendor lock-in. -- **Token Rate Limiting**: Prevent API abuse and optimize cost management. -- **Security Protection**: Including prompt filtering and content moderation to ensure compliance of AI applications. -- **Smart Traffic Management**: Dynamically adjust LLM weights based on cost, latency, and stability. +## Proxy Requests to Supported Model Providers -Apache APISIX is not only an API gateway but also an AI gateway through its plugins, helping AI applications call LLM APIs more efficiently and securely. +The [`ai-proxy`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/) plugin forwards requests to documented model providers and OpenAI-compatible endpoints. It can transform supported request formats, attach provider credentials from gateway configuration, and expose a consistent application-facing endpoint. -## LLM Proxy: Efficient Management of Multiple LLM Backends +Provider compatibility depends on the selected APISIX provider type and the upstream API. Teams should verify request and response fields for each provider instead of assuming every model implements the same interface. -AI applications typically do not rely on a single LLM provider but need to dynamically select the best model based on requirements. For example: +## Configure Multi-Model Routing and Resilience -- Using OpenAI GPT-4 for general text generation and Claude for legal document processing. -- Switching between Mistral and Gemini to optimize cost and throughput. +The [`ai-proxy-multi`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/) plugin distributes requests across configured model instances. Its documented routing policies include weighted round robin and consistent hashing, with optional health checks, bounded retries, and fallback behavior. -**Apache APISIX's LLM Proxy offers the following capabilities:** +APISIX 3.18 also supports a semantic routing algorithm. Operators provide example prompts for each instance and configure an embedding service; the plugin compares the incoming prompt with those examples and selects the closest instance that clears the configured threshold. This is configured intent routing, not automatic optimization based on model cost, latency, answer quality, or business outcomes. -✅ Support for Multiple LLM Providers: Including OpenAI, DeepSeek, Claude, Mistral, Gemini, etc., to avoid vendor lock-in. +Semantic routing does not participate in health checks, retry, or the normal fallback strategy. Its designated fallback is used only when no instance clears the similarity threshold or the embedding request fails. Weighted round robin and consistent hashing continue to use their documented resilience options. -✅ LLM Weight and Priority Management: Adjust traffic distribution based on business needs. + -✅ Multi-LLM Load Balancing: Dynamically adjust LLM weights based on latency, cost, and stability. +These controls can reduce provider-specific routing logic in applications, but they do not guarantee uninterrupted service. Availability still depends on healthy upstreams, network conditions, timeouts, retry limits, and the configured fallback path. -✅ Retry and Fallback Mechanisms: Ensure business continuity if an LLM API fails. +## Enforce Token-Based Usage Limits -✅ Load Balancing Across Different Providers of the Same LLM: +LLM requests can consume very different numbers of prompt and completion tokens. The [`ai-rate-limiting`](https://apisix.apache.org/docs/apisix/plugins/ai-rate-limiting/) plugin applies limits based on token consumption rather than request count alone. -For example: +APISIX supports local and Redis-backed counters for this plugin. Operators can scope policies through gateway configuration and choose limits appropriate for their applications. The plugin enforces configured consumption boundaries; model pricing, budgets, billing, and chargeback remain external responsibilities. -- Privately deployed DeepSeek. -- Official DeepSeek API. -- DeepSeek API from Volcano Engine +## Cache Completed LLM Responses -Users can flexibly allocate traffic weights among different DeepSeek providers based on latency, stability, and price to achieve the best calling strategy. +The [`ai-cache`](https://apisix.apache.org/docs/apisix/plugins/ai-cache/) plugin works with `ai-proxy` or `ai-proxy-multi` to cache completed LLM responses in Redis. Exact matching is enabled by default. Teams can optionally add semantic matching, which requires Redis Stack with RediSearch and a configured embedding service. -These capabilities enable AI applications to adapt flexibly to different LLMs, improve reliability, and reduce API calling costs. +Streaming responses are written only after the terminal event is received. Interrupted streams are not cached, so the plugin does not replay partial responses. Cache eligibility, isolation, expiration, bypass rules, and semantic thresholds still need to be configured for the application's data and freshness requirements. - +Cache entries are scoped by Route by default, not by Consumer. If multiple consumers share a Route, enable `cache_key.include_consumer` or add a trusted tenant-identifying variable through `cache_key.include_vars` to prevent cached responses from being reused across tenants. Review Comment: Fixed in 54291bea2. The EN/ZH guidance now requires authenticated Consumer identity or another trusted server-side variable for tenant isolation, explicitly notes that unauthenticated traffic shares the Route-level cache, and warns that a client-controlled header alone is not a tenant boundary. ########## blog/en/blog/2025/02/24/apisix-ai-gateway-features.md: ########## @@ -4,136 +4,93 @@ keywords: - APISIX - AI gateway - LLM API - - security protection - token rate limiting - - smart traffic management + - model routing - multi-LLM proxy - - cloud-native API -description: "Learn how Apache APISIX AI Gateway handles LLM traffic with model routing, token rate limiting, retry and fallback, prompt controls, and observability." + - prompt controls + - AI response caching + - semantic routing + - AI observability +description: "Learn how Apache APISIX AI Gateway handles LLM traffic with model proxying, configured routing, caching, token limits, prompt controls, RAG, and observability." tags: [Ecosystem] image: https://static.api7.ai/uploads/2025/03/07/Qs4WrU0I_apisix-ai-gateway.webp --- ->This article will provide an in-depth look at the AI gateway features of the current and upcoming versions of APISIX. As a multifunctional API and AI gateway, Apache APISIX offers efficient and secure LLM API calls for AI applications. +The [Apache APISIX AI Gateway](/ai-gateway/) applies model proxying, configured routing, response caching, token limits, prompt controls, retrieval, and gateway-level observability to LLM traffic through open-source plugins. This article maps those capabilities to the APISIX plugins that implement them. <!--truncate--> -## Introduction: The Rise of AI Agents and the Evolution of AI Gateway +## Why AI Traffic Needs Additional Gateway Controls -In recent years, AI agents such as AutoGPT, Chatbots, and AI Assistants have seen rapid development. These applications rely heavily on API calls to large language models (LLMs), which have brought about challenges considering high concurrency, cost control, and security. +LLM requests share many requirements with ordinary API traffic, including authentication, routing, rate limiting, resilience, and observability. They also introduce provider-specific request formats, token-based consumption, long-running streamed responses, and prompt-processing requirements. -Traditional API gateways primarily serve Web APIs and microservices and are not optimized for the unique needs of AI applications. This has led to the emergence of the concept of AI gateway. An AI gateway needs to provide enhanced capabilities in the following areas: +Apache APISIX handles these concerns on the network path between an authorized application and configured model endpoints. It does not select tools, orchestrate agent workflows, evaluate answer quality, or replace application-level authorization. Those responsibilities remain in the application and AI platform layers. -- **Multi-LLM Proxy**: Support for multiple LLM providers to avoid vendor lock-in. -- **Token Rate Limiting**: Prevent API abuse and optimize cost management. -- **Security Protection**: Including prompt filtering and content moderation to ensure compliance of AI applications. -- **Smart Traffic Management**: Dynamically adjust LLM weights based on cost, latency, and stability. +## Proxy Requests to Supported Model Providers -Apache APISIX is not only an API gateway but also an AI gateway through its plugins, helping AI applications call LLM APIs more efficiently and securely. +The [`ai-proxy`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/) plugin forwards requests to documented model providers and OpenAI-compatible endpoints. It can transform supported request formats, attach provider credentials from gateway configuration, and expose a consistent application-facing endpoint. -## LLM Proxy: Efficient Management of Multiple LLM Backends +Provider compatibility depends on the selected APISIX provider type and the upstream API. Teams should verify request and response fields for each provider instead of assuming every model implements the same interface. -AI applications typically do not rely on a single LLM provider but need to dynamically select the best model based on requirements. For example: +## Configure Multi-Model Routing and Resilience -- Using OpenAI GPT-4 for general text generation and Claude for legal document processing. -- Switching between Mistral and Gemini to optimize cost and throughput. +The [`ai-proxy-multi`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/) plugin distributes requests across configured model instances. Its documented routing policies include weighted round robin and consistent hashing, with optional health checks, bounded retries, and fallback behavior. -**Apache APISIX's LLM Proxy offers the following capabilities:** +APISIX 3.18 also supports a semantic routing algorithm. Operators provide example prompts for each instance and configure an embedding service; the plugin compares the incoming prompt with those examples and selects the closest instance that clears the configured threshold. This is configured intent routing, not automatic optimization based on model cost, latency, answer quality, or business outcomes. -✅ Support for Multiple LLM Providers: Including OpenAI, DeepSeek, Claude, Mistral, Gemini, etc., to avoid vendor lock-in. +Semantic routing does not participate in health checks, retry, or the normal fallback strategy. Its designated fallback is used only when no instance clears the similarity threshold or the embedding request fails. Weighted round robin and consistent hashing continue to use their documented resilience options. -✅ LLM Weight and Priority Management: Adjust traffic distribution based on business needs. + -✅ Multi-LLM Load Balancing: Dynamically adjust LLM weights based on latency, cost, and stability. +These controls can reduce provider-specific routing logic in applications, but they do not guarantee uninterrupted service. Availability still depends on healthy upstreams, network conditions, timeouts, retry limits, and the configured fallback path. -✅ Retry and Fallback Mechanisms: Ensure business continuity if an LLM API fails. +## Enforce Token-Based Usage Limits -✅ Load Balancing Across Different Providers of the Same LLM: +LLM requests can consume very different numbers of prompt and completion tokens. The [`ai-rate-limiting`](https://apisix.apache.org/docs/apisix/plugins/ai-rate-limiting/) plugin applies limits based on token consumption rather than request count alone. -For example: +APISIX supports local and Redis-backed counters for this plugin. Operators can scope policies through gateway configuration and choose limits appropriate for their applications. The plugin enforces configured consumption boundaries; model pricing, budgets, billing, and chargeback remain external responsibilities. -- Privately deployed DeepSeek. -- Official DeepSeek API. -- DeepSeek API from Volcano Engine +## Cache Completed LLM Responses -Users can flexibly allocate traffic weights among different DeepSeek providers based on latency, stability, and price to achieve the best calling strategy. +The [`ai-cache`](https://apisix.apache.org/docs/apisix/plugins/ai-cache/) plugin works with `ai-proxy` or `ai-proxy-multi` to cache completed LLM responses in Redis. Exact matching is enabled by default. Teams can optionally add semantic matching, which requires Redis Stack with RediSearch and a configured embedding service. Review Comment: Fixed in 54291bea2. The English and Chinese articles now require Redis Search commands rather than Redis Stack specifically, and link to the Redis integration guide with its Redis Open Source 8.10.1 lab baseline and version-pinning guidance. ########## next/src/components/MainPages/AiGatewayPage.astro: ########## @@ -11,55 +11,119 @@ const features = [ title: t(locale, 'Multi-LLM load balancing', '多 LLM 负载均衡'), description: t( locale, - 'Route across OpenAI, DeepSeek, Claude, Mistral, Gemini, and other providers with health checks and weighted balancing.', - '通过健康检查和权重负载均衡，在 OpenAI、DeepSeek、Claude、Mistral、Gemini 等模型提供商之间路由。', + 'Use weighted round robin or consistent hashing, or configure semantic routing from prompt similarity to per-instance examples.', + '使用加权轮询或一致性哈希，或根据提示词与各实例示例的相似度配置语义路由。', ), icon: '/img/ai-gateway/load-balancing.svg', + href: '/docs/apisix/plugins/ai-proxy-multi/', }, { title: t(locale, 'Token rate limiting', 'Token 限流'), description: t( locale, - 'Control token consumption by Route, Service, Consumer, Consumer Group, or custom attributes in standalone and cluster deployments.', - '按 Route、Service、Consumer、Consumer Group 或自定义属性控制 token 消耗，支持单机与集群部署。', + 'Apply token-based limits with local or Redis-backed counters to control usage before requests reach an LLM provider.', Review Comment: Fixed in 54291bea2. The hub and feature article now state that provider-reported token usage is recorded after a response and that subsequent requests are rejected after the observed quota is consumed. They also note that a large response or concurrent requests can exceed the configured limit before later requests are rejected. ########## next/src/components/MainPages/AiGatewayPage.astro: ########## @@ -11,55 +11,119 @@ const features = [ title: t(locale, 'Multi-LLM load balancing', '多 LLM 负载均衡'), description: t( locale, - 'Route across OpenAI, DeepSeek, Claude, Mistral, Gemini, and other providers with health checks and weighted balancing.', - '通过健康检查和权重负载均衡，在 OpenAI、DeepSeek、Claude、Mistral、Gemini 等模型提供商之间路由。', + 'Use weighted round robin or consistent hashing, or configure semantic routing from prompt similarity to per-instance examples.', + '使用加权轮询或一致性哈希，或根据提示词与各实例示例的相似度配置语义路由。', ), icon: '/img/ai-gateway/load-balancing.svg', + href: '/docs/apisix/plugins/ai-proxy-multi/', }, { title: t(locale, 'Token rate limiting', 'Token 限流'), description: t( locale, - 'Control token consumption by Route, Service, Consumer, Consumer Group, or custom attributes in standalone and cluster deployments.', - '按 Route、Service、Consumer、Consumer Group 或自定义属性控制 token 消耗，支持单机与集群部署。', + 'Apply token-based limits with local or Redis-backed counters to control usage before requests reach an LLM provider.', + '使用本地或 Redis 计数器实施基于 token 的限制，在请求到达 LLM 提供商之前控制用量。', ), icon: '/img/ai-gateway/token-rate-limiting.svg', + href: '/docs/apisix/plugins/ai-rate-limiting/', }, { title: 'AI RAG', description: t( locale, - 'Connect enterprise knowledge to model requests at the gateway layer for grounded, context-aware responses.', - '在网关层为模型请求连接企业知识库，生成有依据、具备上下文的响应。', + 'Augment requests with context retrieved through the plugin\'s documented Azure OpenAI and Azure AI Search integration.', + '通过插件当前支持的 Azure OpenAI 与 Azure AI Search 集成，为请求补充检索到的上下文。', ), icon: '/img/ai-gateway/ai-rag.svg', + href: '/docs/apisix/plugins/ai-rag/', }, { title: t(locale, 'Token observability', 'Token 可观测性'), description: t( locale, - 'Track token usage through access logs and existing observability tools to control abuse and unexpected cost.', - '通过访问日志和现有可观测工具追踪 token 使用，控制滥用和意外成本。', + 'Record model, latency, token usage, and time-to-first-token summaries when AI proxy logging is enabled.', + '启用 AI 代理日志后，记录模型、延迟、token 用量和首 token 响应时间等摘要。', ), icon: '/img/ai-gateway/observability.svg', + href: '/docs/apisix/plugins/ai-proxy/', }, { title: t(locale, 'Retry and fallback', '重试与回退'), description: t( locale, - 'Use health checks, automatic retries, and fallback providers to keep AI applications available when an upstream model fails.', - '通过健康检查、自动重试和备用模型提供商，在上游模型故障时保持 AI 应用可用。', + 'Configure bounded retries and fallback strategies for selected upstream failures without changing application endpoints.', + '针对指定的上游故障配置有限次数的重试和回退策略，无需更改应用访问端点。', ), icon: '/img/ai-gateway/retry-fallback.svg', + href: '/docs/apisix/plugins/ai-proxy-multi/', }, { - title: t(locale, 'Prompt security', '提示词安全'), + title: t(locale, 'Prompt and content controls', '提示词与内容控制'), description: t( locale, - 'Apply prompt guards, decorators, templates, content moderation, logging, and auditing before traffic reaches a model.', - '在流量到达模型之前应用提示词防护、装饰、模板、内容审核、日志和审计。', + 'Apply allow and deny patterns with Prompt Guard, and compose separate plugins for templates, decoration, or moderation.', + '使用 Prompt Guard 配置允许与拒绝规则，并按需组合模板、装饰或内容审核插件。', ), icon: '/img/ai-gateway/security.svg', + href: '/docs/apisix/plugins/ai-prompt-guard/', + }, + { + title: t(locale, 'AI response caching', 'AI 响应缓存'), + description: t( + locale, + 'With AI Proxy, cache completed LLM responses using Redis-backed exact matching and optional semantic matching.', Review Comment: Fixed in 54291bea2. Both the AI response caching and Lakera Guard feature cards now explicitly name and , matching the capability table and plugin documentation. The cache card also now uses a dedicated cache icon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
