kayx23 commented on code in PR #2120:
URL: https://github.com/apache/apisix-website/pull/2120#discussion_r4002837050
##########
next/src/components/MainPages/AiGatewayPage.astro:
##########
@@ -11,55 +11,119 @@ const features = [
title: t(locale, 'Multi-LLM load balancing', '多 LLM 负载均衡'),
description: t(
locale,
- 'Route across OpenAI, DeepSeek, Claude, Mistral, Gemini, and other
providers with health checks and weighted balancing.',
- '通过健康检查和权重负载均衡，在 OpenAI、DeepSeek、Claude、Mistral、Gemini 等模型提供商之间路由。',
+ 'Use weighted round robin or consistent hashing, or configure semantic
routing from prompt similarity to per-instance examples.',
+ '使用加权轮询或一致性哈希，或根据提示词与各实例示例的相似度配置语义路由。',
),
icon: '/img/ai-gateway/load-balancing.svg',
+ href: '/docs/apisix/plugins/ai-proxy-multi/',
},
{
title: t(locale, 'Token rate limiting', 'Token 限流'),
description: t(
locale,
- 'Control token consumption by Route, Service, Consumer, Consumer Group,
or custom attributes in standalone and cluster deployments.',
- '按 Route、Service、Consumer、Consumer Group 或自定义属性控制 token 消耗，支持单机与集群部署。',
+ 'Apply token-based limits with local or Redis-backed counters to control
usage before requests reach an LLM provider.',
Review Comment:
`ai-rate-limiting` is not prepaid. The plugin docs’ own example lets a first
request reach the provider, records the provider-reported token count after the
response, and only then rejects or fails over later requests. A large or
concurrent completion can still exceed the configured quota.
Please drop “before requests reach an LLM provider” / “在请求到达 LLM 提供商之前”. Say
that the plugin enforces the configured token counter after usage is observed.
##########
next/src/components/MainPages/AiGatewayPage.astro:
##########
@@ -11,55 +11,119 @@ const features = [
title: t(locale, 'Multi-LLM load balancing', '多 LLM 负载均衡'),
description: t(
locale,
- 'Route across OpenAI, DeepSeek, Claude, Mistral, Gemini, and other
providers with health checks and weighted balancing.',
- '通过健康检查和权重负载均衡，在 OpenAI、DeepSeek、Claude、Mistral、Gemini 等模型提供商之间路由。',
+ 'Use weighted round robin or consistent hashing, or configure semantic
routing from prompt similarity to per-instance examples.',
+ '使用加权轮询或一致性哈希，或根据提示词与各实例示例的相似度配置语义路由。',
),
icon: '/img/ai-gateway/load-balancing.svg',
+ href: '/docs/apisix/plugins/ai-proxy-multi/',
},
{
title: t(locale, 'Token rate limiting', 'Token 限流'),
description: t(
locale,
- 'Control token consumption by Route, Service, Consumer, Consumer Group,
or custom attributes in standalone and cluster deployments.',
- '按 Route、Service、Consumer、Consumer Group 或自定义属性控制 token 消耗，支持单机与集群部署。',
+ 'Apply token-based limits with local or Redis-backed counters to control
usage before requests reach an LLM provider.',
+ '使用本地或 Redis 计数器实施基于 token 的限制，在请求到达 LLM 提供商之前控制用量。',
),
icon: '/img/ai-gateway/token-rate-limiting.svg',
+ href: '/docs/apisix/plugins/ai-rate-limiting/',
},
{
title: 'AI RAG',
description: t(
locale,
- 'Connect enterprise knowledge to model requests at the gateway layer for
grounded, context-aware responses.',
- '在网关层为模型请求连接企业知识库，生成有依据、具备上下文的响应。',
+ 'Augment requests with context retrieved through the plugin\'s
documented Azure OpenAI and Azure AI Search integration.',
+ '通过插件当前支持的 Azure OpenAI 与 Azure AI Search 集成，为请求补充检索到的上下文。',
),
icon: '/img/ai-gateway/ai-rag.svg',
+ href: '/docs/apisix/plugins/ai-rag/',
},
{
title: t(locale, 'Token observability', 'Token 可观测性'),
description: t(
locale,
- 'Track token usage through access logs and existing observability tools
to control abuse and unexpected cost.',
- '通过访问日志和现有可观测工具追踪 token 使用，控制滥用和意外成本。',
+ 'Record model, latency, token usage, and time-to-first-token summaries
when AI proxy logging is enabled.',
+ '启用 AI 代理日志后，记录模型、延迟、token 用量和首 token 响应时间等摘要。',
),
icon: '/img/ai-gateway/observability.svg',
+ href: '/docs/apisix/plugins/ai-proxy/',
},
{
title: t(locale, 'Retry and fallback', '重试与回退'),
description: t(
locale,
- 'Use health checks, automatic retries, and fallback providers to keep AI
applications available when an upstream model fails.',
- '通过健康检查、自动重试和备用模型提供商，在上游模型故障时保持 AI 应用可用。',
+ 'Configure bounded retries and fallback strategies for selected upstream
failures without changing application endpoints.',
+ '针对指定的上游故障配置有限次数的重试和回退策略，无需更改应用访问端点。',
),
icon: '/img/ai-gateway/retry-fallback.svg',
+ href: '/docs/apisix/plugins/ai-proxy-multi/',
},
{
- title: t(locale, 'Prompt security', '提示词安全'),
+ title: t(locale, 'Prompt and content controls', '提示词与内容控制'),
description: t(
locale,
- 'Apply prompt guards, decorators, templates, content moderation,
logging, and auditing before traffic reaches a model.',
- '在流量到达模型之前应用提示词防护、装饰、模板、内容审核、日志和审计。',
+ 'Apply allow and deny patterns with Prompt Guard, and compose separate
plugins for templates, decoration, or moderation.',
+ '使用 Prompt Guard 配置允许与拒绝规则，并按需组合模板、装饰或内容审核插件。',
),
icon: '/img/ai-gateway/security.svg',
+ href: '/docs/apisix/plugins/ai-prompt-guard/',
+ },
+ {
+ title: t(locale, 'AI response caching', 'AI 响应缓存'),
+ description: t(
+ locale,
+ 'With AI Proxy, cache completed LLM responses using Redis-backed exact
matching and optional semantic matching.',
Review Comment:
The capability table below correctly says `ai-cache` pairs with `ai-proxy`
or `ai-proxy-multi`. This card says only “With AI Proxy”, so multi-instance
routes look unsupported. The Lakera card on the next feature has the same
mismatch.
Please name both plugins here, matching the table and the plugin docs.
##########
blog/en/blog/2025/02/24/apisix-ai-gateway-features.md:
##########
@@ -4,136 +4,93 @@ keywords:
- APISIX
- AI gateway
- LLM API
- - security protection
- token rate limiting
- - smart traffic management
+ - model routing
- multi-LLM proxy
- - cloud-native API
-description: "Learn how Apache APISIX AI Gateway handles LLM traffic with
model routing, token rate limiting, retry and fallback, prompt controls, and
observability."
+ - prompt controls
+ - AI response caching
+ - semantic routing
+ - AI observability
+description: "Learn how Apache APISIX AI Gateway handles LLM traffic with
model proxying, configured routing, caching, token limits, prompt controls,
RAG, and observability."
tags: [Ecosystem]
image:
https://static.api7.ai/uploads/2025/03/07/Qs4WrU0I_apisix-ai-gateway.webp
---
->This article will provide an in-depth look at the AI gateway features of the
current and upcoming versions of APISIX. As a multifunctional API and AI
gateway, Apache APISIX offers efficient and secure LLM API calls for AI
applications.
+The [Apache APISIX AI Gateway](/ai-gateway/) applies model proxying,
configured routing, response caching, token limits, prompt controls, retrieval,
and gateway-level observability to LLM traffic through open-source plugins.
This article maps those capabilities to the APISIX plugins that implement them.
<!--truncate-->
-## Introduction: The Rise of AI Agents and the Evolution of AI Gateway
+## Why AI Traffic Needs Additional Gateway Controls
-In recent years, AI agents such as AutoGPT, Chatbots, and AI Assistants have
seen rapid development. These applications rely heavily on API calls to large
language models (LLMs), which have brought about challenges considering high
concurrency, cost control, and security.
+LLM requests share many requirements with ordinary API traffic, including
authentication, routing, rate limiting, resilience, and observability. They
also introduce provider-specific request formats, token-based consumption,
long-running streamed responses, and prompt-processing requirements.
-Traditional API gateways primarily serve Web APIs and microservices and are
not optimized for the unique needs of AI applications. This has led to the
emergence of the concept of AI gateway. An AI gateway needs to provide enhanced
capabilities in the following areas:
+Apache APISIX handles these concerns on the network path between an authorized
application and configured model endpoints. It does not select tools,
orchestrate agent workflows, evaluate answer quality, or replace
application-level authorization. Those responsibilities remain in the
application and AI platform layers.
-- **Multi-LLM Proxy**: Support for multiple LLM providers to avoid vendor
lock-in.
-- **Token Rate Limiting**: Prevent API abuse and optimize cost management.
-- **Security Protection**: Including prompt filtering and content moderation
to ensure compliance of AI applications.
-- **Smart Traffic Management**: Dynamically adjust LLM weights based on cost,
latency, and stability.
+## Proxy Requests to Supported Model Providers
-Apache APISIX is not only an API gateway but also an AI gateway through its
plugins, helping AI applications call LLM APIs more efficiently and securely.
+The [`ai-proxy`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/)
plugin forwards requests to documented model providers and OpenAI-compatible
endpoints. It can transform supported request formats, attach provider
credentials from gateway configuration, and expose a consistent
application-facing endpoint.
-## LLM Proxy: Efficient Management of Multiple LLM Backends
+Provider compatibility depends on the selected APISIX provider type and the
upstream API. Teams should verify request and response fields for each provider
instead of assuming every model implements the same interface.
-AI applications typically do not rely on a single LLM provider but need to
dynamically select the best model based on requirements. For example:
+## Configure Multi-Model Routing and Resilience
-- Using OpenAI GPT-4 for general text generation and Claude for legal document
processing.
-- Switching between Mistral and Gemini to optimize cost and throughput.
+The
[`ai-proxy-multi`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/)
plugin distributes requests across configured model instances. Its documented
routing policies include weighted round robin and consistent hashing, with
optional health checks, bounded retries, and fallback behavior.
-**Apache APISIX's LLM Proxy offers the following capabilities:**
+APISIX 3.18 also supports a semantic routing algorithm. Operators provide
example prompts for each instance and configure an embedding service; the
plugin compares the incoming prompt with those examples and selects the closest
instance that clears the configured threshold. This is configured intent
routing, not automatic optimization based on model cost, latency, answer
quality, or business outcomes.
-✅ Support for Multiple LLM Providers: Including OpenAI, DeepSeek, Claude,
Mistral, Gemini, etc., to avoid vendor lock-in.
+Semantic routing does not participate in health checks, retry, or the normal
fallback strategy. Its designated fallback is used only when no instance clears
the similarity threshold or the embedding request fails. Weighted round robin
and consistent hashing continue to use their documented resilience options.
-✅ LLM Weight and Priority Management: Adjust traffic distribution based on
business needs.
+
-✅ Multi-LLM Load Balancing: Dynamically adjust LLM weights based on latency,
cost, and stability.
+These controls can reduce provider-specific routing logic in applications, but
they do not guarantee uninterrupted service. Availability still depends on
healthy upstreams, network conditions, timeouts, retry limits, and the
configured fallback path.
-✅ Retry and Fallback Mechanisms: Ensure business continuity if an LLM API
fails.
+## Enforce Token-Based Usage Limits
-✅ Load Balancing Across Different Providers of the Same LLM:
+LLM requests can consume very different numbers of prompt and completion
tokens. The
[`ai-rate-limiting`](https://apisix.apache.org/docs/apisix/plugins/ai-rate-limiting/)
plugin applies limits based on token consumption rather than request count
alone.
-For example:
+APISIX supports local and Redis-backed counters for this plugin. Operators can
scope policies through gateway configuration and choose limits appropriate for
their applications. The plugin enforces configured consumption boundaries;
model pricing, budgets, billing, and chargeback remain external
responsibilities.
-- Privately deployed DeepSeek.
-- Official DeepSeek API.
-- DeepSeek API from Volcano Engine
+## Cache Completed LLM Responses
-Users can flexibly allocate traffic weights among different DeepSeek providers
based on latency, stability, and price to achieve the best calling strategy.
+The [`ai-cache`](https://apisix.apache.org/docs/apisix/plugins/ai-cache/)
plugin works with `ai-proxy` or `ai-proxy-multi` to cache completed LLM
responses in Redis. Exact matching is enabled by default. Teams can optionally
add semantic matching, which requires Redis Stack with RediSearch and a
configured embedding service.
Review Comment:
Semantic matching needs Redis Search commands, not Redis Stack as the only
supported product. This site’s Redis integration page documents Redis Open
Source 8.10.1 with `FT._LIST` in the companion lab, and says earlier Redis Open
Source or Redis Stack releases must be pinned and tested.
Please require the Search/RediSearch commands rather than Redis Stack alone,
so the article does not contradict `/integrations/redis/`.
##########
blog/en/blog/2025/02/24/apisix-ai-gateway-features.md:
##########
@@ -4,136 +4,93 @@ keywords:
- APISIX
- AI gateway
- LLM API
- - security protection
- token rate limiting
- - smart traffic management
+ - model routing
- multi-LLM proxy
- - cloud-native API
-description: "Learn how Apache APISIX AI Gateway handles LLM traffic with
model routing, token rate limiting, retry and fallback, prompt controls, and
observability."
+ - prompt controls
+ - AI response caching
+ - semantic routing
+ - AI observability
+description: "Learn how Apache APISIX AI Gateway handles LLM traffic with
model proxying, configured routing, caching, token limits, prompt controls,
RAG, and observability."
tags: [Ecosystem]
image:
https://static.api7.ai/uploads/2025/03/07/Qs4WrU0I_apisix-ai-gateway.webp
---
->This article will provide an in-depth look at the AI gateway features of the
current and upcoming versions of APISIX. As a multifunctional API and AI
gateway, Apache APISIX offers efficient and secure LLM API calls for AI
applications.
+The [Apache APISIX AI Gateway](/ai-gateway/) applies model proxying,
configured routing, response caching, token limits, prompt controls, retrieval,
and gateway-level observability to LLM traffic through open-source plugins.
This article maps those capabilities to the APISIX plugins that implement them.
<!--truncate-->
-## Introduction: The Rise of AI Agents and the Evolution of AI Gateway
+## Why AI Traffic Needs Additional Gateway Controls
-In recent years, AI agents such as AutoGPT, Chatbots, and AI Assistants have
seen rapid development. These applications rely heavily on API calls to large
language models (LLMs), which have brought about challenges considering high
concurrency, cost control, and security.
+LLM requests share many requirements with ordinary API traffic, including
authentication, routing, rate limiting, resilience, and observability. They
also introduce provider-specific request formats, token-based consumption,
long-running streamed responses, and prompt-processing requirements.
-Traditional API gateways primarily serve Web APIs and microservices and are
not optimized for the unique needs of AI applications. This has led to the
emergence of the concept of AI gateway. An AI gateway needs to provide enhanced
capabilities in the following areas:
+Apache APISIX handles these concerns on the network path between an authorized
application and configured model endpoints. It does not select tools,
orchestrate agent workflows, evaluate answer quality, or replace
application-level authorization. Those responsibilities remain in the
application and AI platform layers.
-- **Multi-LLM Proxy**: Support for multiple LLM providers to avoid vendor
lock-in.
-- **Token Rate Limiting**: Prevent API abuse and optimize cost management.
-- **Security Protection**: Including prompt filtering and content moderation
to ensure compliance of AI applications.
-- **Smart Traffic Management**: Dynamically adjust LLM weights based on cost,
latency, and stability.
+## Proxy Requests to Supported Model Providers
-Apache APISIX is not only an API gateway but also an AI gateway through its
plugins, helping AI applications call LLM APIs more efficiently and securely.
+The [`ai-proxy`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/)
plugin forwards requests to documented model providers and OpenAI-compatible
endpoints. It can transform supported request formats, attach provider
credentials from gateway configuration, and expose a consistent
application-facing endpoint.
-## LLM Proxy: Efficient Management of Multiple LLM Backends
+Provider compatibility depends on the selected APISIX provider type and the
upstream API. Teams should verify request and response fields for each provider
instead of assuming every model implements the same interface.
-AI applications typically do not rely on a single LLM provider but need to
dynamically select the best model based on requirements. For example:
+## Configure Multi-Model Routing and Resilience
-- Using OpenAI GPT-4 for general text generation and Claude for legal document
processing.
-- Switching between Mistral and Gemini to optimize cost and throughput.
+The
[`ai-proxy-multi`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/)
plugin distributes requests across configured model instances. Its documented
routing policies include weighted round robin and consistent hashing, with
optional health checks, bounded retries, and fallback behavior.
-**Apache APISIX's LLM Proxy offers the following capabilities:**
+APISIX 3.18 also supports a semantic routing algorithm. Operators provide
example prompts for each instance and configure an embedding service; the
plugin compares the incoming prompt with those examples and selects the closest
instance that clears the configured threshold. This is configured intent
routing, not automatic optimization based on model cost, latency, answer
quality, or business outcomes.
-✅ Support for Multiple LLM Providers: Including OpenAI, DeepSeek, Claude,
Mistral, Gemini, etc., to avoid vendor lock-in.
+Semantic routing does not participate in health checks, retry, or the normal
fallback strategy. Its designated fallback is used only when no instance clears
the similarity threshold or the embedding request fails. Weighted round robin
and consistent hashing continue to use their documented resilience options.
-✅ LLM Weight and Priority Management: Adjust traffic distribution based on
business needs.
+
-✅ Multi-LLM Load Balancing: Dynamically adjust LLM weights based on latency,
cost, and stability.
+These controls can reduce provider-specific routing logic in applications, but
they do not guarantee uninterrupted service. Availability still depends on
healthy upstreams, network conditions, timeouts, retry limits, and the
configured fallback path.
-✅ Retry and Fallback Mechanisms: Ensure business continuity if an LLM API
fails.
+## Enforce Token-Based Usage Limits
-✅ Load Balancing Across Different Providers of the Same LLM:
+LLM requests can consume very different numbers of prompt and completion
tokens. The
[`ai-rate-limiting`](https://apisix.apache.org/docs/apisix/plugins/ai-rate-limiting/)
plugin applies limits based on token consumption rather than request count
alone.
-For example:
+APISIX supports local and Redis-backed counters for this plugin. Operators can
scope policies through gateway configuration and choose limits appropriate for
their applications. The plugin enforces configured consumption boundaries;
model pricing, budgets, billing, and chargeback remain external
responsibilities.
-- Privately deployed DeepSeek.
-- Official DeepSeek API.
-- DeepSeek API from Volcano Engine
+## Cache Completed LLM Responses
-Users can flexibly allocate traffic weights among different DeepSeek providers
based on latency, stability, and price to achieve the best calling strategy.
+The [`ai-cache`](https://apisix.apache.org/docs/apisix/plugins/ai-cache/)
plugin works with `ai-proxy` or `ai-proxy-multi` to cache completed LLM
responses in Redis. Exact matching is enabled by default. Teams can optionally
add semantic matching, which requires Redis Stack with RediSearch and a
configured embedding service.
-These capabilities enable AI applications to adapt flexibly to different LLMs,
improve reliability, and reduce API calling costs.
+Streaming responses are written only after the terminal event is received.
Interrupted streams are not cached, so the plugin does not replay partial
responses. Cache eligibility, isolation, expiration, bypass rules, and semantic
thresholds still need to be configured for the application's data and freshness
requirements.
-
+Cache entries are scoped by Route by default, not by Consumer. If multiple
consumers share a Route, enable `cache_key.include_consumer` or add a trusted
tenant-identifying variable through `cache_key.include_vars` to prevent cached
responses from being reused across tenants.
Review Comment:
`cache_key.include_consumer` only distinguishes authenticated Consumers.
Unauthenticated traffic on the same Route still shares the default cache key,
and a client-controlled header in `include_vars` is not a tenant boundary.
Please require authentication or a trusted server-side variable before
saying cached responses will not be reused across tenants. The Chinese features
article has the same sentence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
