This is an automated email from the ASF dual-hosted git repository.
Yilialinn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/apisix.git
The following commit(s) were added to refs/heads/master by this push:
new a1e66466a docs: add Gemini, Vertex AI, and Anthropic examples to
ai-proxy plugin (#13174)
a1e66466a is described below
commit a1e66466a5d9c1e7a9d67ea80688615084b1bced
Author: Yilia Lin <[email protected]>
AuthorDate: Wed Apr 22 14:29:10 2026 +0800
docs: add Gemini, Vertex AI, and Anthropic examples to ai-proxy plugin
(#13174)
---
docs/en/latest/plugins/ai-proxy.md | 1526 ++++++++++++++++++++++++++++++++++-
docs/zh/latest/plugins/ai-proxy.md | 1558 +++++++++++++++++++++++++++++++++++-
2 files changed, 3050 insertions(+), 34 deletions(-)
diff --git a/docs/en/latest/plugins/ai-proxy.md
b/docs/en/latest/plugins/ai-proxy.md
index 76059bb3e..d9bf7c9e4 100644
--- a/docs/en/latest/plugins/ai-proxy.md
+++ b/docs/en/latest/plugins/ai-proxy.md
@@ -33,6 +33,9 @@ description: The ai-proxy Plugin simplifies access to LLM and
embedding models p
<link rel="canonical" href="https://docs.api7.ai/hub/ai-proxy"; />
</head>
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
## Description
The `ai-proxy` Plugin simplifies access to LLM and embedding models by
transforming Plugin configurations into the designated request format. It
supports the integration with OpenAI, DeepSeek, Azure, AIMLAPI, Anthropic,
OpenRouter, Gemini, Vertex AI, and other OpenAI-compatible APIs.
@@ -73,12 +76,10 @@ In addition, the Plugin also supports logging LLM request
information in the acc
| logging | object | False | |
| Logging configurations. Does not affect `error.log`. |
| logging.summaries | boolean | False | false |
| If true, logs request LLM model, duration, request, and response
tokens. |
| logging.payloads | boolean | False | false |
| If true, logs request and response payload. |
-| timeout | integer | False | 30000 | ≥ 1
| Request timeout in milliseconds when requesting the LLM service.
Applied per socket operation (connect / send / read block); does not cap the
total duration of a streaming response. |
-| max_stream_duration_ms | integer | False | | ≥ 1
| Maximum wall-clock duration (in milliseconds) for a streaming
AI response. If the upstream keeps sending data past this deadline, the gateway
closes the connection. Unset means no cap. Use this to protect the gateway from
upstream bugs that produce tokens indefinitely. When the limit is hit
mid-stream, the downstream SSE stream is truncated (no protocol-specific
terminator such as `[DONE]`, ` [...]
-| max_response_bytes | integer | False | | ≥ 1
| Maximum total bytes read from the upstream for a single AI
response (streaming or non-streaming). If exceeded, the gateway closes the
connection. For non-streaming responses with `Content-Length`, the check is
performed before reading the body; for chunked (no-`Content-Length`)
non-streaming responses and for streaming responses, the cap is enforced
incrementally as bytes are received. Unset [...]
+| timeout | integer | False | 30000 | 1 - 600000
| Request timeout in milliseconds when requesting the LLM service.
|
| keepalive | boolean | False | true |
| If true, keeps the connection alive when requesting the LLM
service. |
| keepalive_timeout | integer | False | 60000 | ≥ 1000
| Keepalive timeout in milliseconds when connecting to the LLM
service. |
-| keepalive_pool | integer | False | 30 |
| Keepalive pool size for the LLM service connection. |
+| keepalive_pool | integer | False | 30 | ≥ 1
| Keepalive pool size for the LLM service connection. |
| ssl_verify | boolean | False | true |
| If true, verifies the LLM service's certificate. |
## Provider-aware `max_tokens` mapping
@@ -141,6 +142,17 @@ Obtain the OpenAI [API
key](https://openai.com/blog/openai-api) and save it to a
export OPENAI_API_KEY=<your-api-key>
```
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
Create a Route and configure the `ai-proxy` Plugin as such:
```shell
@@ -166,6 +178,135 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+Create a Route with the `ai-proxy` Plugin configured as such:
+
+```yaml title="adc.yaml"
+services:
+ - name: openai-service
+ routes:
+ - name: openai-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="ai-proxy-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: openai-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="ai-proxy-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: openai-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: openai-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+```
+
+</TabItem>
+
+</Tabs>
+
+Apply the configuration to your cluster:
+
+```shell
+kubectl apply -f ai-proxy-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
Send a POST request to the Route with a system prompt and a sample user
question in the request body:
```shell
@@ -204,14 +345,25 @@ You should receive a response similar to the following:
### Proxy to DeepSeek
-The following example demonstrates how you can configure the `ai-proxy` Plugin
to proxy requests to DeekSeek.
+The following example demonstrates how you can configure the `ai-proxy` Plugin
to proxy requests to DeepSeek.
-Obtain the DeekSeek API key and save it to an environment variable:
+Obtain the DeepSeek API key and save it to an environment variable:
```shell
export DEEPSEEK_API_KEY=<your-api-key>
```
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
Create a Route and configure the `ai-proxy` Plugin as such:
```shell
@@ -237,6 +389,135 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+Create a Route with the `ai-proxy` Plugin configured as such:
+
+```yaml title="adc.yaml"
+services:
+ - name: deepseek-service
+ routes:
+ - name: deepseek-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer ${DEEPSEEK_API_KEY}"
+ options:
+ model: deepseek-chat
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="deepseek-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: deepseek-chat
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: deepseek-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="deepseek-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: deepseek-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: deepseek-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: deepseek-chat
+```
+
+</TabItem>
+
+</Tabs>
+
+Apply the configuration to your cluster:
+
+```shell
+kubectl apply -f deepseek-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
Send a POST request to the Route with a sample question in the request body:
```shell
@@ -286,6 +567,17 @@ Obtain the Azure OpenAI API key and save it to an
environment variable:
export AZ_OPENAI_API_KEY=<your-api-key>
```
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
Create a Route and configure the `ai-proxy` Plugin as such:
```shell
@@ -297,7 +589,7 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
"methods": ["POST"],
"plugins": {
"ai-proxy": {
- "provider": "openai-compatible",
+ "provider": "azure-openai",
"auth": {
"header": {
"api-key": "'"$AZ_OPENAI_API_KEY"'"
@@ -307,13 +599,148 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
"model": "gpt-4"
},
"override": {
- "endpoint":
"https://api7-auzre-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+ "endpoint":
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
}
}
}
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+Create a Route with the `ai-proxy` Plugin configured as such:
+
+```yaml title="adc.yaml"
+services:
+ - name: azure-openai-service
+ routes:
+ - name: azure-openai-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: azure-openai
+ auth:
+ header:
+ api-key: "${AZ_OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+ override:
+ endpoint:
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="azure-openai-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: azure-openai
+ auth:
+ header:
+ api-key: "your-api-key"
+ options:
+ model: gpt-4
+ override:
+ endpoint:
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: azure-openai-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="azure-openai-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: azure-openai-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: azure-openai-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: azure-openai
+ auth:
+ header:
+ api-key: "your-api-key"
+ options:
+ model: gpt-4
+ override:
+ endpoint:
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+```
+
+</TabItem>
+
+</Tabs>
+
+Apply the configuration to your cluster:
+
+```shell
+kubectl apply -f azure-openai-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
Send a POST request to the Route with a sample question in the request body:
```shell
@@ -356,7 +783,7 @@ You should receive a response similar to the following:
}
```
-### Proxy to Embedding Models
+### Proxy to OpenAI Embedding Models
The following example demonstrates how you can configure the `ai-proxy` Plugin
to proxy requests to embedding models. This example will use the OpenAI
embedding model endpoint.
@@ -366,6 +793,17 @@ Obtain the OpenAI [API
key](https://openai.com/blog/openai-api) and save it to a
export OPENAI_API_KEY=<your-api-key>
```
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
Create a Route and configure the `ai-proxy` Plugin as such:
```shell
@@ -395,6 +833,144 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+Create a Route with the `ai-proxy` Plugin configured as such:
+
+```yaml title="adc.yaml"
+services:
+ - name: openai-embeddings-service
+ routes:
+ - name: openai-embeddings-route
+ uris:
+ - /embeddings
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: text-embedding-3-small
+ encoding_format: float
+ override:
+ endpoint: "https://api.openai.com/v1/embeddings";
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="openai-embeddings-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: text-embedding-3-small
+ encoding_format: float
+ override:
+ endpoint: "https://api.openai.com/v1/embeddings";
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: openai-embeddings-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /embeddings
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="openai-embeddings-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: openai-embeddings-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: openai-embeddings-route
+ match:
+ paths:
+ - /embeddings
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: text-embedding-3-small
+ encoding_format: float
+ override:
+ endpoint: "https://api.openai.com/v1/embeddings";
+```
+
+</TabItem>
+
+</Tabs>
+
+Apply the configuration to your cluster:
+
+```shell
+kubectl apply -f openai-embeddings-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
Send a POST request to the Route with an input string:
```shell
@@ -433,12 +1009,942 @@ You should receive a response similar to the following:
}
```
+### Proxy to Anthropic
+
+The following example demonstrates how you can configure the `ai-proxy` Plugin
to proxy requests to Anthropic's Claude API for chat completion.
+
+Obtain an Anthropic [API key](https://console.anthropic.com/settings/keys) and
save it to an environment variable:
+
+```shell
+export ANTHROPIC_API_KEY=<your-api-key>
+```
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+Create a Route and configure the `ai-proxy` Plugin as such:
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-anthropic-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "anthropic",
+ "auth": {
+ "header": {
+ "x-api-key": "'"$ANTHROPIC_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "claude-sonnet-4-20250514"
+ }
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+Create a Route with the `ai-proxy` Plugin configured as such:
+
+```yaml title="adc.yaml"
+services:
+ - name: anthropic-service
+ routes:
+ - name: anthropic-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: anthropic
+ auth:
+ header:
+ x-api-key: "${ANTHROPIC_API_KEY}"
+ options:
+ model: claude-sonnet-4-20250514
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="anthropic-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: anthropic
+ auth:
+ header:
+ x-api-key: "your-api-key"
+ options:
+ model: claude-sonnet-4-20250514
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: anthropic-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="anthropic-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: anthropic-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: anthropic-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: anthropic
+ auth:
+ header:
+ x-api-key: "your-api-key"
+ options:
+ model: claude-sonnet-4-20250514
+```
+
+</TabItem>
+
+</Tabs>
+
+Apply the configuration to your cluster:
+
+```shell
+kubectl apply -f anthropic-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
+The configuration above specifies `anthropic` as the provider and attaches the
Anthropic API key in the `x-api-key` header.
+
+Send a POST request to the Route with a system prompt and a sample user
question in the request body:
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+You should receive a response similar to the following:
+
+```json
+{
+ "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
+ "type": "message",
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "1+1 equals 2."
+ }
+ ],
+ "model": "claude-sonnet-4-20250514",
+ "stop_reason": "end_turn",
+ "usage": {
+ "input_tokens": 19,
+ "output_tokens": 11
+ }
+}
+```
+
+### Convert Anthropic Requests to OpenAI-Compatible Backend
+
+The following example demonstrates how the `ai-proxy` Plugin can accept
requests in the Anthropic Messages API format and automatically convert them to
the OpenAI-compatible format before forwarding to any OpenAI-compatible backend
(such as OpenAI, DeepSeek, or other compatible services). This is useful when
client applications send Anthropic-formatted requests but you want to use a
different LLM backend.
+
+The protocol conversion is triggered automatically when the Route URI is set
to `/v1/messages` (the Anthropic Messages API endpoint). The Plugin will
convert Anthropic-formatted requests to OpenAI-compatible format and transform
the responses back to Anthropic format.
+
+Obtain an API key for your chosen OpenAI-compatible backend service and save
it to an environment variable. This example uses OpenAI:
+
+```shell
+export BACKEND_API_KEY=<your-api-key>
+```
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+Create a Route with the URI set to `/v1/messages` to trigger automatic
Anthropic protocol conversion, and configure the `ai-proxy` Plugin as such:
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-anthropic-convert-route",
+ "uri": "/v1/messages",
+ "methods": ["POST"],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$BACKEND_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "gpt-4"
+ }
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+Create a Route with the URI set to `/v1/messages` to trigger automatic
Anthropic protocol conversion, and configure the `ai-proxy` Plugin as such:
+
+```yaml title="adc.yaml"
+services:
+ - name: anthropic-convert-service
+ routes:
+ - name: anthropic-convert-route
+ uris:
+ - /v1/messages
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${BACKEND_API_KEY}"
+ options:
+ model: gpt-4
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="anthropic-convert-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: anthropic-convert-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /v1/messages
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="anthropic-convert-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: anthropic-convert-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: anthropic-convert-route
+ match:
+ paths:
+ - /v1/messages
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+```
+
+</TabItem>
+
+</Tabs>
+
+Apply the configuration to your cluster:
+
+```shell
+kubectl apply -f anthropic-convert-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
+The backend provider can be any OpenAI-compatible provider, such as `openai`,
`deepseek`, or others.
+
+Send a POST request to the Route in Anthropic Messages API format:
+
+```shell
+curl "http://127.0.0.1:9080/v1/messages"; -X POST \
+ -H "Content-Type: application/json" \
+ -H "x-api-key: ${BACKEND_API_KEY}" \
+ -H "anthropic-version: 2023-06-01" \
+ -d '{
+ "model": "gpt-4",
+ "max_tokens": 1024,
+ "messages": [
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+Although the request is sent in Anthropic format, it will be automatically
converted to OpenAI format and forwarded to the backend. The response is
converted back to Anthropic format:
+
+```json
+{
+ "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
+ "type": "message",
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "1+1 equals 2."
+ }
+ ],
+ "model": "gpt-4",
+ "stop_reason": "end_turn",
+ "usage": {
+ "input_tokens": 12,
+ "output_tokens": 8
+ }
+}
+```
+
+The Plugin supports all features of the Anthropic Messages API, including
streaming (SSE), system prompts, and tool use (function calling). The protocol
conversion handles the bidirectional mapping between Anthropic and OpenAI
formats transparently.
+
+### Proxy to Selected Model using Request Body Parameter
+
+The following example demonstrates how you can proxy requests to different
models on the same URI, based on the user-specified model in the user requests.
You will be using the `post_arg.*` variable to fetch the value of the request
body parameter.
+
+The example will use OpenAI and DeepSeek as the example LLM services. Obtain
the OpenAI and DeepSeek API keys and save them to environment variables:
+
+```shell
+export OPENAI_API_KEY=<your-api-key>
+export DEEPSEEK_API_KEY=<your-api-key>
+```
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+Create a Route to the OpenAI API with the `ai-proxy` Plugin. The Route URI is
`/anything` and it matches requests where the body parameter `model` is set to
`openai`:
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-openai-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "vars": [[ "post_arg.model", "==", "openai" ]],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$OPENAI_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "gpt-4"
+ }
+ }
+ }
+ }'
+```
+
+Create another Route `/anything` to the DeepSeek API with the `ai-proxy`
Plugin. This Route matches requests where the body parameter `model` is set to
`deepseek`:
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-deepseek-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "vars": [[ "post_arg.model", "==", "deepseek" ]],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "deepseek",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$DEEPSEEK_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "deepseek-chat"
+ }
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+Create two Routes with the `ai-proxy` Plugin configured for different
providers:
+
+```yaml title="adc.yaml"
+services:
+ - name: multi-model-service
+ routes:
+ - name: openai-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ vars:
+ - - post_arg.model
+ - ==
+ - openai
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+ - name: deepseek-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ vars:
+ - - post_arg.model
+ - ==
+ - deepseek
+ plugins:
+ ai-proxy:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer ${DEEPSEEK_API_KEY}"
+ options:
+ model: deepseek-chat
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+Body parameter matching is not supported in HTTPRoute. The supported matching
mechanisms are `path`, `method`, `headers`, and `queryParams`. This example
cannot be completed with Gateway API.
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+Body parameter matching is currently not supported in ApisixRoute. The
supported matching mechanisms are based on `Header`, `Query`, or `Path`. This
example cannot be completed with APISIX CRDs.
+
+</TabItem>
+
+</Tabs>
+
+</TabItem>
+
+</Tabs>
+
+Send a POST request to the Route with `model` set to `openai`:
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "openai",
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+You should receive a response similar to the following:
+
+```json
+{
+ ...,
+ "model": "gpt-4-0613",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "1+1 equals 2.",
+ "refusal": null
+ },
+ "logprobs": null,
+ "finish_reason": "stop"
+ }
+ ],
+ ...
+}
+```
+
+Send a POST request to the Route with `model` set to `deepseek`:
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "deepseek",
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+You should receive a response similar to the following:
+
+```json
+{
+ ...,
+ "model": "deepseek-chat",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "The sum of 1 and 1 is 2. This is a basic arithmetic
operation where you combine two units to get a total of two units."
+ },
+ "logprobs": null,
+ "finish_reason": "stop"
+ }
+ ],
+ ...
+}
+```
+
+You can also configure `post_arg.*` to fetch nested request body parameter.
For instance, if the request format is:
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": {
+ "name": "openai"
+ },
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+You can configure the `vars` on the Route to be `[[ "post_arg.model.name",
"==", "openai" ]]`.
+
+### Send Request Log to Logger
+
+The following example demonstrates how you can log request and response
information, including LLM model, token, and payload, and push them to a
logger. Before proceeding, you should first set up a logger, such as Kafka. See
[`kafka-logger`](./kafka-logger.md) for more information.
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+Create a Route to your LLM service and configure logging details. Enable
`summaries` to log request LLM model, duration, request and response tokens.
Enable `payloads` to log request and response payload. Update the
`kafka-logger` configuration with your Kafka address, topic, and key:
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-openai-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$OPENAI_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "gpt-4"
+ },
+ "logging": {
+ "summaries": true,
+ "payloads": true
+ }
+ },
+ "kafka-logger": {
+ "brokers": [
+ {
+ "host": "127.0.0.1",
+ "port": 9092
+ }
+ ],
+ "kafka_topic": "test2",
+ "key": "key1",
+ "batch_max_size": 1
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+Create a Route with both `ai-proxy` and `kafka-logger` Plugins. Enable
`summaries` to log request LLM model, duration, request and response tokens.
Enable `payloads` to log request and response payload. Update the
`kafka-logger` configuration with your Kafka address, topic, and key:
+
+```yaml title="adc.yaml"
+services:
+ - name: logging-service
+ routes:
+ - name: logging-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+ logging:
+ summaries: true
+ payloads: true
+ kafka-logger:
+ brokers:
+ - host: 127.0.0.1
+ port: 9092
+ kafka_topic: test2
+ key: key1
+ batch_max_size: 1
+```
+
+Synchronize the configuration to the gateway:
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="logging-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-logging-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+ logging:
+ summaries: true
+ payloads: true
+ - name: kafka-logger
+ config:
+ brokers:
+ - host: kafka.aic.svc.cluster.local
+ port: 9092
+ kafka_topic: test2
+ key: key1
+ batch_max_size: 1
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: logging-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-logging-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="logging-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: logging-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: logging-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+ logging:
+ summaries: true
+ payloads: true
+ - name: kafka-logger
+ enable: true
+ config:
+ brokers:
+ - host: kafka.aic.svc.cluster.local
+ port: 9092
+ kafka_topic: test2
+ key: key1
+ batch_max_size: 1
+```
+
+</TabItem>
+
+</Tabs>
+
+Apply the configuration to your cluster:
+
+```shell
+kubectl apply -f logging-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
+Send a POST request to the Route:
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+You should receive a response similar to the following:
+
+```json
+{
+ ...,
+ "model": "gpt-4-0613",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "1+1 equals 2.",
+ "refusal": null
+ },
+ "logprobs": null,
+ "finish_reason": "stop"
+ }
+ ],
+ ...
+}
+```
+
+In the Kafka topic, you should also see a log entry corresponding to the
request with the LLM summary and request/response payload.
+
### Include LLM Information in Access Log
The following example demonstrates how you can log LLM request related
information in the gateway's access log to improve analytics and audit. The
following variables are available:
* `request_llm_model`: LLM model name specified in the request.
-* `apisix_upstream_response_time`: Time taken for APISIX to send the request
to the upstream service and receive the full response
+* `apisix_upstream_response_time`: Time taken for APISIX to send the request
to the upstream service and receive the full response.
* `request_type`: Type of request, where the value could be
`traditional_http`, `ai_chat`, or `ai_stream`.
* `llm_time_to_first_token`: Duration from request sending to the first token
received from the LLM service, in milliseconds.
* `llm_model`: LLM model.
diff --git a/docs/zh/latest/plugins/ai-proxy.md
b/docs/zh/latest/plugins/ai-proxy.md
index c72169d25..dab32be27 100644
--- a/docs/zh/latest/plugins/ai-proxy.md
+++ b/docs/zh/latest/plugins/ai-proxy.md
@@ -33,6 +33,9 @@ description: ai-proxy 插件通过将插件配置转换为所需的请求格式
<link rel="canonical" href="https://docs.api7.ai/hub/ai-proxy"; />
</head>
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
## 描述
`ai-proxy` 插件通过将插件配置转换为指定的请求格式，简化了对 LLM 和嵌入模型的访问。它支持与
OpenAI、DeepSeek、Azure、AIMLAPI、Anthropic、OpenRouter、Gemini、Vertex AI 和其他 OpenAI
兼容的 API 集成。
@@ -51,18 +54,18 @@ description: ai-proxy 插件通过将插件配置转换为所需的请求格式
| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述
|
|--------------------|--------|----------|---------|------------------------------------------|-------------|
-| provider | string | 是 | | [openai, deepseek,
azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai,
openai-compatible] | LLM 服务提供商。当设置为 `openai` 时，插件将代理请求到
`https://api.openai.com/chat/completions`。当设置为 `deepseek` 时，插件将代理请求到
`https://api.deepseek.com/chat/completions`。当设置为 `aimlapi` 时，插件使用 OpenAI
兼容驱动程序，默认将请求代理到 `https://api.aimlapi.com/v1/chat/completions`。当设置为 `anthropic`
时，插件将代理请求到 `https://api.anthropic.com/v1/chat/completions`。当设置为 `openrouter`
时，插件 [...]
-| provider_conf | object | 否 | |
| 特定提供商的配置。当 `provider` 设置为 `vertex-ai` 且未配置 `override` 时必填。 |
-| provider_conf.project_id | string | 是 | |
| Google Cloud 项目 ID。 |
-| provider_conf.region | string | 是 | |
| Google Cloud 区域。 |
+| provider | string | 是 | | [openai, deepseek,
azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai,
openai-compatible] | LLM 服务提供商。当设置为 `openai` 时，插件将代理请求到
`https://api.openai.com/chat/completions`。当设置为 `deepseek` 时，插件将代理请求到
`https://api.deepseek.com/chat/completions`。当设置为 `aimlapi` 时，插件使用 OpenAI
兼容驱动程序，默认将请求代理到 `https://api.aimlapi.com/v1/chat/completions`。当设置为 `anthropic`
时，插件将代理请求到 `https://api.anthropic.com/v1/chat/completions`。当设置为 `openrouter`
时，插件 [...]
+| provider_conf | object | 否 | |
| 特定提供商的配置。当 `provider` 设置为 `vertex-ai` 且未配置 `override` 时必需。 |
+| provider_conf.project_id | string | 是 | |
| Google Cloud 项目 ID。 |
+| provider_conf.region | string | 是 | |
| Google Cloud 区域。 |
| auth | object | 是 | |
| 身份验证配置。 |
| auth.header | object | 否 | |
| 身份验证标头。必须配置 `header` 或 `query` 中的至少一个。 |
| auth.query | object | 否 | |
| 身份验证查询参数。必须配置 `header` 或 `query` 中的至少一个。 |
| auth.gcp | object | 否 | |
| Google Cloud Platform (GCP) 身份验证配置。 |
-| auth.gcp.service_account_json | string | 否 | |
| GCP 服务帐户 JSON 文件的内容。也可以通过设置“GCP_SERVICE_ACCOUNT”环境变量来配置。 |
-| auth.gcp.max_ttl | integer | 否 | | minimum = 1
| 用于缓存 GCP 访问令牌的最大 TTL（以秒为单位）。 |
-| auth.gcp.expire_early_secs | integer | 否 | 60 | minimum = 0
| 在访问令牌实际过期时间之前使其过期的秒数，以避免边缘情况。 |
-| options | object | 否 | |
| 模型配置。除了 `model` 之外，您还可以配置其他参数，它们将在请求体中转发到上游 LLM 服务。例如，如果您使用
OpenAI，可以配置其他参数，如 `temperature`、`top_p` 和 `stream`。有关更多可用选项，请参阅您的 LLM 提供商的 API
文档。 |
+| auth.gcp.service_account_json | string | 否 | |
| GCP 服务账号 JSON 文件内容。也可以通过设置 `GCP_SERVICE_ACCOUNT` 环境变量来配置。 |
+| auth.gcp.max_ttl | integer | 否 | | ≥ 1
| GCP 访问令牌缓存的最大 TTL（秒）。 |
+| auth.gcp.expire_early_secs | integer | 否 | 60 | ≥ 0
| 在访问令牌实际过期之前提前过期的秒数，以避免边缘情况。 |
+| options | object | 否 | |
| 模型配置。除了 `model` 之外，你还可以配置其他参数，它们将在请求体中转发到上游 LLM 服务。例如，如果你使用
OpenAI，可以配置其他参数，如 `temperature`、`top_p` 和 `stream`。有关更多可用选项，请参阅你的 LLM 提供商的 API
文档。 |
| options.model | string | 否 | |
| LLM 模型的名称，如 `gpt-4` 或 `gpt-3.5`。请参阅 LLM 提供商的 API 文档以了解可用模型。 |
| override | object | 否 | |
| 覆盖设置。 |
| override.endpoint | string | 否 | |
| 自定义 LLM 提供商端点，当 `provider` 为 `openai-compatible` 时必需。 |
@@ -73,12 +76,10 @@ description: ai-proxy 插件通过将插件配置转换为所需的请求格式
| logging | object | 否 | |
| 日志配置。不影响 `error.log`。 |
| logging.summaries | boolean | 否 | false |
| 如果为 true，记录请求 LLM 模型、持续时间、请求和响应令牌。 |
| logging.payloads | boolean | 否 | false |
| 如果为 true，记录请求和响应负载。 |
-| timeout | integer | 否 | 30000 | ≥ 1
| 请求 LLM 服务时的请求超时时间（毫秒）。应用于单次 socket 操作（连接 / 发送 / 读取块），不限制流式响应的总时长。 |
-| max_stream_duration_ms | integer | 否 | | ≥ 1
| 流式 AI 响应的总墙钟时长上限（毫秒）。若上游在此时间后仍持续发送数据，网关将关闭连接。未设置时不限制。用于防护上游持续输出
token 导致网关 CPU 被打满的异常情况。中途触发上限时，下游 SSE 流会被截断（不再发送协议特定的终止标记，例如
`[DONE]`、`message_stop` 或 `response.completed`），客户端应将缺失的终止标记视为响应未完成。 |
-| max_response_bytes | integer | 否 | | ≥ 1
| 单次 AI 响应（流式或非流式）允许从上游读取的最大总字节数。超出时关闭连接。非流式响应若存在
`Content-Length`，在读取 body 之前预检；否则（chunked 传输）与流式响应一样在接收字节的过程中增量检查。未设置时不限制。 |
+| timeout | integer | 否 | 30000 | 1 - 600000
| 请求 LLM 服务时的请求超时时间（毫秒）。 |
| keepalive | boolean | 否 | true |
| 如果为 true，在请求 LLM 服务时保持连接活跃。 |
| keepalive_timeout | integer | 否 | 60000 | ≥ 1000
| 连接到 LLM 服务时的保活超时时间（毫秒）。 |
-| keepalive_pool | integer | 否 | 30 |
| LLM 服务连接的保活池大小。 |
+| keepalive_pool | integer | 否 | 30 | ≥ 1
| LLM 服务连接的保活池大小。 |
| ssl_verify | boolean | 否 | true |
| 如果为 true，验证 LLM 服务的证书。 |
## Provider-aware `max_tokens` mapping
@@ -123,7 +124,7 @@ description: ai-proxy 插件通过将插件配置转换为所需的请求格式
:::note
-您可以使用以下命令从 `config.yaml` 获取 `admin_key` 并保存到环境变量中：
+你可以使用以下命令从 `config.yaml` 获取 `admin_key` 并保存到环境变量中：
```bash
admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed
's/"//g')
@@ -141,6 +142,17 @@ admin_key=$(yq '.deployment.admin.admin_key[0].key'
conf/config.yaml | sed 's/"/
export OPENAI_API_KEY=<your-api-key>
```
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
创建路由并配置 `ai-proxy` 插件：
```shell
@@ -166,6 +178,135 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+创建包含 `ai-proxy` 插件配置的路由：
+
+```yaml title="adc.yaml"
+services:
+ - name: openai-service
+ routes:
+ - name: openai-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="ai-proxy-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: openai-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="ai-proxy-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: openai-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: openai-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+```
+
+</TabItem>
+
+</Tabs>
+
+将配置应用到集群：
+
+```shell
+kubectl apply -f ai-proxy-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
向路由发送 POST 请求，在请求体中包含系统提示和示例用户问题：
```shell
@@ -180,7 +321,7 @@ curl "http://127.0.0.1:9080/anything"; -X POST \
}'
```
-您应该收到类似以下的响应：
+你应该收到类似以下的响应：
```json
{
@@ -212,6 +353,17 @@ curl "http://127.0.0.1:9080/anything"; -X POST \
export DEEPSEEK_API_KEY=<your-api-key>
```
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
创建路由并配置 `ai-proxy` 插件：
```shell
@@ -237,6 +389,135 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+创建包含 `ai-proxy` 插件配置的路由：
+
+```yaml title="adc.yaml"
+services:
+ - name: deepseek-service
+ routes:
+ - name: deepseek-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer ${DEEPSEEK_API_KEY}"
+ options:
+ model: deepseek-chat
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="deepseek-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: deepseek-chat
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: deepseek-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="deepseek-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: deepseek-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: deepseek-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: deepseek-chat
+```
+
+</TabItem>
+
+</Tabs>
+
+将配置应用到集群：
+
+```shell
+kubectl apply -f deepseek-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
向路由发送 POST 请求，在请求体中包含示例问题：
```shell
@@ -256,7 +537,7 @@ curl "http://127.0.0.1:9080/anything"; -X POST \
}'
```
-您应该收到类似以下的响应：
+你应该收到类似以下的响应：
```json
{
@@ -286,7 +567,18 @@ curl "http://127.0.0.1:9080/anything"; -X POST \
export AZ_OPENAI_API_KEY=<your-api-key>
```
-创建路由并配置 `ai-proxy` 插件：
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+创建路由并配置 `ai-proxy` 插件，将 `provider` 设置为 `azure-openai`，在 `api-key` 标头中附加 Azure
OpenAI API 密钥，并指定 Azure OpenAI 端点：
```shell
curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
@@ -297,7 +589,7 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
"methods": ["POST"],
"plugins": {
"ai-proxy": {
- "provider": "openai-compatible",
+ "provider": "azure-openai",
"auth": {
"header": {
"api-key": "'"$AZ_OPENAI_API_KEY"'"
@@ -307,13 +599,148 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
"model": "gpt-4"
},
"override": {
- "endpoint":
"https://api7-auzre-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+ "endpoint":
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
}
}
}
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+创建包含 `ai-proxy` 插件配置的路由，将 `provider` 设置为 `azure-openai`，在 `api-key` 标头中附加
Azure OpenAI API 密钥，并指定 Azure OpenAI 端点：
+
+```yaml title="adc.yaml"
+services:
+ - name: azure-openai-service
+ routes:
+ - name: azure-openai-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: azure-openai
+ auth:
+ header:
+ api-key: "${AZ_OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+ override:
+ endpoint:
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="azure-openai-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: azure-openai
+ auth:
+ header:
+ api-key: "your-api-key"
+ options:
+ model: gpt-4
+ override:
+ endpoint:
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: azure-openai-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="azure-openai-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: azure-openai-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: azure-openai-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: azure-openai
+ auth:
+ header:
+ api-key: "your-api-key"
+ options:
+ model: gpt-4
+ override:
+ endpoint:
"https://api7-azure-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview";
+```
+
+</TabItem>
+
+</Tabs>
+
+将配置应用到集群：
+
+```shell
+kubectl apply -f azure-openai-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
向路由发送 POST 请求，在请求体中包含示例问题：
```shell
@@ -339,7 +766,7 @@ curl "http://127.0.0.1:9080/anything"; -X POST \
}'
```
-您应该收到类似以下的响应：
+你应该收到类似以下的响应：
```json
{
@@ -356,7 +783,7 @@ curl "http://127.0.0.1:9080/anything"; -X POST \
}
```
-### 代理到嵌入模型
+### 代理到 OpenAI 嵌入模型
以下示例演示了如何配置 `ai-proxy` 插件以将请求代理到嵌入模型。此示例将使用 OpenAI 嵌入模型端点。
@@ -366,7 +793,18 @@ curl "http://127.0.0.1:9080/anything"; -X POST \
export OPENAI_API_KEY=<your-api-key>
```
-创建路由并配置 `ai-proxy` 插件：
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+创建路由并配置 `ai-proxy` 插件，将 `provider` 设置为 `openai`，指定嵌入模型名称，添加 `encoding_format`
参数以配置返回的嵌入向量为浮点数列表，并使用 `override` 将默认端点覆盖为 [嵌入 API
端点](https://platform.openai.com/docs/api-reference/embeddings)：
```shell
curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
@@ -395,6 +833,144 @@ curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
}'
```
+</TabItem>
+
+<TabItem value="adc">
+
+创建包含 `ai-proxy` 插件配置的路由，将 `provider` 设置为 `openai`，指定嵌入模型名称，添加
`encoding_format` 参数，并使用 `override` 将默认端点覆盖为 [嵌入 API
端点](https://platform.openai.com/docs/api-reference/embeddings)：
+
+```yaml title="adc.yaml"
+services:
+ - name: openai-embeddings-service
+ routes:
+ - name: openai-embeddings-route
+ uris:
+ - /embeddings
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: text-embedding-3-small
+ encoding_format: float
+ override:
+ endpoint: "https://api.openai.com/v1/embeddings";
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="openai-embeddings-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: text-embedding-3-small
+ encoding_format: float
+ override:
+ endpoint: "https://api.openai.com/v1/embeddings";
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: openai-embeddings-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /embeddings
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="openai-embeddings-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: openai-embeddings-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: openai-embeddings-route
+ match:
+ paths:
+ - /embeddings
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: text-embedding-3-small
+ encoding_format: float
+ override:
+ endpoint: "https://api.openai.com/v1/embeddings";
+```
+
+</TabItem>
+
+</Tabs>
+
+将配置应用到集群：
+
+```shell
+kubectl apply -f openai-embeddings-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
向路由发送 POST 请求，包含输入字符串：
```shell
@@ -405,7 +981,7 @@ curl "http://127.0.0.1:9080/embeddings"; -X POST \
}'
```
-您应该收到类似以下的响应：
+你应该收到类似以下的响应：
```json
{
@@ -433,6 +1009,940 @@ curl "http://127.0.0.1:9080/embeddings"; -X POST \
}
```
+### 代理到 Anthropic
+
+以下示例演示了如何配置 `ai-proxy` 插件以将请求代理到 Anthropic 的 Claude API 进行聊天补全。
+
+获取 Anthropic [API 密钥](https://console.anthropic.com/settings/keys)并保存到环境变量：
+
+```shell
+export ANTHROPIC_API_KEY=<your-api-key>
+```
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+创建路由并配置 `ai-proxy` 插件，将 `provider` 设置为 `anthropic`，并在 `x-api-key` 标头中附加
Anthropic API 密钥：
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-anthropic-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "anthropic",
+ "auth": {
+ "header": {
+ "x-api-key": "'"$ANTHROPIC_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "claude-sonnet-4-20250514"
+ }
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+创建包含 `ai-proxy` 插件配置的路由，将 `provider` 设置为 `anthropic`，并在 `x-api-key` 标头中附加
Anthropic API 密钥：
+
+```yaml title="adc.yaml"
+services:
+ - name: anthropic-service
+ routes:
+ - name: anthropic-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: anthropic
+ auth:
+ header:
+ x-api-key: "${ANTHROPIC_API_KEY}"
+ options:
+ model: claude-sonnet-4-20250514
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="anthropic-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: anthropic
+ auth:
+ header:
+ x-api-key: "your-api-key"
+ options:
+ model: claude-sonnet-4-20250514
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: anthropic-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="anthropic-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: anthropic-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: anthropic-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: anthropic
+ auth:
+ header:
+ x-api-key: "your-api-key"
+ options:
+ model: claude-sonnet-4-20250514
+```
+
+</TabItem>
+
+</Tabs>
+
+将配置应用到集群：
+
+```shell
+kubectl apply -f anthropic-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
+向路由发送 POST 请求，在请求体中包含系统提示和示例用户问题：
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+你应该收到类似以下的响应：
+
+```json
+{
+ "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
+ "type": "message",
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "1+1 equals 2."
+ }
+ ],
+ "model": "claude-sonnet-4-20250514",
+ "stop_reason": "end_turn",
+ "usage": {
+ "input_tokens": 19,
+ "output_tokens": 11
+ }
+}
+```
+
+### 将 Anthropic 请求转换为 OpenAI 兼容后端
+
+以下示例演示了 `ai-proxy` 插件如何接受 Anthropic Messages API 格式的请求，并自动将其转换为 OpenAI
兼容格式，然后转发到任何 OpenAI 兼容后端（如 OpenAI、DeepSeek 或其他兼容服务）。当客户端应用程序发送 Anthropic
格式的请求但你希望使用不同的 LLM 后端时，这非常有用。
+
+当路由 URI 设置为 `/v1/messages`（Anthropic Messages API 端点）时，协议转换会自动触发。插件会将
Anthropic 格式的请求转换为 OpenAI 兼容格式，并将响应转换回 Anthropic 格式。
+
+获取你选择的 OpenAI 兼容后端服务的 API 密钥并保存到环境变量。此示例使用 OpenAI：
+
+```shell
+export BACKEND_API_KEY=<your-api-key>
+```
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+创建路由并配置 `ai-proxy` 插件。将 URI 设置为 `/v1/messages` 以触发自动 Anthropic 协议转换，后端提供商可以是任何
OpenAI 兼容的提供商：
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-anthropic-convert-route",
+ "uri": "/v1/messages",
+ "methods": ["POST"],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$BACKEND_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "gpt-4"
+ }
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+创建包含 `ai-proxy` 插件配置的路由。将 URI 设置为 `/v1/messages` 以触发自动 Anthropic
协议转换，后端提供商可以是任何 OpenAI 兼容的提供商：
+
+```yaml title="adc.yaml"
+services:
+ - name: anthropic-convert-service
+ routes:
+ - name: anthropic-convert-route
+ uris:
+ - /v1/messages
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${BACKEND_API_KEY}"
+ options:
+ model: gpt-4
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="anthropic-convert-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: anthropic-convert-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /v1/messages
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="anthropic-convert-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: anthropic-convert-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: anthropic-convert-route
+ match:
+ paths:
+ - /v1/messages
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+```
+
+</TabItem>
+
+</Tabs>
+
+将配置应用到集群：
+
+```shell
+kubectl apply -f anthropic-convert-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
+以 Anthropic Messages API 格式向路由发送 POST 请求：
+
+```shell
+curl "http://127.0.0.1:9080/v1/messages"; -X POST \
+ -H "Content-Type: application/json" \
+ -H "x-api-key: ${BACKEND_API_KEY}" \
+ -H "anthropic-version: 2023-06-01" \
+ -d '{
+ "model": "gpt-4",
+ "max_tokens": 1024,
+ "messages": [
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+尽管请求以 Anthropic 格式发送，但它将自动转换为 OpenAI 格式并转发到后端。响应将转换回 Anthropic 格式：
+
+```json
+{
+ "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
+ "type": "message",
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "1+1 equals 2."
+ }
+ ],
+ "model": "gpt-4",
+ "stop_reason": "end_turn",
+ "usage": {
+ "input_tokens": 12,
+ "output_tokens": 8
+ }
+}
+```
+
+该插件支持 Anthropic Messages API 的所有功能，包括流式传输 (SSE)、系统提示和工具使用（函数调用）。协议转换透明地处理
Anthropic 和 OpenAI 格式之间的双向映射。
+
+### 使用请求体参数代理到选定模型
+
+以下示例演示了如何基于用户请求中指定的模型参数，在同一 URI 上将请求代理到不同的模型。你可以使用 `post_arg.*` 变量来获取请求体参数的值。
+
+此示例将使用 OpenAI 和 DeepSeek 作为示例 LLM 服务。获取 OpenAI 和 DeepSeek API 密钥并保存到环境变量：
+
+```shell
+export OPENAI_API_KEY=<your-api-key>
+export DEEPSEEK_API_KEY=<your-api-key>
+```
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+创建一个到 OpenAI API 的路由，使用 `vars` 匹配请求体参数 `model` 为 `openai` 的请求：
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-openai-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "vars": [[ "post_arg.model", "==", "openai" ]],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$OPENAI_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "gpt-4"
+ }
+ }
+ }
+ }'
+```
+
+创建另一个到 DeepSeek API 的路由 `/anything`，使用 `vars` 匹配请求体参数 `model` 为 `deepseek` 的请求：
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-deepseek-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "vars": [[ "post_arg.model", "==", "deepseek" ]],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "deepseek",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$DEEPSEEK_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "deepseek-chat"
+ }
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+创建两个包含 `ai-proxy` 插件的路由，分别配置不同的提供商。使用 `vars` 匹配请求体参数 `model` 来决定路由到哪个提供商：
+
+```yaml title="adc.yaml"
+services:
+ - name: multi-model-service
+ routes:
+ - name: openai-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ vars:
+ - - post_arg.model
+ - ==
+ - openai
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+ - name: deepseek-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ vars:
+ - - post_arg.model
+ - ==
+ - deepseek
+ plugins:
+ ai-proxy:
+ provider: deepseek
+ auth:
+ header:
+ Authorization: "Bearer ${DEEPSEEK_API_KEY}"
+ options:
+ model: deepseek-chat
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+:::info
+
+HTTPRoute 不支持请求体参数匹配。支持的匹配机制为 `path`、`method`、`headers` 和
`queryParams`。此示例无法使用 Gateway API 完成。
+
+:::
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+:::info
+
+ApisixRoute 当前不支持请求体参数匹配。支持的匹配机制基于 `Header`、`Query` 或 `Path`。此示例无法使用 APISIX
CRD 完成。
+
+:::
+
+</TabItem>
+
+</Tabs>
+
+</TabItem>
+
+</Tabs>
+
+向路由发送 POST 请求，将 `model` 设置为 `openai`：
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "openai",
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+你应该收到类似以下的响应：
+
+```json
+{
+ ...,
+ "model": "gpt-4-0613",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "1+1 equals 2.",
+ "refusal": null
+ },
+ "logprobs": null,
+ "finish_reason": "stop"
+ }
+ ],
+ ...
+}
+```
+
+向路由发送 POST 请求，将 `model` 设置为 `deepseek`：
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "deepseek",
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+你应该收到类似以下的响应：
+
+```json
+{
+ ...,
+ "model": "deepseek-chat",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "The sum of 1 and 1 is 2. This is a basic arithmetic
operation where you combine two units to get a total of two units."
+ },
+ "logprobs": null,
+ "finish_reason": "stop"
+ }
+ ],
+ ...
+}
+```
+
+你还可以配置 `post_arg.*` 来获取嵌套的请求体参数。例如，如果请求格式为：
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": {
+ "name": "openai"
+ },
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+你可以将路由上的 `vars` 配置为 `[[ "post_arg.model.name", "==", "openai" ]]`。
+
+### 发送请求日志到日志记录器
+
+以下示例演示了如何记录请求和响应信息（包括 LLM 模型、令牌和负载），并将其推送到日志记录器。在开始之前，你应该先设置一个日志记录器，例如
Kafka。有关更多信息，请参阅 [`kafka-logger`](./kafka-logger.md)。
+
+<Tabs
+groupId="api"
+defaultValue="admin-api"
+values={[
+{label: 'Admin API', value: 'admin-api'},
+{label: 'ADC', value: 'adc'},
+{label: 'Ingress Controller', value: 'aic'}
+]}>
+
+<TabItem value="admin-api">
+
+创建到 LLM 服务的路由，并配置日志详情。将 `logging.summaries` 设置为 `true` 以记录请求 LLM
模型、持续时间、请求和响应令牌，将 `logging.payloads` 设置为 `true` 以记录请求和响应负载。同时配置 `kafka-logger`
插件，设置 Kafka 地址、主题、密钥，并将 `batch_max_size` 设置为 `1` 以立即发送日志条目：
+
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "ai-proxy-openai-route",
+ "uri": "/anything",
+ "methods": ["POST"],
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai",
+ "auth": {
+ "header": {
+ "Authorization": "Bearer '"$OPENAI_API_KEY"'"
+ }
+ },
+ "options": {
+ "model": "gpt-4"
+ },
+ "logging": {
+ "summaries": true,
+ "payloads": true
+ }
+ },
+ "kafka-logger": {
+ "brokers": [
+ {
+ "host": "127.0.0.1",
+ "port": 9092
+ }
+ ],
+ "kafka_topic": "test2",
+ "key": "key1",
+ "batch_max_size": 1
+ }
+ }
+ }'
+```
+
+</TabItem>
+
+<TabItem value="adc">
+
+创建包含 `ai-proxy` 和 `kafka-logger` 插件的路由。将 `logging.summaries` 设置为 `true` 以记录请求
LLM 模型、持续时间、请求和响应令牌，将 `logging.payloads` 设置为 `true` 以记录请求和响应负载：
+
+```yaml title="adc.yaml"
+services:
+ - name: logging-service
+ routes:
+ - name: logging-route
+ uris:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ ai-proxy:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer ${OPENAI_API_KEY}"
+ options:
+ model: gpt-4
+ logging:
+ summaries: true
+ payloads: true
+ kafka-logger:
+ brokers:
+ - host: 127.0.0.1
+ port: 9092
+ kafka_topic: test2
+ key: key1
+ batch_max_size: 1
+```
+
+将配置同步到网关：
+
+```shell
+adc sync -f adc.yaml
+```
+
+</TabItem>
+
+<TabItem value="aic">
+
+<Tabs
+groupId="k8s-api"
+defaultValue="gateway-api"
+values={[
+{label: 'Gateway API', value: 'gateway-api'},
+{label: 'APISIX CRD', value: 'apisix-crd'}
+]}>
+
+<TabItem value="gateway-api">
+
+```yaml title="logging-ic.yaml"
+apiVersion: apisix.apache.org/v1alpha1
+kind: PluginConfig
+metadata:
+ namespace: aic
+ name: ai-proxy-logging-plugin-config
+spec:
+ plugins:
+ - name: ai-proxy
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+ logging:
+ summaries: true
+ payloads: true
+ - name: kafka-logger
+ config:
+ brokers:
+ - host: kafka.aic.svc.cluster.local
+ port: 9092
+ kafka_topic: test2
+ key: key1
+ batch_max_size: 1
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ namespace: aic
+ name: logging-route
+spec:
+ parentRefs:
+ - name: apisix
+ rules:
+ - matches:
+ - path:
+ type: Exact
+ value: /anything
+ method: POST
+ filters:
+ - type: ExtensionRef
+ extensionRef:
+ group: apisix.apache.org
+ kind: PluginConfig
+ name: ai-proxy-logging-plugin-config
+```
+
+</TabItem>
+
+<TabItem value="apisix-crd">
+
+```yaml title="logging-ic.yaml"
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+ namespace: aic
+ name: logging-route
+spec:
+ ingressClassName: apisix
+ http:
+ - name: logging-route
+ match:
+ paths:
+ - /anything
+ methods:
+ - POST
+ plugins:
+ - name: ai-proxy
+ enable: true
+ config:
+ provider: openai
+ auth:
+ header:
+ Authorization: "Bearer your-api-key"
+ options:
+ model: gpt-4
+ logging:
+ summaries: true
+ payloads: true
+ - name: kafka-logger
+ enable: true
+ config:
+ brokers:
+ - host: kafka.aic.svc.cluster.local
+ port: 9092
+ kafka_topic: test2
+ key: key1
+ batch_max_size: 1
+```
+
+</TabItem>
+
+</Tabs>
+
+将配置应用到集群：
+
+```shell
+kubectl apply -f logging-ic.yaml
+```
+
+</TabItem>
+
+</Tabs>
+
+向路由发送 POST 请求：
+
+```shell
+curl "http://127.0.0.1:9080/anything"; -X POST \
+ -H "Content-Type: application/json" \
+ -d '{
+ "messages": [
+ { "role": "system", "content": "You are a mathematician" },
+ { "role": "user", "content": "What is 1+1?" }
+ ]
+ }'
+```
+
+你应该收到类似以下的响应：
+
+```json
+{
+ ...,
+ "model": "gpt-4-0613",
+ "choices": [
+ {
+ "index": 0,
+ "message": {
+ "role": "assistant",
+ "content": "1+1 equals 2.",
+ "refusal": null
+ },
+ "logprobs": null,
+ "finish_reason": "stop"
+ }
+ ],
+ ...
+}
+```
+
+在 Kafka 主题中，你应该还会看到与请求对应的日志条目，其中包含 LLM 摘要和请求/响应负载。
+
### 在访问日志中包含 LLM 信息
以下示例演示了如何在网关的访问日志中记录 LLM 请求相关信息，以改进分析和审计。以下变量可用：
@@ -455,7 +1965,7 @@ nginx_config:
重新加载 APISIX 以使配置更改生效。
-现在，如果您创建路由并按照[代理到 OpenAI 示例](#代理到-openai)发送请求，您应该收到类似以下的响应：
+现在，如果你创建路由并按照[代理到 OpenAI 示例](#代理到-openai)发送请求，你应该收到类似以下的响应：
```json
{
@@ -489,7 +1999,7 @@ nginx_config:
}
```
-在网关的访问日志中，您应该看到类似以下的日志条目：
+在网关的访问日志中，你应该看到类似以下的日志条目：
```text
192.168.215.1 - - [21/Mar/2025:04:28:03 +0000] api.openai.com "POST /anything
HTTP/1.1" 200 804 2.858 "-" "curl/8.6.0" - - - 5765 "http://api.openai.com";
"5c5e0b95f8d303cb81e4dc456a4b12d9" "ai_chat" "2858" "gpt-4" "gpt-4" "23" "8"
