Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway milestones ok dubbogo也是支持probe的 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-17209552 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user mochengqian edited a comment on the discussion: [AI] AI gateway
milestones
** @zerui 让我学习pixiu竞品的优秀cluster设计.** 下面的你们看看合理、需要吗 @Alanxtl @AlexStocks
**我的prompt:**“我的目标是深度研究 Apache dubbo-go-pixiu 的“集群模块”优化方向。请你不要只做泛泛竞品分析，而是以“如何让
Kong / APISIX / Envoy AI Gateway / Higress / LiteLLM / Portkey / TrueFoundry
等成熟 AI 智能网关的高频重度用户，看到 Pixiu 的集群模块后愿意称赞并迁移”为最终目标，进行系统性研究。”
**得出优化建议:** 把 Pixiu cluster 从"静态配置的 upstream 容器"升级成"可编程的多协议统一运行时"——让 AI
provider、Dubbo service、K8s pod、MCP tool 这些异构后端都先编译成统一的 immutable snapshot +
mutable state，然后用同一套 picker/scorer/balancer 做智能选址（cost-aware / quota-aware /
failure-classified），最终实现"一次编写负载均衡逻辑，到处复用"。
# [RFC] 统一多协议 Cluster Runtime：从 Upstream 容器到可编程运行时
> 基于《Pixiu 集群模块深度研究报告》与 PR #932 现状，提出将 cluster 从"通用 upstream
> 容器"演进为"多协议、多形态后端统一运行时"的完整技术路线。本 RFC 先在 Discussion #696 试水，社区有兴趣后提交为正式 issue。
> **评审更新（2026-06-07，@Alanxtl 三点反馈，已采纳）：**
> 1. **两层分离 / sync.Map 取消。** 不再用 `Cluster.states sync.Map` 这个与 snapshot
> 并行的独立结构（它看起来像在读路径上重新引入锁、且需要手动同步生命周期）。改为把 `stateByID
> map[string]*EndpointState` **直接放进 immutable snapshot**，重建时按 endpoint ID
> **继承指针**——和现在 `addressByID` 按引用共享、health 按 ID 继承是同一套手法。无并行结构、无新锁、热路径只多一次
> atomic load，state GC 由可达性自动完成。
> 2. **加权 scorer 取消。** persona 调研结论：真实用户是自用的平台 / 自托管推理团队，不是做成本套利的二级分发商。`pkg/` 里
> cost/quota/budget 零踪迹，#764 已把 provider
> 业务逻辑明确踢出网关核心。因此**不内置任何加权打分模型**；选址保持二元（cooldown 就跳过）；未来若做打分，persona B 的目标是
> load/latency（现有 `LoadBalancerStrategy` 加一个策略即可），cost 模型一律做成**可插拔接口让 operator
> 自己实现**。
> 3. **Multi-Kind 健康检查 = 接口 + 内置默认 + 允许 override（Envoy 模型）。** static 用现有 HTTP
> probe、K8s 读 pod readiness、Dubbo 用心跳，都放在一个 operator
> 可覆盖的默认实现后面。不是强制用户自己实现，也不是写死一种。（此前 open-question #5 现已定调。）
## 执行摘要（30 秒版）
**现状**：cluster 是静态配置的 upstream 容器，AI 失败状态（cooldown）活在 filter 的全局 map 里，picker
看不到；Dubbo/K8s/MCP 各自独立实现，无法复用选址逻辑。
**目标**：把 cluster 升级成**统一运行时** —— 静态 cluster、Dubbo service cluster、Kubernetes
service cluster、AI provider cluster、MCP service cluster，全部先编译成 immutable
runtime snapshot，再由统一 selector/scorer/picker 执行选址。
**路径**：三层渐进演进（每层可独立验收、向后兼容）
1. **State 基底**（Phase 1）— per-endpoint
`EndpointState`（cooldown/inflight/failure-class），解决 #696 的 key-level cooldown 问题
2. **Load-aware 选址 + 可插拔打分**（Phase 2）— load 字段（inflight/EWMA）进 state，load-aware
LB 策略 + 可插拔 `Scorer` 接口进 picker；**不内置 cost/quota 加权模型**
3. **Multi-Kind 统一**（Phase 3）— `ClusterKind` 接口，Dubbo/K8s/MCP 各实现
`CompileSnapshot`，picker 只看统一 snapshot
**关键设计点**：immutable snapshot（沿用 #932）+ per-endpoint state 指针**内嵌于 snapshot、按 ID
跨重建继承**（atomic 原地更新）—— 解决"快变状态 vs 零分配读路径"的张力，且不引入并行结构或新锁。
---
## 动机：为什么要动 cluster core？
### 当前痛点（Discussion #696 + Issue #905）
1. **AI 失败状态不可见**
`sharedCooldownStore` 在 filter
侧（`pkg/filter/llm/proxy/filter.go:138`），picker 看不到 cooldown。单个 API key 触发
429，整个 endpoint 被 filter 循环跳过，但对 picker 来说 endpoint 仍"健康" —— 健康与可用脱节。
2. **二元健康无法表达失败语义**
`context_length_exceeded` 是请求/endpoint **不匹配**（该请求需要更大 context 模型），不是
endpoint 故障 —— 正确反应是路由到大 context endpoint，而非 cooldown。Binary healthy/unhealthy
表达不了这点。
3. **多协议后端各自为政**
静态 upstream（现有）、Dubbo service（规划中）、K8s service（规划中）、MCP tool（规划中）各自独立实现，无法共享
LB/健康检查/选址逻辑。每加一种后端就复制一套代码。
### 为什么要统一？（报告的核心洞察）
**观察**：AI provider、Dubbo service、K8s pod、MCP tool —— 表面看完全不同，但**选址决策的输入是同构的**：
- 都有"成员列表"（endpoints / services / pods / tools）
- 都有"健康状态"（provider 429 / service down / pod terminating / tool unavailable）
- 都有"负载指标"（inflight / qps / cpu / concurrent calls）
- 都有"能力标签"（model supports / service version / pod labels / tool schema）
**结论**：如果把这些异构后端先**编译**成统一的 runtime view（immutable snapshot + mutable
state），picker 就可以**一次编写、到处复用**。这是 Envoy `Host` 抽象、Istio `ServiceEntry` 统一的同一个思路。
**收益**：
- 新协议后端只需实现"config → snapshot 编译器"，自动继承现有 LB/健康检查/metrics
- load-aware / cooldown-aware 选址，天然可用于 Dubbo（按并发/延迟）、K8s（按 pod 负载）
- 一套 benchmark/test 覆盖所有 cluster kind
---
## 现状分析（origin/develop as of 2026-06-03）
### Cluster Runtime（pkg/cluster/cluster.go）
```go
type Cluster struct {
endpoints atomic.Pointer[EndpointSnapshot] // Line 43
// ...
}
```
- PR #932 已合并：`EndpointSnapshot` immutable，通过 `atomic.Pointer` 发布
- 成员/健康变化时 CAS 重建（`RefreshEndpointsFrom:95`, `UpdateEndpointHealth:109`）
- 预计算 `all` / `healthy` / `byID` / `byAddress` 索引 → 读路径 ~0 alloc
### AI Failure State（pkg/filter/llm/proxy/filter.go）
```go
var sharedCooldownStore = newCooldownStore() // Line 138
```
- 进程级全局 map：`map[cooldownKey]cooldownEntry`，`sync.Mutex` 保护
- 容量上限 1024，达到 load factor 后触发 sweep
- Filter fallback 循环里检查 cooldown，picker **看不到**
### 问题
AI 失败状态变化频率（per-request）与 health 变化频率（health-check interval）相差 3 个数量级。如果把
cooldown 塞进 immutable snapshot，每次失败都触发 O(N) 重建 + map 分配 → 毁掉 #932 的零分配读路径。
---
## 核心设计：两层分离 + 编译时抽象
### 设计张力的解法
**问题重述**：snapshot 不可变（#932 收益），但 AI 状态高频变（per-request）。
**解法**：分两层，不同变化频率用不同策略
1. **Immutable membership snapshot**（慢变）— 成员 + 健康 + 索引，仍用 #932 的 CAS 重建
2. **Per-endpoint `EndpointState`**（快变）— 通过**内嵌于 snapshot 的 `stateByID`
索引**访问；重建时按 endpoint ID **继承 `*EndpointState` 指针**，字段用 atomic 原地更新
Snapshot 重建时，ID 不变的
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user mochengqian added a comment to the discussion: [AI] AI gateway milestones good catch! @Alanxtl 1. 思考后发现确实没必要,您的建议将采纳 2. Scorer 这个确实没啥必要,您的建议将采纳. 3. Multi-Kind 健康检查 ，我的倾向是两者兼顾吧: 接口 + 内置默认实现、允许 override，参考的Envoy那套：static 用现有 HTTP probe、K8s 读 pod readiness、Dubbo 用心跳，都放在一个 operator 可覆盖的默认后面。不是必须用户自己实现，也不是我们写死一种。 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-17207604 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [x] 响应时间 - [x] 响应合规性和请求合规性 - [x] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway milestones 其他的我觉得没问题挺好的 我有两个疑惑的点 1. 两层分离的endpoint 有必要吗？本来endpoint是没有snapshot的 然后加了snapshot导致不能实时更新 现在又要引入一个没有snapshot的第二层 那这个第二层的读取依然要加锁 无锁读取依然实现不了 感觉就很多余 2. 关于 Scorer 这个东西，我觉得得先明确一下现在我们ai-gateway的用户群体是谁 如果是面向给下游做二级分发的人 这个东西完全可以让使用者自己定义 我们再弄一个加权模型就感觉不太聪明的样子 如果是面向普通llm使用者那我觉得就用healthy这个指标就够了呀 3. Multi-Kind 统一 这个事情我觉得很好 但是 “多 cluster kind 的健康检查如何统一” 这个事情我觉得还得再考虑一下 想一下这个是应该给使用者开放接口让他们自己实现 还是我们内置实现 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-17206755 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user mochengqian edited a comment on the discussion: [AI] AI gateway
milestones
** @zerui 让我学习pixiu竞品的优秀cluster设计.** 下面的你们看看合理、需要吗 @Alanxtl @AlexStocks
**我的prompt:**“我的目标是深度研究 Apache dubbo-go-pixiu 的“集群模块”优化方向。请你不要只做泛泛竞品分析，而是以“如何让
Kong / APISIX / Envoy AI Gateway / Higress / LiteLLM / Portkey / TrueFoundry
等成熟 AI 智能网关的高频重度用户，看到 Pixiu 的集群模块后愿意称赞并迁移”为最终目标，进行系统性研究。”
**得出优化建议:** 把 Pixiu cluster 从"静态配置的 upstream 容器"升级成"可编程的多协议统一运行时"——让 AI
provider、Dubbo service、K8s pod、MCP tool 这些异构后端都先编译成统一的 immutable snapshot +
mutable state，然后用同一套 picker/scorer/balancer 做智能选址（cost-aware / quota-aware /
failure-classified），最终实现"一次编写负载均衡逻辑，到处复用"。
# [RFC] 统一多协议 Cluster Runtime：从 Upstream 容器到可编程运行时
> 基于《Pixiu 集群模块深度研究报告》与 PR #932 现状，提出将 cluster 从"通用 upstream
> 容器"演进为"多协议、多形态后端统一运行时"的完整技术路线。本 RFC 先在 Discussion #696 试水，社区有兴趣后提交为正式 issue。
## 执行摘要（30 秒版）
**现状**：cluster 是静态配置的 upstream 容器，AI 失败状态（cooldown）活在 filter 的全局 map 里，picker
看不到；Dubbo/K8s/MCP 各自独立实现，无法复用选址逻辑。
**目标**：把 cluster 升级成**统一运行时** —— 静态 cluster、Dubbo service cluster、Kubernetes
service cluster、AI provider cluster、MCP service cluster，全部先编译成 immutable
runtime snapshot，再由统一 selector/scorer/picker 执行选址。
**路径**：三层渐进演进（每层可独立验收、向后兼容）
1. **State 基底**（Phase 1）— per-endpoint
`EndpointState`（cooldown/inflight/failure-class），解决 #696 的 key-level cooldown 问题
2. **AI Runtime 填满**（Phase 2）— cost/quota/model-capability 进
state，scorer/semantic router 进 picker
3. **Multi-Kind 统一**（Phase 3）— `ClusterKind` 接口，Dubbo/K8s/MCP 各实现
`CompileSnapshot`，picker 只看统一 snapshot
**关键设计点**：immutable snapshot（沿用 #932）+ stable per-endpoint state（跨 snapshot
存活、atomic 更新）—— 解决"快变状态 vs 零分配读路径"的张力。
---
## 动机：为什么要动 cluster core？
### 当前痛点（Discussion #696 + Issue #905）
1. **AI 失败状态不可见**
`sharedCooldownStore` 在 filter
侧（`pkg/filter/llm/proxy/filter.go:138`），picker 看不到 cooldown。单个 API key 触发
429，整个 endpoint 被 filter 循环跳过，但对 picker 来说 endpoint 仍"健康" —— 健康与可用脱节。
2. **二元健康无法表达失败语义**
`context_length_exceeded` 是请求/endpoint **不匹配**（该请求需要更大 context 模型），不是
endpoint 故障 —— 正确反应是路由到大 context endpoint，而非 cooldown。Binary healthy/unhealthy
表达不了这点。
3. **多协议后端各自为政**
静态 upstream（现有）、Dubbo service（规划中）、K8s service（规划中）、MCP tool（规划中）各自独立实现，无法共享
LB/健康检查/选址逻辑。每加一种后端就复制一套代码。
### 为什么要统一？（报告的核心洞察）
**观察**：AI provider、Dubbo service、K8s pod、MCP tool —— 表面看完全不同，但**选址决策的输入是同构的**：
- 都有"成员列表"（endpoints / services / pods / tools）
- 都有"健康状态"（provider 429 / service down / pod terminating / tool unavailable）
- 都有"负载指标"（inflight / qps / cpu / concurrent calls）
- 都有"能力标签"（model supports / service version / pod labels / tool schema）
**结论**：如果把这些异构后端先**编译**成统一的 runtime view（immutable snapshot + mutable
state），picker 就可以**一次编写、到处复用**。这是 Envoy `Host` 抽象、Istio `ServiceEntry` 统一的同一个思路。
**收益**：
- 新协议后端只需实现"config → snapshot 编译器"，自动继承现有 LB/健康检查/metrics
- AI 的 cost-aware/quota-aware 选址，天然可用于 Dubbo（按 CPU quota）、K8s（按 pod 资源）
- 一套 benchmark/test 覆盖所有 cluster kind
---
## 现状分析（origin/develop as of 2026-06-03）
### Cluster Runtime（pkg/cluster/cluster.go）
```go
type Cluster struct {
endpoints atomic.Pointer[EndpointSnapshot] // Line 43
// ...
}
```
- PR #932 已合并：`EndpointSnapshot` immutable，通过 `atomic.Pointer` 发布
- 成员/健康变化时 CAS 重建（`RefreshEndpointsFrom:95`, `UpdateEndpointHealth:109`）
- 预计算 `all` / `healthy` / `byID` / `byAddress` 索引 → 读路径 ~0 alloc
### AI Failure State（pkg/filter/llm/proxy/filter.go）
```go
var sharedCooldownStore = newCooldownStore() // Line 138
```
- 进程级全局 map：`map[cooldownKey]cooldownEntry`，`sync.Mutex` 保护
- 容量上限 1024，达到 load factor 后触发 sweep
- Filter fallback 循环里检查 cooldown，picker **看不到**
### 问题
AI 失败状态变化频率（per-request）与 health 变化频率（health-check interval）相差 3 个数量级。如果把
cooldown 塞进 immutable snapshot，每次失败都触发 O(N) 重建 + map 分配 → 毁掉 #932 的零分配读路径。
---
## 核心设计：两层分离 + 编译时抽象
### 设计张力的解法
**问题重述**：snapshot 不可变（#932 收益），但 AI 状态高频变（per-request）。
**解法**：分两层，不同变化频率用不同策略
1. **Immutable membership snapshot**（慢变）— 成员 + 健康 + 索引，仍用 #932 的 CAS 重建
2. **Stable per-endpoint `EndpointState`**（快变）— 按 endpoint ID 持有、跨 snapshot
重建存活、字段用 atomic 原地更新
Snapshot 重建时，ID 不变的 endpoint **复用同一个 `EndpointState` 指针** —— 新 snapshot 继承老
state，零重建。
```go
// Phase 1 sketch — not final API
type EndpointState struct {
// Atomic fields — per-request update, no allocation
cooldownUntilNano atomic.Int64
lastFailureClass atomic.Int32 // FailureClass enum
inflight atomic.Int64
// Phase 2 additions (后续)
// costPerToken atomic.Uint64 // fixed-point
// quotaRemainingatomic.Int64
// ewmaLatencyNs atomic.Int64
}
type Cluster struct {
endpoints atomic.Pointer[EndpointSnapshot] // immutable view
statessync.Map // map[endpointID]*EndpointState, stable
}
```
Picker 读 snapshot 拿 healthy 列表（零分配），遍历时查 `states[ep.ID]` 拿 cooldown（一次
atomic.Load）。
**这是本 RFC 最需要评审的设计**——它保持 #932 不变量、解决快变状态问题，但引入了"snapshot + state
双查"的复杂度。备选方案见后文。
### 失败分类（Failure Classifier）
报告识别的核心洞察：**不是所有失败都该 cooldown endpoint**。
| FailureClass| 触发条件
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user mochengqian added a comment to the discussion: [AI] AI gateway milestones 感谢指出！我的表述不够精确。Picker 确实能看到 Endpoint.UnHealthy（健康检查结果，通过 snapshot.healthy 列表），但看不到 sharedCooldownStore 里的 AI 失败 cooldown。现在 filter 在 pick 之后、调用之前检查 cooldown，不 match 就 PickNextEndpoint 重来,确实有问题. GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-17164119 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user mochengqian added a comment to the discussion: [AI] AI gateway
milestones
**zerui yang让我学习pixiu竞品的优秀cluster设计.** 你们看看合理、需要吗
**我的prompt:**“我的目标是深度研究 Apache dubbo-go-pixiu 的“集群模块”优化方向。请你不要只做泛泛竞品分析，而是以“如何让
Kong / APISIX / Envoy AI Gateway / Higress / LiteLLM / Portkey / TrueFoundry
等成熟 AI 智能网关的高频重度用户，看到 Pixiu 的集群模块后愿意称赞并迁移”为最终目标，进行系统性研究。”
**得出优化建议:** 把 Pixiu cluster 从"静态配置的 upstream 容器"升级成"可编程的多协议统一运行时"——让 AI
provider、Dubbo service、K8s pod、MCP tool 这些异构后端都先编译成统一的 immutable snapshot +
mutable state，然后用同一套 picker/scorer/balancer 做智能选址（cost-aware / quota-aware /
failure-classified），最终实现"一次编写负载均衡逻辑，到处复用"。
# [RFC] 统一多协议 Cluster Runtime：从 Upstream 容器到可编程运行时
> 基于《Pixiu 集群模块深度研究报告》与 PR #932 现状，提出将 cluster 从"通用 upstream
> 容器"演进为"多协议、多形态后端统一运行时"的完整技术路线。本 RFC 先在 Discussion #696 试水，社区有兴趣后提交为正式 issue。
## 执行摘要（30 秒版）
**现状**：cluster 是静态配置的 upstream 容器，AI 失败状态（cooldown）活在 filter 的全局 map 里，picker
看不到；Dubbo/K8s/MCP 各自独立实现，无法复用选址逻辑。
**目标**：把 cluster 升级成**统一运行时** —— 静态 cluster、Dubbo service cluster、Kubernetes
service cluster、AI provider cluster、MCP service cluster，全部先编译成 immutable
runtime snapshot，再由统一 selector/scorer/picker 执行选址。
**路径**：三层渐进演进（每层可独立验收、向后兼容）
1. **State 基底**（Phase 1）— per-endpoint
`EndpointState`（cooldown/inflight/failure-class），解决 #696 的 key-level cooldown 问题
2. **AI Runtime 填满**（Phase 2）— cost/quota/model-capability 进
state，scorer/semantic router 进 picker
3. **Multi-Kind 统一**（Phase 3）— `ClusterKind` 接口，Dubbo/K8s/MCP 各实现
`CompileSnapshot`，picker 只看统一 snapshot
**关键设计点**：immutable snapshot（沿用 #932）+ stable per-endpoint state（跨 snapshot
存活、atomic 更新）—— 解决"快变状态 vs 零分配读路径"的张力。
---
## 动机：为什么要动 cluster core？
### 当前痛点（Discussion #696 + Issue #905）
1. **AI 失败状态不可见**
`sharedCooldownStore` 在 filter
侧（`pkg/filter/llm/proxy/filter.go:138`），picker 看不到 cooldown。单个 API key 触发
429，整个 endpoint 被 filter 循环跳过，但对 picker 来说 endpoint 仍"健康" —— 健康与可用脱节。
2. **二元健康无法表达失败语义**
`context_length_exceeded` 是请求/endpoint **不匹配**（该请求需要更大 context 模型），不是
endpoint 故障 —— 正确反应是路由到大 context endpoint，而非 cooldown。Binary healthy/unhealthy
表达不了这点。
3. **多协议后端各自为政**
静态 upstream（现有）、Dubbo service（规划中）、K8s service（规划中）、MCP tool（规划中）各自独立实现，无法共享
LB/健康检查/选址逻辑。每加一种后端就复制一套代码。
### 为什么要统一？（报告的核心洞察）
**观察**：AI provider、Dubbo service、K8s pod、MCP tool —— 表面看完全不同，但**选址决策的输入是同构的**：
- 都有"成员列表"（endpoints / services / pods / tools）
- 都有"健康状态"（provider 429 / service down / pod terminating / tool unavailable）
- 都有"负载指标"（inflight / qps / cpu / concurrent calls）
- 都有"能力标签"（model supports / service version / pod labels / tool schema）
**结论**：如果把这些异构后端先**编译**成统一的 runtime view（immutable snapshot + mutable
state），picker 就可以**一次编写、到处复用**。这是 Envoy `Host` 抽象、Istio `ServiceEntry` 统一的同一个思路。
**收益**：
- 新协议后端只需实现"config → snapshot 编译器"，自动继承现有 LB/健康检查/metrics
- AI 的 cost-aware/quota-aware 选址，天然可用于 Dubbo（按 CPU quota）、K8s（按 pod 资源）
- 一套 benchmark/test 覆盖所有 cluster kind
---
## 现状分析（origin/develop as of 2026-06-03）
### Cluster Runtime（pkg/cluster/cluster.go）
```go
type Cluster struct {
endpoints atomic.Pointer[EndpointSnapshot] // Line 43
// ...
}
```
- PR #932 已合并：`EndpointSnapshot` immutable，通过 `atomic.Pointer` 发布
- 成员/健康变化时 CAS 重建（`RefreshEndpointsFrom:95`, `UpdateEndpointHealth:109`）
- 预计算 `all` / `healthy` / `byID` / `byAddress` 索引 → 读路径 ~0 alloc
### AI Failure State（pkg/filter/llm/proxy/filter.go）
```go
var sharedCooldownStore = newCooldownStore() // Line 138
```
- 进程级全局 map：`map[cooldownKey]cooldownEntry`，`sync.Mutex` 保护
- 容量上限 1024，达到 load factor 后触发 sweep
- Filter fallback 循环里检查 cooldown，picker **看不到**
### 问题
AI 失败状态变化频率（per-request）与 health 变化频率（health-check interval）相差 3 个数量级。如果把
cooldown 塞进 immutable snapshot，每次失败都触发 O(N) 重建 + map 分配 → 毁掉 #932 的零分配读路径。
---
## 核心设计：两层分离 + 编译时抽象
### 设计张力的解法
**问题重述**：snapshot 不可变（#932 收益），但 AI 状态高频变（per-request）。
**解法**：分两层，不同变化频率用不同策略
1. **Immutable membership snapshot**（慢变）— 成员 + 健康 + 索引，仍用 #932 的 CAS 重建
2. **Stable per-endpoint `EndpointState`**（快变）— 按 endpoint ID 持有、跨 snapshot
重建存活、字段用 atomic 原地更新
Snapshot 重建时，ID 不变的 endpoint **复用同一个 `EndpointState` 指针** —— 新 snapshot 继承老
state，零重建。
```go
// Phase 1 sketch — not final API
type EndpointState struct {
// Atomic fields — per-request update, no allocation
cooldownUntilNano atomic.Int64
lastFailureClass atomic.Int32 // FailureClass enum
inflight atomic.Int64
// Phase 2 additions (后续)
// costPerToken atomic.Uint64 // fixed-point
// quotaRemainingatomic.Int64
// ewmaLatencyNs atomic.Int64
}
type Cluster struct {
endpoints atomic.Pointer[EndpointSnapshot] // immutable view
statessync.Map // map[endpointID]*EndpointState, stable
}
```
Picker 读 snapshot 拿 healthy 列表（零分配），遍历时查 `states[ep.ID]` 拿 cooldown（一次
atomic.Load）。
**这是本 RFC 最需要评审的设计**——它保持 #932 不变量、解决快变状态问题，但引入了"snapshot + state
双查"的复杂度。备选方案见后文。
### 失败分类（Failure Classifier）
报告识别的核心洞察：**不是所有失败都该 cooldown endpoint**。
| FailureClass| 触发条件 | 反应
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [x] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [x] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [ ] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [x] 响应时间 - [ ] 响应合规性和请求合规性 - [ ] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [x] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [x] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [x] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [ ] 日志 - [x] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [x] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [ ] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [ ] 响应合规性和请求合规性 - [x] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [ ] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [x] 响应时间 - [ ] 响应合规性和请求合规性 - [ ] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ Step1: 基础 - [x] 基础 AI 协议支持 (SSE/HTTP) #657 - [x] Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] Token 用量获取 #659 Step2: 兼容 - [x] 面向密钥的负载均衡 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512) - [x] 密钥的健康性检测 [apache/dubbo-go-pixiu#731](https://github.com/apache/dubbo-go-pixiu/pull/731) [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522) - [x] fallback策略 #685 #692 - [x] retry策略 #685 #692 - [x] LLM 供应商的兼容 #678(需持续适配) - [x] OpenAI 请求适配层，该功能不由 pixiu 网关层面实现 [apache/dubbo-go-pixiu#696 (comment)](https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597) - [x] LLM服务发现 (Nacos 注册中心) #746 Step2.1: 遗留问题 - [x] 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 Step3: 完善 - [x] AI 可观测性 [apache/dubbo-go-pixiu#733](https://github.com/apache/dubbo-go-pixiu/pull/733) - [ ] Token 用量 - [ ] 响应时间 - [x] 响应合规性和请求合规性 - [ ] 日志 - [ ] 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway milestones 啥叫“跳过处于 cooldown 的 endpoint”这件事从来没进入选址决策” 根据我的理解 picker得到EndpointSnapshot 会遍历这些endpoint 访问是否healthy healthy这个变量应该不是Snapshot吧 访问是否healthy直接获得的就是现在是否healthy（这一点确定一下，我不是很确定，如果是这样的话那你说的这些感觉就不需要做） GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-17163227 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [x] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [x] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [x] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [x] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [x] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [x] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [x] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [x] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones None GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user mochengqian added a comment to the discussion: [AI] AI gateway milestones **承接本帖的方向：把 AI 失败状态下沉进 cluster picker** 接着本帖里 key 级 cooldown 的讨论（某个 API key 触发 429 不应把整个 endpoint 标成不健康）——我想先探一下大家对这个方向上一个小而聚焦的步骤是否有兴趣。 **当前现状。** #932 之后，cluster 已经有了通过 `atomic.Pointer` 发布的不可变 `EndpointSnapshot`（成员 + 健康），但 picker 选址只看健康状态。AI 失败状态（cooldown）目前活在 LLM filter 的进程级全局 map 里（`pkg/filter/llm/proxy/filter.go` 的 `sharedCooldownStore`），picker 看不到它。所以 fallback 是 filter 侧的一个循环，“跳过处于 cooldown 的 endpoint”这件事从来没进入选址决策。 **建议方向——做的是状态基底，不是新算法。** 对 endpoint 失败做分类（429 / 5xx / timeout / context-length / quota），并把 per-endpoint 的 runtime 状态放进 cluster，让 picker 能够跳过处于 cooldown 的 endpoint。其中最值得一提的是 `context_length_exceeded`：它是请求与 endpoint 的“不匹配”，而不是 endpoint 故障——正确的反应是把这个请求路由到 context 更大的模型，而不是把该 endpoint 冷却掉。二元健康（healthy/unhealthy）无法表达这一点，而这和上面 key 级的问题是同一个形状。 **唯一需要先达成共识的设计点。** 我们不能简单地在 snapshot 的 endpoint 上加一个 `cooldownUntil` 字段：cooldown / inflight / latency 是每个请求都在变的，如果每次请求都去重建那个不可变 snapshot，会毁掉 #932 刚拿到的零分配读路径。因此建议把两者拆开： - 不可变的成员/健康 snapshot（沿用 #932 不变），以及 - 一个按 endpoint ID 索引、跨 snapshot 重建而存活的 per-endpoint `EndpointState`，其中快变字段用 atomic 承载；当 ID 不变时，重建时复用同一个 state 指针。 这与 Envoy 把静态 host config 和可变的负载/异常统计分开的做法是一致的。 **时序与边界。** 这是 #939（把 cooldown store 改为注入、移除全局变量）之后自然的下一步，并假设 #958（config/runtime 解耦）确立的 ownership 模型。它刻意不包含 scorer（cost / quota / latency 加权打分）、语义路由，以及多 cluster-kind——这些各自是后续独立的步骤。 不知道大家对这个方向是否有兴趣？如果有，我会把它整理成一个完整的 `[RFC]` issue，附上接口草图、兼容性说明，以及测试 / benchmark 计划。 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-17162858 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [x] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址）#764 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] ~6. OpenAI 请求适配层~，该功能不由 pixiu 网关层面实现 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step2.1: 遗留问题 - [ ] 1. 删除 provider 有关的字段和功能（根据供应商名称自动组装endpoint地址），该功能不由 pixiu 网关层面实现 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 7. LLM服务发现 (Nacos 注册中心) #746 ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [x] 1. AI 可观测性 https://github.com/apache/dubbo-go-pixiu/pull/733 - [x] Token 用量 - [x] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [x] 1. AI 可观测性 - [ ] Token 用量 - [ ] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl deleted a comment on the discussion: [AI] AI gateway milestones ai-gateway observability grafana demonstrate https://github.com/user-attachments/assets/24111f30-da89-4c7d-a730-0be60135cf89"; /> https://github.com/user-attachments/assets/9ad8079a-0788-4043-ac22-85a21bb69247"; /> GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14078958 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway milestones ai-gateway observability grafana demonstrate https://github.com/user-attachments/assets/3378a427-f910-499c-a54b-c272288ecd3c"; /> https://github.com/user-attachments/assets/c063704e-66da-4594-9090-520b3948c11f"; /> GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14080520 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway milestones ai-gateway observability grafana demonstrate https://github.com/user-attachments/assets/24111f30-da89-4c7d-a730-0be60135cf89"; /> https://github.com/user-attachments/assets/9ad8079a-0788-4043-ac22-85a21bb69247"; /> GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14078958 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] Token 用量 - [ ] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] Token 用量 - [ ] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [x] ~1. 面向密钥的负载均衡~ https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] Token 用量 - [ ] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] ~1. 面向密钥的负载均衡~ - [x] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/pull/731 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] Token 用量 - [ ] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user AlexStocks added a comment to the discussion: [AI] AI gateway milestones 这还用说，当然方案一好呀，配置这都不算啥事，核心逻辑清晰才算好 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14021600 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway milestones # LLM cluster endpoint 层级的设计 现在的代码用的是方案2，但是我越来越觉得方案1要好（工作量要少很多） --- ### 方案一：一个 Endpoint 对应一个 API Key ***这种设计将一个 apikey 视为一个 `Endpoint`。*** ```yaml # 方案一：原子化的 Endpoint clusters: - name: "chat" lb_policy: "round_robin" # 假设是轮询 endpoints: - id: 1 provider: "deepseek" api_key: "key-1" - id: 2 provider: "deepseek" api_key: "key-2" - id: 3 provider: "openai" api_key: "key-3" ``` 优势 (Pros) 1. **逻辑简单且统一**：负载均衡、健康探测、熔断、统计等所有核心功能，都围绕 `Endpoint` 这个唯一的对象。实现非常干净，每个 `Endpoint` 的状态（健康/不健康）是独立的。 2. **充分复用现有设施**：正如你所说，如果你的系统已经将 `Endpoint` 作为最小单元，那么几乎不需要改动核心的调度逻辑。这大大降低了开发成本和风险。 3. **故障隔离清晰**：`key-1` 对应的 `endpoint-1` 挂了，负载均衡器会清晰地将其从池中移除，完全不影响 `endpoint-2`。责任非常明确。 劣势 (Cons) 1. **配置繁琐且冗余**：这是最大的问题。如果要给 DeepSeek 添加 10 个 Key，用户必须复制粘贴 10 次 `endpoint` 块，并修改 `id` 和 `key`。这不仅麻烦，还容易出错。 2. **从属关系不清晰**：在配置文件中，`endpoint-1` 和 `endpoint-2` 看起来是两个完全独立的实体，但实际上它们都属于同一个 `provider` (DeepSeek)。这种逻辑上的分组关系丢失了，不利于人类理解。 - ### 方案二：一个 Endpoint 对应多个 API Key ***这种设计将一个供应商视为一个 `Endpoint`，apikey是更小的一个层级。*** ```yaml # 方案二：分组的 Endpoint clusters: - name: "chat" lb_policy: "round_robin" # 这个策略现在只作用于 Endpoint 列表 并不作用于api_keys列表 endpoints: - id: 1 provider: "deepseek" api_keys: # 一个 key 列表 - name: "apikey-1" key: "key-1" - name: "apikey-2" key: "key-2" ``` 优势 (Pros) 1. **配置直观且优雅**：从属关系一目了然。用户可以清晰地看到 "DeepSeek" 这个服务下配置了两个 Key。添加或删除 Key 也非常方便，只需在 `api_keys` 数组中操作即可。 劣势 (Cons) 1. **核心逻辑需要重构/重写**：这是致命的缺点。 * **二级负载均衡**：`cluster` 层的 `lb_policy` 将请求分发到 `endpoint-1`。然后呢？`endpoint-1` 内部必须有自己的逻辑来决定本次请求是用 `key-1` 还是 `key-2`。你需要实现一个 "Endpoint 内的 Key 负载均衡器"。 * **健康探测粒度问题**：如果使用 `key-1` 的请求失败（例如 QPS 超限），你不能将整个 `endpoint-1` 标记为不健康，因为 `key-2` 可能还是好的。你需要将健康状态维护在更细的 `api_key` 粒度上。这意味着你的健康探测和熔断机制需要重写，以支持这种父子关系。 * **代码重复**：本质上，你把原本在 `cluster` 层的调度逻辑，在 `endpoint` 内部又实现了一遍，导致了逻辑复杂化和代码重复。 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a comment on the discussion: [AI] AI gateway milestones # LLM cluster endpoint 层级的设计 现在的代码用的是方案2，但是我越来越觉得方案1要好（工作量要少很多，并且逻辑更加清晰） --- ### 方案一：一个 Endpoint 对应一个 API Key ***这种设计将一个 apikey 视为一个 `Endpoint`。*** ```yaml # 方案一：原子化的 Endpoint clusters: - name: "chat" lb_policy: "round_robin" # 假设是轮询 endpoints: - id: 1 provider: "deepseek" api_key: "key-1" - id: 2 provider: "deepseek" api_key: "key-2" - id: 3 provider: "openai" api_key: "key-3" ``` 优势 (Pros) 1. **逻辑简单且统一**：负载均衡、健康探测、熔断、统计等所有核心功能，都围绕 `Endpoint` 这个唯一的对象。实现非常干净，每个 `Endpoint` 的状态（健康/不健康）是独立的。 2. **充分复用现有设施**：正如你所说，如果你的系统已经将 `Endpoint` 作为最小单元，那么几乎不需要改动核心的调度逻辑。这大大降低了开发成本和风险。 3. **故障隔离清晰**：`key-1` 对应的 `endpoint-1` 挂了，负载均衡器会清晰地将其从池中移除，完全不影响 `endpoint-2`。责任非常明确。 劣势 (Cons) 1. **配置繁琐且冗余**：这是最大的问题。如果要给 DeepSeek 添加 10 个 Key，用户必须复制粘贴 10 次 `endpoint` 块，并修改 `id` 和 `key`。这不仅麻烦，还容易出错。 2. **从属关系不清晰**：在配置文件中，`endpoint-1` 和 `endpoint-2` 看起来是两个完全独立的实体，但实际上它们都属于同一个 `provider` (DeepSeek)。这种逻辑上的分组关系丢失了，不利于人类理解。 - ### 方案二：一个 Endpoint 对应多个 API Key ***这种设计将一个供应商视为一个 `Endpoint`，apikey是更小的一个层级。*** ```yaml # 方案二：分组的 Endpoint clusters: - name: "chat" lb_policy: "round_robin" # 这个策略现在只作用于 Endpoint 列表 并不作用于api_keys列表 endpoints: - id: 1 provider: "deepseek" api_keys: # 一个 key 列表 - name: "apikey-1" key: "key-1" - name: "apikey-2" key: "key-2" ``` 优势 (Pros) 1. **配置直观且优雅**：从属关系一目了然。用户可以清晰地看到 "DeepSeek" 这个服务下配置了两个 Key。添加或删除 Key 也非常方便，只需在 `api_keys` 数组中操作即可。 劣势 (Cons) 1. **核心逻辑需要重构/重写**：这是致命的缺点。 * **二级负载均衡**：`cluster` 层的 `lb_policy` 将请求分发到 `endpoint-1`。然后呢？`endpoint-1` 内部必须有自己的逻辑来决定本次请求是用 `key-1` 还是 `key-2`。你需要实现一个 "Endpoint 内的 Key 负载均衡器"。 * **健康探测粒度问题**：如果使用 `key-1` 的请求失败（例如 QPS 超限），你不能将整个 `endpoint-1` 标记为不健康，因为 `key-2` 可能还是好的。你需要将健康状态维护在更细的 `api_key` 粒度上。这意味着你的健康探测和熔断机制需要重写，以支持这种父子关系。 * **代码重复**：本质上，你把原本在 `cluster` 层的调度逻辑，在 `endpoint` 内部又实现了一遍，导致了逻辑复杂化和代码重复。 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14020512 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [ ] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] Token 用量 - [ ] 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [ ] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [ ] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [ ] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/686 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 7. LLM服务发现 (Nacos 注册中心) - [ ] 1. 面向密钥的负载均衡 - [ ] 2. 密钥的健康性检测 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597 ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 - [ ] 2. 密钥的健康性检测 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/686 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [ ] 2. 密钥的健康性检测 - [x] 3. fallback策略 #685 #692 - [x] 4. retry策略 #685 #692 - [x] 5. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/686 - [ ] 7. LLM服务发现 (Nacos 注册中心) ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway milestones ### API密钥的健康探测 对于密钥数组中的某个密钥，若触发了failover则视为该密钥不健康，对于不健康的密钥在下次请求的时候不会使用它发送请求，待超时机制过后重新探测其健康性 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-14013522 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user AlexStocks added a comment to the discussion: [AI] AI gateway milestones 可以更新下 task list，把 nacos 支持也加进去 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973631 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl added a comment to the discussion: [AI] AI gateway
milestones
Pixiu AI 网关将输入统一为openai格式，通过一个适配器filter转换成对应的供应商的格式
```mermaid
graph TD
subgraph "客户端 (Client)"
UserRequest("👤 用户/客户端")
end
subgraph "AI 网关 (AI Gateway)"
A["入口: 接收标准OpenAI格式请求"]
B{"选择对应的适配器"}
subgraph "适配层 (Adapter Layer)"
C1["适配器: OpenAI (透传)"]
C2["适配器: Anthropic Claude"]
C3["适配器: Google Gemini"]
C4["适配器: ...更多"]
end
A --> B
B -- "model: 'gpt-4o'" --> C1
B -- "model: 'claude-3-opus-20240229'" --> C2
B -- "model: 'gemini-1.5-pro'" --> C3
end
subgraph "LLM 供应商 (LLM Providers)"
D1[☁️ OpenAI API]
D2[☁️ Anthropic API]
D3[☁️ Google AI API]
end
C1 --> D1
C2 --> D2
C3 --> D3
%% --- 转换过程示例 (Conversion Example) ---
subgraph "转换示例: OpenAI -> Anthropic"
direction LR
Input("输入请求 (OpenAI 格式):
POST /v1/chat/completions
{
'model': 'claude-3-opus-20240229',
'messages': [
{ 'role': 'system', 'content': '你是一个乐于助人的助手。' },
{ 'role': 'user', 'content': '你好，克劳德！' } ],
'max_tokens': 1024 }"
)
Output("输出请求 (Anthropic 格式):
POST /v1/messages
{
'model': 'claude-3-opus-20240229',
'system': '你是一个乐于助人的助手。',
'messages': [
{ 'role': 'user', 'content': '你好，克劳德！' } ],
'max_tokens': 1024 }"
)
Input -- "通过 Claude 适配器处理" --> Output
end
UserRequest --> A
%% Styling
style A fill:#D6EAF8,stroke:#3498DB
style B fill:#E8DAEF,stroke:#8E44AD
style C2 fill:#D5F5E3,stroke:#2ECC71
style Input fill:#FEF9E7,stroke:#F1C40F
style Output fill:#FDEDEC,stroke:#E74C3C
style C2 fill:#D5F5E3,stroke:#2ECC71
style A fill:#D6EAF8,stroke:#3498DB
style B fill:#E8DAEF,stroke:#8E44AD
style Output fill:#FDEDEC,stroke:#E74C3C
```
GitHub link:
https://github.com/apache/dubbo-go-pixiu/discussions/696#discussioncomment-13973597
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [x] 2. fallback策略 #685 #692 - [x] 3. retry策略 #685 #692 - [x] 4. LLM 供应商的兼容 #678(需持续适配) - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/686 ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [x] 2. fallback策略 #685 #692 - [x] 3. retry策略 #685 #692 - [x] 4. LLM 供应商的兼容 #678(需持续适配) - [ ] 5. MCP Server的兼容 - [ ] 6. OpenAI 请求适配层 https://github.com/apache/dubbo-go-pixiu/discussions/686 ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [x] 2. fallback策略 #685 #692 - [x] 3. retry策略 #685 #692 - [x] 4. LLM 供应商的兼容 #678(需持续适配) - [ ] 5. MCP Server的兼容 ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659 ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [x] 2. fallback策略 #685 #692 - [x] 3. retry策略 #685 #692 - [x] 1. LLM 供应商的兼容 #678(需持续适配) - [ ] 4. MCP Server的兼容 ### Step3: 完善 - [ ] 1. AI 可观测性 - [ ] 1.1. Token 用量 - [ ] 1.2. 响应时间 - [ ] 2. 响应合规性和请求合规性 - [ ] 3. 日志 - [ ] 4. 更智能的路由 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [D] [AI] AI gateway milestones [dubbo-go-pixiu]
GitHub user Alanxtl edited a discussion: [AI] AI gateway milestones https://jimmysong.io/blog/ai-gateway-in-depth/ ### Step1: 基础 - [x] 1. 基础 AI 协议支持 (SSE/HTTP) #657 - [x] 2. Streamable HTTP ([doc](https://spec.modelcontextprotocol.io/specification/2025-03-26/basic/transports/#streamable-http)/[pr](https://github.com/modelcontextprotocol/specification/pull/206)) #674 - [x] 3. Token 用量获取 #659(半成品) ### Step2: 兼容 - [ ] 1. 面向密钥的负载均衡 - [x] 2. fallback策略 #685 #692 - [x] 3. retry策略 #685 #692 - [x] 1. LLM 供应商的兼容 #678(需持续适配) - [ ] 4. MCP Server的兼容 - [ ] 5. AI 可观测性 GitHub link: https://github.com/apache/dubbo-go-pixiu/discussions/696 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
