unsw-luopan opened a new issue, #12879: URL: https://github.com/apache/apisix/issues/12879
### Current Behavior My config about ai-proxy-multi health check: curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \ -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" \ -d '{ "id": "ai-proxy-multi-route", "uri": "/llm/qwen-test", "methods": [ "POST" ], "plugins": { "ai-proxy-multi": { "instances": [ { "name": "qwen32b-test", "provider": "openai-compatible", "weight": 8, "auth": { "header": { "Authorization": "Bearer aaa" } }, "options": { "model": "qwen3-32b-int4-local" }, "override": { "endpoint": "http://10.150.9.179:9998/v1/chat/completions"; }, "checks": { "active": { "type": "http", "host": "10.150.9.179", "port": 9998, "https_verify_certificate": false, "http_path": "/v1/models/qwen3-32b-int4-local", "healthy": { "interval": 1, "successes": 1 }, "unhealthy": { "interval": 1, "http_failures": 1 } } } }, { "name": "qwen8b-test", "provider": "openai-compatible", "weight": 2, "auth": { "header": { "Authorization": "Bearer aaa" } }, "options": { "model": "qwen3-8b" }, "override": { "endpoint": "http://172.16.3.30:9997/v1/chat/completions"; }, "checks": { "active": { "type": "http", "host": "172.16.3.30", "port": 9997, "https_verify_certificate": false, "http_path": "/v1/models/qwen3-8b", "healthy": { "interval": 1, "successes": 1 }, "unhealthy": { "interval": 1, "http_failures": 1 } } } } ] } } }' After create the route, I can see health check logs in my llm log, so this route create successfully. Then I kill one of my model, and create request for /llm/qwen-test, which is the load balancer in my config. However, apisix still route my request to my dead llm. By the way, my health check endpoint return 400 instead of 500 if model is dead. I checked the document about health check topic in apisix, which use 500 code as defult. But in ai-proxy-multi health check, there is no way to change the unhealthy http code. Then I wraped a service about health check, and return code 500 if llm is dead, but apisix still route my request to dead llm. ### Expected Behavior Apisix does not route request to dead llm. ### Error Logs No error log in apisix. ### Steps to Reproduce apisix: 3.14.1-debian ### Environment - APISIX version (run `apisix version`): - Operating system (run `uname -a`): - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): - APISIX Dashboard version, if relevant: - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
