namannagar89 opened a new issue, #13736: URL: https://github.com/apache/apisix/issues/13736
### Current Behavior When `ai-proxy` makes the outbound **HTTPS** request to a live LLM endpoint, the nginx **worker process crashes with `SIGILL` (signal 4, "illegal instruction")**. The client receives an empty reply, and the gateway log shows: ``` [alert] 1#1: worker process NN exited on signal 4 ``` One crash per request. The worker is respawned by the master, so there's no permanent outage, but every ai-proxy request that reaches the TLS handshake to the upstream LLM kills its worker. The crash is **not** provider-specific and **not** in the Vertex JWT-signing/token path (we initially suspected that and disproved it — see below). It reproduces whenever ai-proxy opens a real HTTPS/TLS connection to the LLM endpoint. ### Expected Behavior ai-proxy proxies the request to the LLM over HTTPS and returns the response (or a normal error), without crashing the worker. ### What we observed / how we isolated it | Test | Result | |------|--------| | `provider: vertex-ai` → Vertex (HTTPS) | worker `exited on signal 4`, empty reply | | `provider: openai`, `override.endpoint: http://127.0.0.1:1/...` (unreachable, **no TLS**) | **no crash**, clean HTTP 500 | | `provider: openai` → `https://api.openai.com/...` (real **HTTPS**, dummy key) | worker `exited on signal 4`, empty reply (reproduced twice) | | Same request after moving the gateway to a different machine family (Intel E2 → AMD T2D) | still crashes | So the trigger is the **outbound HTTPS/TLS call to a live endpoint** (reproduced with both `openai` and `vertex-ai`); there's no crash when the endpoint is unreachable (no TLS handshake). It reproduces on both Intel and AMD CPUs. Note: the gateway's normal reverse-proxy TLS to other HTTPS upstreams works fine on the same nodes; only ai-proxy's outbound HTTPS path crashes. ### Steps to Reproduce 1. Run APISIX `3.17.0` (image `apache/apisix:3.17.0-ubuntu`). 2. Create a route using `ai-proxy` with any provider whose endpoint is real HTTPS, e.g.: ```jsonc // PUT /apisix/admin/routes/1 { "uri": "/ai", "plugins": { "ai-proxy": { "provider": "openai", "auth": { "header": { "Authorization": "Bearer sk-dummy" } }, "options": { "model": "gpt-4" } } }, "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:1": 1 } } } ``` 3. Send a request: ```bash curl -sv http://<gateway>/ai -H 'Content-Type: application/json' \ -d '{"messages":[{"role":"user","content":"hi"}]}' # => curl: (52) Empty reply from server ``` 4. Gateway log shows `[alert] ... worker process NN exited on signal 4`. (A dummy key is fine — the crash happens during/around the TLS connection, before a valid API response is needed.) ### Environment - APISIX version: **3.17.0** (`apache/apisix:3.17.0-ubuntu`) - OS/Platform: GKE (Ubuntu-based image); reproduced on `e2-standard-4` (Intel Xeon, no AVX-512) and `t2d-standard-8` (AMD EPYC Milan, no AVX-512) - ai-proxy plugins load correctly (note: on clusters where the `prometheus` plugin is disabled, ai-proxy fails to load because it requires the `prometheus-cache` shared dict — a separate issue; we declared that dict so ai-proxy loads, then hit this crash). ### What we could not determine We do not yet have a coredump/backtrace, so the exact faulting library/instruction is unknown. `SIGILL` (illegal instruction) on a stock amd64 build, in the outbound TLS/crypto path, on CPUs lacking AVX-512, suggests a native code path emitting an instruction the CPU doesn't support — but that's unconfirmed. Happy to gather a backtrace with guidance (e.g. recommended way to enable coredumps for the worker in the official image). ### Question Has anyone else hit a `SIGILL` from ai-proxy's outbound HTTPS path, particularly on non-AVX-512 CPUs? Any guidance on capturing a usable backtrace from the official image would help pinpoint the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
