Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop closed issue #12387: help request: Websocket service receives RST packet from Apisix Pods during Node Termination URL: https://github.com/apache/apisix/issues/12387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3520249318 If there is still a problem, please open it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3116914986 Hi @horkasai, any updates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3082117488 Can you provide an example of your route configuration and the apisix configuration file? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
horkasai commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3077603382 yes, it has to be healthy, because none of the upstream pods were disrupted during the process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3055913402 Do you use the upstream health check of apisix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3022313571 You only need to filter out the error-type logs, similar to `[error]`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
horkasai commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3022385188 I'm seeing quite a few connection refused errors from APISIX around the time my node was terminated. content:2025/06/30 05:26:44 [error] 51#51: *70538 connect() failed (111: Connection refused) while connecting to upstream, client: 10.172.38.82 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3021961470 Can you provide a more detailed description? I don't understand the question. Is there a stable step to reproduce your problem? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
horkasai commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3022245058 I can't seem to find any unusual logs in APISIX, perhaps maybe because I have over 70 million of logs within the span of 10 minutes. Is there any specific keywords I can look out for? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
Baoyuantop commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3022133958 Are there any error logs that can be observed in APISIX? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
horkasai commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3022090492 **Description** Basically I have a WebSocket service that is upstreamed by APISIX, everything is working as expected until a node in my EKS cluster gets terminated/replaced by Karpenter. This node can be totally unrelated to my services and apisix, so long it is within the same cluster, the issue will occur. **The Issue** During a node termination/replacement, a handful of incoming requests will be fail to receive by my WebSocket service. Upon checking the logs, the connection was still connected between the client and the server. Moreover, using tcpdump, I can see there are a number of RST packets sent to my WebSocket endpoint from the APISIX pods, which could explains why my service was not receiving any requests from the client. I've tried removing APISIX and directly accessing my WebSocket service through AWS NLB endpoint, the issue did not occur at all. **Steps to Reproduce** 1. Have a WebSocket Service Upstreamed By Apisix 2. Send 4-5k requests per second to the service 3. Terminate a node in the same cluster (can be in a different node group to the WebSocket service) 4. Observe the service behaviour during the node termination process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: Websocket service receives RST packet from Apisix Pods during Node Termination [apisix]
horkasai commented on issue #12387: URL: https://github.com/apache/apisix/issues/12387#issuecomment-3017608790 APISIX versionL 3.12.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
