Re: [I] help request: service health check interval [apisix]
SkyeYoung commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-3173093071 @hgaong If you still have any questions, please open it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
SkyeYoung closed issue #12067: help request: service health check interval URL: https://github.com/apache/apisix/issues/12067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
SkyeYoung commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-3162230653 Hi, I tried to reproduce locally based on the existing conditions you gave, but the result was that it did not reproduce. (APISIX: 3.13.0, 3.11.0) Please try to provide the minimal reproduction steps and environment so that I can assist you in locating the problem. The following are some of the results I replicated locally: Details ```text Monitoring health check requests for 10 seconds... This will help analyze the health check frequency issue === Verifying Services === Checking Docker containers: ✅ apisix is running ✅ upstream1 is running ✅ upstream2 is running Testing health check endpoints: ✅ upstream1 health check endpoint responding ✅ upstream2 health check endpoint responding Reloading nginx configurations... Triggering health checks by sending initial requests... Sending request 1 to trigger health check initialization... Sending request 2 to trigger health check initialization... Sending request 3 to trigger health check initialization... Sending request 4 to trigger health check initialization... Sending request 5 to trigger health check initialization... Waiting for health checks to initialize... Starting health check monitoring... Monitoring upstream1 health checks... Monitoring upstream2 health checks... === Route and Service Information === [2025-08-07 02:16:27] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:27 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:27] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:27 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" Total routes configured: 600 Monitoring in progress... (10s) [2025-08-07 02:16:28] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:28 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:28] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:28 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:29] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:29 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:29] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:29 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:30] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:30 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:30] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:30 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:31] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:31 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:31] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:31 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:32] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:32 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:32] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:32 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:33] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:33 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:33] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:33 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:34] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:34 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:34] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:34 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:35] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:35 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:35] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:35 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:36] upstream1: 172.23.0.5 - - [07/Aug/2025:02:16:36 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" [2025-08-07 02:16:36] upstream2: 172.23.0.5 - - [07/Aug/2025:02:16:36 +] "GET /apisix/heartbeat HTTP/1.1" 200 81 "-" "-" "/apisix/heartbeat" === Health Check Analysis === upstream1: Total health check requests: 11 Requests per second: 1 Expected (1s interval): 1 request/second Actual vs Expected ratio: 1:1 ✅ Health check frequency appears normal upstream2: Total health check requests: 11 Requests per second: 1 Expected (1s interval): 1 request/second Actual vs Expected ratio: 1:1
Re: [I] help request: service health check interval [apisix]
SkyeYoung commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-3157081285 @hgaong em, sorry, I don't quite understand. What exactly are you referring to? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-3154570668 > Hi [@hgaong](https://github.com/hgaong), any updates? Sorry for having too many heartbeat requests and not daring to activate it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
Baoyuantop commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-3152953351 Hi @hgaong, any updates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
Baoyuantop commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-3050892757 Hi @hgaong, can you use the health check control API to get the status for debugging? https://apisix.apache.org/docs/apisix/control-api/#get-v1healthcheck -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] help request: service health check interval [apisix]
hgaong opened a new issue, #12067:
URL: https://github.com/apache/apisix/issues/12067
### Description
service
```
{
"id": "557101231575663294",
"create_time": 1741587810,
"update_time": 1742293721,
"name": "test-api",
"upstream": {
"nodes": [
{
"host": "172.31.19.1",
"port": 80,
"weight": 100
},
{
"host": "172.31.20.9",
"port": 80,
"weight": 100
}
],
"retries": 0,
"timeout": {
"connect": 30,
"send": 30,
"read": 30
},
"type": "roundrobin",
"checks": {
"active": {
"concurrency": 2,
"healthy": {
"http_statuses": [
200,
302,
301,
404
],
"interval": 1,
"successes": 2
},
"http_path": "/apisix/hearbeat",
"port": 80,
"timeout": 1,
"type": "http",
"unhealthy": {
"http_failures": 2,
"http_statuses": [
429,
500,
501,
502,
503,
504,
505
],
"interval": 1,
"tcp_failures": 2,
"timeouts": 3
}
}
},
"scheme": "http",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 640
}
},
"plugins": {
"ip-restriction": {
"_meta": {
"disable": true
},
"blacklist": [
"108.136.33.21"
]
},
"limit-req": {
"_meta": {
"disable": true
},
"burst": 0,
"key": "http_x_forwarded_for",
"key_type": "var",
"rate": 10,
"rejected_code": 429,
"rejected_msg": "ff"
},
"ua-restriction": {
"_meta": {
"disable": true
},
"bypass_missing": false,
"denylist": [
"curl"
]
}
}
}
```
Then I configured around 600 routes, each of which referenced this service.
The observed situation is that each node of the upstream service has about 20
heartbeat requests per second, which is inconsistent with the configured
interval of 1 second. why ?
### Environment
- APISIX version (run `apisix version`):3.11
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-3047977126 there is still a problem.but i dont have permissions to reopen this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
Baoyuantop closed issue #12067: help request: service health check interval URL: https://github.com/apache/apisix/issues/12067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
Baoyuantop commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2896194818 If there is still a problem, please open it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2785536285 When I restarted Apisix, the abnormal heartbeat disappeared. Guess it's caused by residual heartbeat tasks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
Baoyuantop commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2756820906 How did you deploy APISIX? Are there multiple instances running? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2772230209 > > How did you deploy APISIX? Are there multiple instances running? > > I deployed two Apisix nodes, but the Spring Boot service log shows that heartbeat requests only come from the Apisix node with traffic. And when I turned off Apisix's health check, all heartbeat requests from the Spring Boot service disappeared. When I reopened it, heartbeat requests appeared again. Another interesting phenomenon is that when I modify the heartbeat path under the apisix service to`/apisix/heartbeat/ss`, high-frequency `/apisix/heartbeat` requests still exist. When I turn off the apisix health check, both high-frequency `/apisix/heartbeat` requests and low-frequency high-frequency `/apisix/heartbeat` will disappear simultaneously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2772223010 > How did you deploy APISIX? Are there multiple instances running? I deployed two Apisix nodes, but the Spring Boot service log shows that heartbeat requests only come from the Apisix node with traffic. And when I turned off Apisix's health check, all heartbeat requests from the Spring Boot service disappeared. When I reopened it, heartbeat requests appeared again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2739037828 > > The observed situation is that each node of the upstream service has about 20 heartbeat requests per second > > How did you tally this information? Maybe try adding `upstream.checks.active.req_headers` configuration to make sure all requests are issued by APISIX's health checks Through SkyWalking and the logs of my service, I have counted 20 heartbeats. It can be confirmed that only APISix will call this path "http_path": "/apisix/heartbeat". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2747827661 > Can you provide the configuration file for APISIX? It would help to reproduce the problem. config.yaml ``` apisix: node_listen: 9080 # APISIX listening port enable_ipv6: false deployment: admin: allow_admin: # https://nginx.org/en/docs/http/ngx_http_access_module.html#allow - 0.0.0.0/0 # We need to restrict ip access rules for security. 0.0.0.0/0 is for test. admin_key: - name: "admin" key: secret # apikey role: admin # admin: manage all configuration data etcd: host: # it's possible to define multiple etcd hosts addresses of the same etcd cluster. - "http://etcd:2379"; # multiple etcd address prefix: "/apisix" # apisix configurations prefix timeout: 30 # 30 seconds plugin_attr: skywalking: service_name: APISIX service_instance_name: "APISIX Instance Name" endpoint_addr: http://sw:12800 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
hgaong commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2747826797 config.yaml ``` apisix: node_listen: 9080 # APISIX listening port enable_ipv6: false deployment: admin: allow_admin: # https://nginx.org/en/docs/http/ngx_http_access_module.html#allow - 0.0.0.0/0 # We need to restrict ip access rules for security. 0.0.0.0/0 is for test. admin_key: - name: "admin" key: secret # apikey role: admin # admin: manage all configuration data etcd: host: # it's possible to define multiple etcd hosts addresses of the same etcd cluster. - "http://etcd:2379"; # multiple etcd address prefix: "/apisix" # apisix configurations prefix timeout: 30 # 30 seconds plugin_attr: skywalking: service_name: APISIX service_instance_name: "APISIX Instance Name" endpoint_addr: http://sw:12800 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
Baoyuantop commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2742975143 Can you provide the configuration file for APISIX? It would help to reproduce the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: service health check interval [apisix]
Baoyuantop commented on issue #12067: URL: https://github.com/apache/apisix/issues/12067#issuecomment-2736671227 > The observed situation is that each node of the upstream service has about 20 heartbeat requests per second How did you tally this information? Maybe try adding `upstream.checks.active.req_headers` configuration to make sure all requests are issued by APISIX's health checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
