jujiale opened a new issue, #11390:
URL: https://github.com/apache/apisix/issues/11390
### Description
Hello，I suffered the following situation in our prd apisix cluster and one
dev apisix node
our prd has 4 env, each has 3 apisix instance, deployed with rpm, one
cluster( we all it **A** in here) appear a odd thing, let me describe:
1. we modify the cluster A config in apisix-dashboard, and we submit it, in
etcd, I have found it is modify correctly, but when I use /v1/route/route_id,
found that the whole config in cluster A instance is old version, and no matter
how many times modify the config, the config in etcd is correactly, and the
update_time is correct, but the config in instance is old, and the update time
is very old, and nevery change.
for example : etcd config
`
/test/apisix/routes/515483732765836994
{"id":"515483732765836994","create_time":1716781847,"update_time":1720085553,"uris":
["/menu.service.query/m","/menu.service.query/c/*"],"name":"aaa","priority":10,"methods":["GET","POST","PUT","DELETE","PATCH","HEAD","OPTIONS","CONNECT","TRACE"],"host":"xxx.com","upstream_id":"515483516306196172","status":1}
`
when I invoke /v1/route/route_id, config like below:
`
{
"key": "/test/apisix/routes/515483732765836994",
"createdIndex": 946,
"has_domain": false,
"clean_handlers": {},
"modifiedIndex": 946,
"update_count": 0,
"orig_modifiedIndex": 946,
"value": {
"priority": 10,
"host": "xxx.com",
"name": "xxxxx",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"id": "515483732765836994",
"uris": [
"/menu.service.query/m",
"/menu.service.query/w",
"/menu.service.query/pm/*"
],
"update_time": 1716781847,
"create_time": 1716781847,
"status": 1,
"upstream_id": "515483516306196172"
}
}
`
we could see that the uris is not the same, and the update_time is not the
same, but in other cluster, it works well
2.apisix log shows:
`
172.24.61.52, server: _, request: "POST /menu.service.query/w
HTTP/1.1", host: "xxx.com"
2024/07/04 16:00:55 [error] 16235#16235: *143253446 [lua]
config_util.lua:86: failed to find clean_handler with idx 1, client:
172.24.61.47, server: _, request: "POST /menu.service.query/wpil HTTP/1.1",
host: "xxx.com"
2024/07/04 16:00:55 [error] 16234#16234: *143283913 [lua]
config_etcd.lua:584: failed to fetch data from etcd:
/test/apisix/apisix/core/config_util.lua:104: attempt to index local 'item' (a
boolean value)
stack traceback:
/test/apisix/apisix/core/config_util.lua:104: in function
'fire_all_clean_handlers'
/test/apisix/apisix/core/config_etcd.lua:315: in function
'sync_data'
/test/apisix/apisix/core/config_etcd.lua:541: in function
</test/apisix/apisix/core/config_etcd.lua:532>
[C]: in function 'xpcall'
/test/apisix/apisix/core/config_etcd.lua:532: in function
</test/apisix/apisix/core/config_etcd.lua:513>, etcd key:
/test/apisix/upstreams, context: ngx.timer
2024/07/04 16:00:55 [error] 16235#16235: *143280176 [lua]
config_util.lua:86: failed to find clean_handler with idx 1, client:
172.24.61.47, server: _, request: "POST /menu.service.validate/w HTTP/1.1",
host: "xxx.com"
2024/07/04 16:00:55 [error] 16240#16240: *143284010 [lua]
config_etcd.lua:584: failed to fetch data from etcd:
/test/apisix/apisix/core/config_util.lua:104: attempt to index local 'item' (a
boolean value)
stack traceback:
/test/apisix/apisix/core/config_util.lua:104: in function
'fire_all_clean_handlers'
/test/apisix/apisix/core/config_etcd.lua:315: in function
'sync_data'
/test/apisix/apisix/core/config_etcd.lua:541: in function
</test/apisix/apisix/core/config_etcd.lua:532>
[C]: in function 'xpcall'
/test/apisix/apisix/core/config_etcd.lua:532: in function
</test/apisix/apisix/core/config_etcd.lua:513>, etcd key:
/test/apisix/janus/routes, context: ngx.timer
`
3.capture the 2379 port in apisix instance, found:
`
66
{"error":{"grpc_code":1,"http_code":408,"message":"context
canceled","http_status":"Request Timeout"}}
0
`
also found many request is timeout beyond 30s, as below:
4. I could confirm that the etcd is health, even I restart etcd, the
scenario also exist. and apisix to etcd network is correct, some /v3/watch
could return correctly, but apisix seems not use the config.
because we use 2.15.0 in prd env, so we could not upgrade it randomly
want to know if it is apisix bug, if it is , we plan merge some changes to
solve it, and why config could not sync to apisix instance
### Environment
- APISIX version (run `apisix version`): 2.15.0
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`):3.5.0
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
