[GitHub] [apisix] spacewander commented on issue #3937: [discuss]: when a node in the etcd cluster fails, no error log is output
spacewander commented on issue #3937: URL: https://github.com/apache/apisix/issues/3937#issuecomment-809834044 The error log is right. If it reports the connection is refused, that means it fails to get any data from etcd at that moment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3937: [discuss]: when a node in the etcd cluster fails, no error log is output
spacewander commented on issue #3937: URL: https://github.com/apache/apisix/issues/3937#issuecomment-809849343 If it reports an error log in 18:26, it doesn't get data from another etcd node. @Firstsawyou Does the error log happen forever or just for a moment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3937: [discuss]: when a node in the etcd cluster fails, no error log is output
spacewander commented on issue #3937: URL: https://github.com/apache/apisix/issues/3937#issuecomment-809916240 > > > > If it reports an error log in 18:26, it doesn't get data from another etcd node at that time. > > @Firstsawyou > > Does the error log happen forever or just for a moment? > > The error log happen forever when a node in the etcd cluster(3 nodes) fails, at that time ,apisix can get data from other etcd node and work corretcly. Interesting. @Firstsawyou Could you investigate this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3937: [discuss]: when a node in the etcd cluster fails, no error log is output
spacewander commented on issue #3937: URL: https://github.com/apache/apisix/issues/3937#issuecomment-810161651 > > > @spacewander @Firstsawyou > When I use ./apisix start to start the apisix, I found it will init etcd. I read the "etcd init" code and found it will check the status of all etcd nodes. So when when a node in the etcd cluster(3 nodes) fails, the etcd_init will fail too , and the apisix can't start. > I think when a node in the etcd cluster(3 nodes) fails, the etcd_init should success because it do no affect on the normal operation of APISIX, and the apisix should start normally. It does. Why do you want to configure a bad node inside APISIX? Start APISIX in an unhealthy situation is not a good idea. Consider one of your fail nodes has a wrong auth configuration which can't be detected if we just skip it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3937: [discuss]: when a node in the etcd cluster fails, no error log is output
spacewander commented on issue #3937: URL: https://github.com/apache/apisix/issues/3937#issuecomment-810704950 The `etcd init` doesn't just do the node check. As its name indicates, it **does the init job**. We need this operation to ensure the data in etcd is initialized correctly to avoid unexpected responses. If we skip this for some nodes, there is no way to ensure they are correctly initialized. As for "normally start and use apisix in a production environment in case of the extreme cases such as a node in the etcd cluster(3 nodes) fails", **this problem is mostly an etcd HA problem**. You can use 3 virtual hosts for etcd and ensure they are mapping into healthy nodes. If it is no enough, you can introduce retry when starting APISIX. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3937: [discuss]: when a node in the etcd cluster fails, no error log is output
spacewander commented on issue #3937: URL: https://github.com/apache/apisix/issues/3937#issuecomment-810708396 > > > > If we skip this for some nodes, there is no way to ensure they are correctly initialized. > > Why? ETCD is self-replicated. People might configure wrong node. Don't be surprise, it happened before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
