exceptionfactory commented on PR #8013: URL: https://github.com/apache/nifi/pull/8013#issuecomment-1890155711
> @exceptionfactory Pushed the state clean-up. > > While I was testing it, I found a more generic issue unfortunately: If you run ConsumeAzureEventHub in a cluster, the processor instances use the shared cluster state provider (e.g. Zookeeper) and maintain the state together, handle the partition balancing, etc. It works properly until all nodes use the shared cluster state. But if you disconnect a node from the cluster (on the UI or I guess the same happens when there is a network issue between the nodes), the disconnected node becomes a standalone node and it does not use the shared state anymore but starts to maintain its own state in the local state provider from scratch. As the disconnected node does not know about the other consumer instances, it believes it is the only consumer in the group and can own all the partitions. Both the remaining part of the cluster and the disconnected node try to own the same partitions. The checkpoints will be maintained in two places (cluster vs local state) and it will lead to message dupl ication. It would be better if the processor on the disconnected node was stopped and did not create its local state at all. Do you think it is possible? I'm afraid it is how the framework works though. Thanks for making the changes and highlighting the concern @turcsanyip. Changing a node from a cluster member to a standalone instance effectively changes the state source as you noted. However, if a node is part of a cluster and temporarily has cluster communication problems, this is different than manually disconnecting and removing the node from the cluster. With that background, I don't think that scenario should be a concern because switching cluster membership should be a manual process. Are you observing different behavior? With that background, it seems like this should be close to completion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
