poorbarcode opened a new pull request, #24580: URL: https://github.com/apache/pulsar/pull/24580
### Motivation **Background: how ZK metadata store handles the node exists error when creating a new node** ZK metadata store throws a `BadVersionException` error if it receives a `NODEEXISTS` error when creating a new node. https://github.com/apache/pulsar/blob/v4.0.5/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKMetadataStore.java#L269-L272 ```java if (code == Code.NODEEXISTS) { // We're emulating a request to create a node, so the version is invalid op.getFuture().completeExceptionally(getException(Code.BADVERSION, op.getPath())); } ``` **Background: how ZK metadata store handles disconnect error when executing operations** ZK metadata store will retry the operation if it receives a `CONNECTIONLOSS` error when executing operations. https://github.com/apache/pulsar/blob/v4.0.5/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKMetadataStore.java#L200-L227 ```java if (code == Code.CONNECTIONLOSS) { // Retry with the individual operations executor.schedule(() -> { ops.forEach(o -> batchOperation(Collections.singletonList(o))); }, 100, TimeUnit.MILLISECONDS); } ``` **Background: how ZK clients/servers limit the max length of packet** - ZK clients/servers limit the max length of a packet to 1MB by default. - Users can use a Java environment variable `jute.maxbuffer` to change the maximum packet length, which controls the maximum packet length for both requests and receives. - Once the maximum packet length is reached, the ZK client will throw an error and reconnect. **Issue: ZK client throws an error and reconnects when the max length of the packet is exceeded by the server** - As the background "How ZK clients/servers limit the max length of packet" said, not only is the request packet is in limiting, but the response packet is not limited. - The requests that type `Get` or `GET_CHILDREN` will receive a response from the server, and the response packet may be larger than the max length of packet length. - When the response packet is larger than the maximum packet length, the ZK client will throw an error and reconnect. - The reconnecting will cause the `BadVersionException` error mentioned in the background "How ZK metadata store handles the node exists error when creating a new node". - Then the operation of creating a node can never be completed. You can reproduce the issue by running the new test `testReceivedHugeResponse` ### Modifications Add a mechanism to limit `get/getChildren` operations of the ZK metadata store. This mechanism merely avoids problems with as few resources as possible, but it cannot prevent all problems. To solve all the problems, there will be a PIP later to make the Settings configurable, see also the section "Next things to do" ### Next things to do I will submit a PIP - Make `MetadataNodePayloadLenEstimator` configurable - Add metrics of the node path that has the max packet length of each type - Fix the issue of out-of-order command execution in the metadata store. I will describe this problem in detail in PIP - Solve the following issue **The Metadata store can not confirm whether the bad version error is expected or not after a reconnection.** - ZK client sent a `Put` request. - Server has received the request and is handling it. - Server is responding. - The channel reconnected before the `Put` request was responded. - The `Put` request is sent again, but the node already exists(or already modified), so Pulsar get a `BadVersionException` error. - **(Highlight)** The previous change is caused by the first try, but the second try is stuck due to the ZK node has been changed by the first try. - Once this situation happens, we can only restart the broker or reload topics to fix it, since we do not know whether the bad version error is expected or not. ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
