rzo1 commented on issue #7653: URL: https://github.com/apache/storm/issues/7653#issuecomment-5617647468
Thanks a lot, the logs show what happens. Timeline for `dep-apeframework-0.0.11710-SNAPSHOT-f2d87a33-a8ad-4a95-a567-71e96a2b8695.jar`: | Time | Instance | What happens | |---|---|---| | 05:19:38.715 | nimbus-1 (leader) | `Removing dependency blobs of …Clmcla…`: deletes the blob and its ZooKeeper nodes | | 05:19:38.823 | nimbus-0 | `Created download session`: nimbus-2 downloads the blob from nimbus-0 | | 05:19:40.640 | nimbus-2 | `set-path …stormnimbus-2…-1`: registers the key again, from scratch at version 1 | | 05:19:40.870, 05:19:42.269 | nimbus-0 | downloads it back from nimbus-2, `set-path …-0`, then `…-1` | | 05:20:09.977 | supervisor-2 | localizer cleanup asks the leader whether the blob still exists | | 05:20:10.235, 05:20:11.466 | nimbus-1 (leader) | downloads it back from nimbus-2 (download session on nimbus-2 at 05:20:10.233), `set-path …-0`, then `…-1` | The leader deletes `/blobstore/<key>` child by child, so the delete is not atomic. nimbus-2's blob sync ran in between: its own entry was already gone while nimbus-0's was still there, so it downloaded the blob from nimbus-0 as if it had been updated. By the time it called `createStateInZookeeper`, the key node was gone, and it registered the key again as new. The other instances then copied it back from nimbus-2, including the leader. This also explains why supervisor-2 deletes the topology's jar/code/conf at 05:20:10 but keeps the dependency blob. The `Could not remove the dependency blobs of …Oca…` warning at 05:21:08 is harmless: the cleanup pass at 05:20:58 had already removed that topology's dependency blob, and the second pass only found the code blob gone. https://github.com/apache/storm/pull/9082 does two things: - It stops a non-leader Nimbus from registering a key that was deleted from ZooKeeper. - It adds a sweep that removes dependency blobs no topology has referred to for `nimbus.inbox.jar.expiration.secs`, which also cleans up blobs that already leaked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
