Hi devs, I've been struggling to resolve specific scenario, and found Local BlobStore cares about Nimbus failure scenarios, but not about removing keys.
For example, I heard that Nimbus crashed in specific scenario, and error stack trace pointed to below code: https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/blobstore/KeySequenceNumber.java#L138-L149 checkExists (L138 <https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/blobstore/KeySequenceNumber.java#L138>) succeeds but getChildren (L149 <https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/blobstore/KeySequenceNumber.java#L149>) throws NoNodeException, in result sequenceNumbers.last() throws NoSuchElementException. We could have a look deeply and make some workarounds, but given that ZK is accessible from every Nimbuses, we can't ensure every paths are safe. I guess that BlobStore needs global lock or single controller to handle all the things right. I'm also open to any workarounds or other ideas. What do you think? Thanks, Jungtaek Lim (HeartSaVioR)
