I just skimmed the blobstore code, and some questions about the implementation. I'd be really appreciated if someone understanding BlobStore answers below questions. (Questions are based on current 1.x branch, but I think master branch is similar.)
1. In get-version-for-key, it always create KeySequenceNumber instance so synchronized keyword on KeySequenceNumber.getKeySequenceNumber() is no effect. But looks like the method shouldn't be run concurrently on same key, and furthermore every Nimbuses can call the method concurrently. Then we might need to have a kind of global lock on obtaining version of the key. Is my understanding right? 2. We rely on blob-sync function which is skipped on leader Nimbus, then how leader Nimbus synchronizes its blobs? 3. What's the purpose of setup-blobstore function? It seems slight different from blob-sync function (not updating the blobs) and it runs with both leader and follow Nimbus. 4. In KeySequenceNumber.getKeySequenceNumber(), in some situations version number goes 0 (initial version - 1). Is it meant to mark blob as invalid? Then can we use 0 for version number if any exceptions are caught on that method? Since shallowing the whole exceptions in that method seems intended but we met crash on that case, and even we throw the exception and pass to caller, rolling back is not easy. Thanks in advance, Jungtaek Lim (HeartSaVioR) 2017년 1월 24일 (화) 오후 12:25, Jungtaek Lim <[email protected]>님이 작성: > More details: there were three Nimbuses N1, N2, N3 and N1 was the leader. > We submitted topology T1 and after submission we restarted N1. N2 got > leader and we killed T1. While N1 is initializing and syncing up its > topology blobs, N2 concurrently removes the ZK path and also max sequence > number path for topology blob in progress of killing topology. This race > condition is only occurring on Local BlobStore since removing ZK path is > done only if Nimbus is using Local BlobStore. > > So it's the former case, and stopping current sync phase and restarting > sync is an ideal way since we're just guaranteeing eventually consistent. > I'll take a look at the codebase to see how we can apply, but it should be > great help for me if someone is familiar with BlobStore codebase and > willing to handle it. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2017년 1월 23일 (월) 오후 11:33, Bobby Evans <[email protected]>님이 작성: > > HA for the blobstore was set up so that ZK would hold the source of truth > and then the other nimbus nodes would be eventually consistent with each > other. I'm not totally sure of the issue, because I don't understand if > this is happening in the context of a follower trying to keep up to date, > or with a leader entering the data. If it is the latter we need some > better fencing to prevent multiple leaders trying to write to the DB at the > same time. If it is the former we need some better code so the follower > can read/update the replica it has without the possibility of trying to > auto-vivify a node that is being deleted. Generally in these cases we > would declare the race safe and then just start the sync process over again. > > > - Bobby > > > On Monday, January 23, 2017, 2:12:18 AM CST, Jungtaek Lim < > [email protected]> wrote: > Hi devs, > > I've been struggling to resolve specific scenario, and found Local > BlobStore cares about Nimbus failure scenarios, but not about removing > keys. > > For example, I heard that Nimbus crashed in specific scenario, and error > stack trace pointed to below code: > > https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/blobstore/KeySequenceNumber.java#L138-L149 > > checkExists (L138 > < > https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/blobstore/KeySequenceNumber.java#L138 > >) > succeeds but getChildren (L149 > < > https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/blobstore/KeySequenceNumber.java#L149 > >) > throws NoNodeException, in result sequenceNumbers.last() throws > NoSuchElementException. > > We could have a look deeply and make some workarounds, but given that ZK is > accessible from every Nimbuses, we can't ensure every paths are safe. > > I guess that BlobStore needs global lock or single controller to handle all > the things right. I'm also open to any workarounds or other ideas. > > What do you think? > > Thanks, > Jungtaek Lim (HeartSaVioR) > >
