It looks like under some circumstances (which I haven't worked out yet) that the InterprocessMutex acquire() is not working correctly when reconnecting to ZK. Still digging into why this is.
There also seems to be a bug in the SemaphoreClient, unless I'm missing something. At lines 126 and 140 it does compareAndSet() calls but throws an exception if they return true. As far as I can work out, this means that whenever the lock is acquired, an exception gets thrown indicating that there are Multiple acquirers. This test is failing fairly consistently. It seems to be the remaining test that keeps failing in the Jenkins build also cheers On Wed, Jun 1, 2016 at 3:10 PM, Cameron McKenzie <[email protected]> wrote: > Looks like I was incorrect. The NoWatcherException is being thrown on > success as well, and the problem is not in the cluster restart. Will keep > digging. > > On Wed, Jun 1, 2016 at 2:52 PM, Cameron McKenzie <[email protected]> > wrote: > >> TestInterProcessSemaphoreCluster.testCluster() is failling (assertion at >> line 294). Again, it seems like some sort of race condition with the >> watcher removal. >> >> When I run it in Eclipse, it fails maybe 25% of the time. When it fails >> it seems that it's got something to do with watcher removal. When the test >> passes, this error is not logged. >> >> org.apache.zookeeper.KeeperException$NoWatcherException: KeeperErrorCode >> = No such watcher for /foo/bar/lock/leases >> at >> org.apache.zookeeper.ZooKeeper$ZKWatchManager.containsWatcher(ZooKeeper.java:377) >> at >> org.apache.zookeeper.ZooKeeper$ZKWatchManager.removeWatcher(ZooKeeper.java:252) >> at >> org.apache.zookeeper.WatchDeregistration.unregister(WatchDeregistration.java:58) >> at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:712) >> at org.apache.zookeeper.ClientCnxn.access$1500(ClientCnxn.java:97) >> at >> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:948) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:99) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1236) >> >> Is it possible it's something to do with the way that the cluster is >> restarted at line 282? The old cluster is not shutdown, a new one is just >> created. >> >> >> On Wed, Jun 1, 2016 at 10:44 AM, Jordan Zimmerman < >> [email protected]> wrote: >> >>> I’ll try to address this as part of CURATOR-333 >>> >>> > On May 31, 2016, at 7:08 PM, Cameron McKenzie <[email protected]> >>> wrote: >>> > >>> > Maybe we need to look at some way of providing a hook for tests to wait >>> > reliably for asynch tasks to finish? >>> > >>> > The latest round of tests ran OK. One test failed on an unrelated thing >>> > (ConnectionLoss), but this appears to be a transient thing as it's >>> worked >>> > ok the next time around. >>> > >>> > I will start getting a release together. Thanks for you help with the >>> > updated tests. >>> > cheers >>> > >>> > On Wed, Jun 1, 2016 at 9:12 AM, Jordan Zimmerman < >>> [email protected] >>> >> wrote: >>> > >>> >> The problem is in-flight watchers and async background calls. There’s >>> no >>> >> way to cancel these and they can take time to occur - even after a >>> recipe >>> >> instance is closed. >>> >> >>> >> -Jordan >>> >> >>> >>> On May 31, 2016, at 5:11 PM, Cameron McKenzie < >>> [email protected]> >>> >> wrote: >>> >>> >>> >>> Ok, running it again now. >>> >>> >>> >>> Is the problem that the watcher clean up for the recipes is done >>> >>> asynchronously after they are closed? >>> >>> >>> >>> On Wed, Jun 1, 2016 at 1:35 AM, Jordan Zimmerman < >>> >> [email protected] >>> >>>> wrote: >>> >>> >>> >>>> OK - please try now. I added a loop in the “no watchers” checker. If >>> >> there >>> >>>> are remaining watchers, it will sleep a bit and try again. >>> >>>> >>> >>>> -Jordan >>> >>>> >>> >>>>> On May 31, 2016, at 1:33 AM, Cameron McKenzie < >>> [email protected]> >>> >>>> wrote: >>> >>>>> >>> >>>>> Looks like these failures are intermittent. Running them directly >>> in >>> >>>>> Eclipse they seem to be passing. I will run the whole thing again >>> in >>> >> the >>> >>>>> morning and see how it goes. >>> >>>>> >>> >>>>> On Tue, May 31, 2016 at 2:29 PM, Cameron McKenzie < >>> >>>> [email protected]> >>> >>>>> wrote: >>> >>>>> >>> >>>>>> There are still 2 tests failing for me: >>> >>>>>> >>> >>>>>> FAILURE! - in >>> >>>>>> org.apache.curator.framework.recipes.cache.TestPathChildrenCache >>> >>>>>> >>> >>>> >>> >> >>> testKilledSession(org.apache.curator.framework.recipes.cache.TestPathChildrenCache) >>> >>>>>> Time elapsed: 17.488 sec <<< FAILURE! >>> >>>>>> java.lang.AssertionError: One or more child watchers are still >>> >>>> registered: >>> >>>>>> [/test] >>> >>>>>> at >>> >>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.imps.TestCleanState.closeAndTestClean(TestCleanState.java:53) >>> >>>>>> at >>> >>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testKilledSession(TestPathChildrenCache.java:707) >>> >>>>>> >>> >>>>>> FAILURE! - in >>> >>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.locks.TestInterProcessSemaphoreCluster >>> >>>>>> >>> >>>> >>> >> >>> testCluster(org.apache.curator.framework.recipes.locks.TestInterProcessSemaphoreCluster) >>> >>>>>> Time elapsed: 96.641 sec <<< FAILURE! >>> >>>>>> java.lang.AssertionError: expected [true] but found [false] >>> >>>>>> at org.testng.Assert.fail(Assert.java:94) >>> >>>>>> at org.testng.Assert.failNotEquals(Assert.java:494) >>> >>>>>> at org.testng.Assert.assertTrue(Assert.java:42) >>> >>>>>> at org.testng.Assert.assertTrue(Assert.java:52) >>> >>>>>> at >>> >>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.locks.TestInterProcessSemaphoreCluster.testCluster(TestInterProcessSemaphoreCluster.java:294) >>> >>>>>> >>> >>>>>> Failed tests: >>> >>>>>> >>> >>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testKilledSession(org.apache.curator.framework.recipes.cache.TestPathChildrenCache) >>> >>>>>> Run 1: TestPathChildrenCache.testKilledSession:707 One or more >>> child >>> >>>>>> watchers are still registered: [/test] >>> >>>>>> Run 2: PASS >>> >>>>>> >>> >>>>>> TestInterProcessSemaphoreCluster.testCluster:294 expected [true] >>> but >>> >>>>>> found [false] >>> >>>>>> >>> >>>>>> Tests run: 495, Failures: 2, Errors: 0, Skipped: 0 >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> On Tue, May 31, 2016 at 12:52 PM, Cameron McKenzie < >>> >>>> [email protected] >>> >>>>>>> wrote: >>> >>>>>> >>> >>>>>>> Thanks, CURATOR-332 wasn't pushed. I will run the tests against >>> that, >>> >>>> and >>> >>>>>>> if it's all good will merge into CURATOR-3.0 >>> >>>>>>> cheers >>> >>>>>>> >>> >>>>>>> On Tue, May 31, 2016 at 12:32 PM, Jordan Zimmerman < >>> >>>>>>> [email protected]> wrote: >>> >>>>>>> >>> >>>>>>>> Actually - I don’t remember if branch CURATOR-332 is merged >>> yet. I >>> >>>>>>>> made/pushed my changes in CURATOR-332 >>> >>>>>>>> >>> >>>>>>>> -jordan >>> >>>>>>>> >>> >>>>>>>>> On May 26, 2016, at 10:04 PM, Cameron McKenzie < >>> >>>> [email protected]> >>> >>>>>>>> wrote: >>> >>>>>>>>> >>> >>>>>>>>> I'm still seeing 6 failed tests that seem related to the same >>> stuff >>> >>>>>>>> after >>> >>>>>>>>> merging your fix: >>> >>>>>>>>> >>> >>>>>>>>> Failed tests: >>> >>>>>>>>> >>> >>>>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testBasics(org.apache.curator.framework.recipes.cache.TestPathChildrenCache) >>> >>>>>>>>> Run 1: TestPathChildrenCache.testBasics:863 One or more child >>> >>>> watchers >>> >>>>>>>>> are still registered: [/test] >>> >>>>>>>>> Run 2: TestPathChildrenCache.testBasics:863 One or more child >>> >>>> watchers >>> >>>>>>>>> are still registered: [/test] >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testBasicsOnTwoCachesWithSameExecutor(org.apache.curator.framework.recipes.cache.TestPathChildrenCache) >>> >>>>>>>>> Run 1: >>> >>>> TestPathChildrenCache.testBasicsOnTwoCachesWithSameExecutor:934 >>> >>>>>>>>> One or more child watchers are still registered: [/test] >>> >>>>>>>>> Run 2: >>> >>>> TestPathChildrenCache.testBasicsOnTwoCachesWithSameExecutor:934 >>> >>>>>>>>> One or more child watchers are still registered: [/test] >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testEnsurePath(org.apache.curator.framework.recipes.cache.TestPathChildrenCache) >>> >>>>>>>>> Run 1: TestPathChildrenCache.testEnsurePath:363 One or more >>> child >>> >>>>>>>>> watchers are still registered: [/one/two/three] >>> >>>>>>>>> Run 2: TestPathChildrenCache.testEnsurePath:363 One or more >>> child >>> >>>>>>>>> watchers are still registered: [/one/two/three] >>> >>>>>>>>> >>> >>>>>>>>> TestInterProcessSemaphoreCluster.testCluster:294 expected >>> [true] >>> >> but >>> >>>>>>>>> found [false] >>> >>>>>>>>> >>> >>>>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.shared.TestSharedCount.testMultiClientVersioned(org.apache.curator.framework.recipes.shared.TestSharedCount) >>> >>>>>>>>> Run 1: PASS >>> >>>>>>>>> Run 2: TestSharedCount.testMultiClientVersioned:256 One or more >>> >> data >>> >>>>>>>>> watchers are still registered: [/count] >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>> >>> >>>> >>> >> >>> org.apache.curator.framework.recipes.shared.TestSharedCount.testSimple(org.apache.curator.framework.recipes.shared.TestSharedCount) >>> >>>>>>>>> Run 1: TestSharedCount.testSimple:174 One or more data >>> watchers are >>> >>>>>>>> still >>> >>>>>>>>> registered: [/count] >>> >>>>>>>>> Run 2: TestSharedCount.testSimple:174 One or more data >>> watchers are >>> >>>>>>>> still >>> >>>>>>>>> registered: [/count] >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> Tests run: 491, Failures: 6, Errors: 0, Skipped: 0 >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> On Fri, May 27, 2016 at 3:30 AM, Jordan Zimmerman < >>> >>>>>>>>> [email protected]> wrote: >>> >>>>>>>>> >>> >>>>>>>>>> I see the problem. The fix is not simple though so I’ll spend >>> some >>> >>>>>>>> time on >>> >>>>>>>>>> it. The TL;DR is that exists watchers are still supposed to >>> get >>> >> set >>> >>>>>>>> when >>> >>>>>>>>>> there is a KeeperException.NoNode and the code isn’t handling >>> it. >>> >>>> But, >>> >>>>>>>>>> while I was looking at the code I realized there are some >>> >>>> significant >>> >>>>>>>>>> additional problems. Curator, here, is trying to mirror what >>> >>>>>>>> ZooKeeper does >>> >>>>>>>>>> internally which is insanely complicated. In hindsight, the >>> whole >>> >> ZK >>> >>>>>>>>>> watcher mechanism should’ve been decoupled from the mutator >>> APIs. >>> >>>>>>>> But, of >>> >>>>>>>>>> course, that’s easy for me to say now. >>> >>>>>>>>>> >>> >>>>>>>>>> -Jordan >>> >>>>>>>>>> >>> >>>>>>>>>>> On May 26, 2016, at 1:10 AM, Cameron McKenzie < >>> >>>>>>>> [email protected]> >>> >>>>>>>>>> wrote: >>> >>>>>>>>>>> >>> >>>>>>>>>>> Thanks Scott, >>> >>>>>>>>>>> Those tests are now passing for me. >>> >>>>>>>>>>> >>> >>>>>>>>>>> Jordan, testNodeCache:testBasics() is failing consistently >>> on the >>> >>>> 3.0 >>> >>>>>>>>>>> branch. It appears that this is actually potentially a bug >>> in the >>> >>>>>>>>>>> NodeCache. It ends up leaking a Watcher reference. I've had a >>> >> quick >>> >>>>>>>> look >>> >>>>>>>>>>> through, but I haven't dived in in any detail. It's the >>> >>>>>>>>>>> WatcherRemovalManager stuff I think. If you've got time, can >>> you >>> >>>>>>>> have a >>> >>>>>>>>>>> look? If not, let me know and I'll do some more digging. >>> >>>>>>>>>>> >>> >>>>>>>>>>> cheers >>> >>>>>>>>>>> >>> >>>>>>>>>>> On Thu, May 26, 2016 at 11:47 AM, Cameron McKenzie < >>> >>>>>>>>>> [email protected]> >>> >>>>>>>>>>> wrote: >>> >>>>>>>>>>> >>> >>>>>>>>>>>> Thanks Scott. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Push the fix to master and merge it into 3.0. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Then I guess, I'll push new versions of 2.11 and 3.2 onto >>> Nexus. >>> >>>>>>>>>>>> cheers >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> On Thu, May 26, 2016 at 11:44 AM, Scott Blum < >>> >>>> [email protected] >>> >>>>>>>>> >>> >>>>>>>>>>>> wrote: >>> >>>>>>>>>>>> >>> >>>>>>>>>>>>> Alright, I have a fix, but it wants to be applied to both >>> >> master >>> >>>>>>>> and >>> >>>>>>>>>> 3.0. >>> >>>>>>>>>>>>> Where should I push the fix? >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> On Wed, May 25, 2016 at 6:10 PM, Cameron McKenzie < >>> >>>>>>>>>> [email protected] >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Thanks Scott, >>> >>>>>>>>>>>>>> If you just checkout the CURATOR-3.0 branch, they are >>> failing >>> >>>>>>>> there. >>> >>>>>>>>>>>>>> cheers >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> On Thu, May 26, 2016 at 2:06 AM, Scott Blum < >>> >>>>>>>> [email protected]> >>> >>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> Sure, what SHA are they failing at Cam? >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> On Wed, May 25, 2016 at 9:36 AM, Jordan Zimmerman < >>> >>>>>>>>>>>>>>> [email protected]> wrote: >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> Scott can you take a look? >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> -Jordan >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> On May 25, 2016, at 4:35 AM, Cameron McKenzie < >>> >>>>>>>>>>>>>> [email protected]> >>> >>>>>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> Tree cache tests are still failing. I've tried a few >>> times >>> >>>> but >>> >>>>>>>> no >>> >>>>>>>>>>>>>> love: >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>> >>> TestTreeCacheEventOrdering>TestEventOrdering.testEventOrdering:151 >>> >>>>>>>>>>>>>>>> actual 6 >>> >>>>>>>>>>>>>>>>> expected -31: >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> I will have a look into what's going on in the morning. >>> >> Given >>> >>>>>>>> that >>> >>>>>>>>>>>>>>> these >>> >>>>>>>>>>>>>>>>> may take some messing about to fix up, do we just want >>> to >>> >>>> vote >>> >>>>>>>> on >>> >>>>>>>>>>>>>>> 2.11.0 >>> >>>>>>>>>>>>>>>>> separately, as that is all ready to go? >>> >>>>>>>>>>>>>>>>> cheers >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 5:34 PM, Jordan Zimmerman < >>> >>>>>>>>>>>>>>>>> [email protected]> wrote: >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> Great news. Thanks. >>> >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> ==================== >>> >>>>>>>>>>>>>>>>>> Jordan Zimmerman >>> >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> On May 25, 2016, at 12:37 AM, Cameron McKenzie < >>> >>>>>>>>>>>>>>> [email protected] >>> >>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> I have fixed up the test case, all good now. >>> >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 1:45 PM, Cameron McKenzie < >>> >>>>>>>>>>>>>>>>>> [email protected]> >>> >>>>>>>>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>> Looks like it was introduced with the schema >>> validation >>> >>>>>>>> stuff. >>> >>>>>>>>>>>>> It >>> >>>>>>>>>>>>>>> now >>> >>>>>>>>>>>>>>>>>> does >>> >>>>>>>>>>>>>>>>>>>> an ACL check prior to the backgrounding call. >>> Because >>> >> the >>> >>>>>>>> unit >>> >>>>>>>>>>>>>> test >>> >>>>>>>>>>>>>>>>>> uses a >>> >>>>>>>>>>>>>>>>>>>> bogus ACL provider it just throws an exception >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>> final String adjustedPath = >>> >>>>>>>>>>>>>>>>>>>> adjustPath(client.fixForNamespace(givenPath, >>> >>>>>>>>>>>>>>>>>> createMode.isSequential())); >>> >>>>>>>>>>>>>>>>>>>> List<ACL> aclList = >>> acling.getAclList(adjustedPath); >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>> >>> >> client.getSchemaSet().getSchema(givenPath).validateCreate(createMode, >>> >>>>>>>>>>>>>>>>>> data, >>> >>>>>>>>>>>>>>>>>>>> aclList); >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>> String returnPath = null; >>> >>>>>>>>>>>>>>>>>>>> if ( backgrounding.inBackground() ) >>> >>>>>>>>>>>>>>>>>>>> { >>> >>>>>>>>>>>>>>>>>>>> pathInBackground(adjustedPath, data, >>> givenPath); >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>> So, I guess the answer is to get the test to force a >>> >>>> failure >>> >>>>>>>>>>>>> in a >>> >>>>>>>>>>>>>>>>>>>> different way. With the UnhandledErrorListener, the >>> >>>>>>>>>>>>> expectation is >>> >>>>>>>>>>>>>>>> that >>> >>>>>>>>>>>>>>>>>>>> this only gets called on backgrounding operations? >>> >>>>>>>>>>>>>>>>>>>> cheers >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 1:39 PM, Cameron McKenzie < >>> >>>>>>>>>>>>>>>>>> [email protected]> >>> >>>>>>>>>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>> Same on the master branch, but it passes there, so >>> >> maybe >>> >>>>>>>>>>>>>> something >>> >>>>>>>>>>>>>>>> has >>> >>>>>>>>>>>>>>>>>>>>> legitimately broken the test. Will let you know if >>> I >>> >> get >>> >>>>>>>>>>>>> stuck. >>> >>>>>>>>>>>>>>>>>>>>> cheers >>> >>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 1:36 PM, Jordan Zimmerman < >>> >>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>> >>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>> Let me know if you need help. >>> >>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>> It might be a bad merge. Have you compared it to >>> the >>> >>>>>>>> master >>> >>>>>>>>>>>>>>> branch? >>> >>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>> -JZ >>> >>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>>> On May 24, 2016, at 10:31 PM, Cameron McKenzie < >>> >>>>>>>>>>>>>>>>>> [email protected]> >>> >>>>>>>>>>>>>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>> Guys, >>> >>>>>>>>>>>>>>>>>>>>>>> There's a test >>> >>>> TestFrameworkBackground:testErrorListener >>> >>>>>>>>>>>>> that >>> >>>>>>>>>>>>>> is >>> >>>>>>>>>>>>>>>>>>>>>> failing >>> >>>>>>>>>>>>>>>>>>>>>>> consistently on the CURATOR-3.0 branch. >>> >>>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>> I can't see how it has ever worked. It seems to >>> try >>> >> and >>> >>>>>>>>>>>>> provoke >>> >>>>>>>>>>>>>>> an >>> >>>>>>>>>>>>>>>>>>>>>> error >>> >>>>>>>>>>>>>>>>>>>>>>> via a bad ACL provider. >>> >>>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>> But this ACL provider is called by the >>> >>>> CreateBuilderImpl >>> >>>>>>>>>>>>> prior >>> >>>>>>>>>>>>>> to >>> >>>>>>>>>>>>>>>> the >>> >>>>>>>>>>>>>>>>>>>>>>> backgrounding call, which means that the >>> exception >>> >> that >>> >>>>>>>> it >>> >>>>>>>>>>>>>> throws >>> >>>>>>>>>>>>>>>>>>>>>> happens >>> >>>>>>>>>>>>>>>>>>>>>>> in the main Thread of the unit test. So, it just >>> >> throws >>> >>>>>>>> an >>> >>>>>>>>>>>>>>>>>>>>>>> UnsupportedOperationException which is >>> propogated up >>> >>>> the >>> >>>>>>>>>>>>> stack >>> >>>>>>>>>>>>>> at >>> >>>>>>>>>>>>>>>>>> which >>> >>>>>>>>>>>>>>>>>>>>>>> point the unit test fails. >>> >>>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>> So, I will look at fixing this, but I just don't >>> >>>>>>>> understand >>> >>>>>>>>>>>>> how >>> >>>>>>>>>>>>>>> it >>> >>>>>>>>>>>>>>>>>> ever >>> >>>>>>>>>>>>>>>>>>>>>>> worked? >>> >>>>>>>>>>>>>>>>>>>>>>> cheers >>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>> >>> >>>>>> >>> >>>> >>> >>>> >>> >> >>> >> >>> >>> >> >
