Re: Reducing the unit test running time
On 16 January 2012 17:36, Patrick Hunt wrote: > On Sun, Jan 15, 2012 at 11:39 PM, Henry Robinson > wrote: > > Hi - > > > > The unit tests are taking longer and longer to run, particularly > locally. I > > was poking about looking for some easy wins, and I noticed that a lot of > > the time is spent waiting for servers to come up, which is heavily > > dependent on the tick time. Lo and behold, dropping the tick time on (for > > example) QuorumPeerMainTest from 4s to 100ms made the test suite quicker > by > > 30s. > > > > On builds.apache.org it's not a great idea to reduce the tick time too > far > > because it generally runs on more contended hardware so timeouts get hit, > > but what if we just increase the session expiration time commensurately? > We > > could set a 500ms tick time with a 30s (or more) max session expiration > > time. Latencies due to waiting for servers to start should be lower, but > > the tests should remain as stable. > > > > Any thoughts? Any other ways we can tighten up the test suite runtime? > > I'd be concerned that we were testing with a different setting than > most users set. Would we be more or less likely to find issues by > setting this lower? > That's a good point, but I don't know that we can really say the tests as they stand are at all representative of what real users are doing. The unit tests have ensembles co-located on the same machine, with very synthetic workloads - I don't think they mimic production environments, nor should they. Many of the tests start a cluster and then wait for some condition to be true, or a timeout to occur (then aborting). I'm suggesting keeping most of the timeouts to be similar lengths, but to poll more frequently so that we don't waste time waiting to wake up to check the condition, if that makes sense. I like your idea of splitting tests into categories. I think a lot of the current tests should exist in the test-commit category but currently take a bit long to run. The 'hammer' tests are great examples of tests that should be in the full suite, since they're not really testing for a specific property, but the QuorumPeerMain tests are mostly testing a very specific scenario. I filed ZOOKEEPER-1363 to deal with splitting the tests up by category (ZOOKEEPER-725, the only other place I saw this mentioned, is a bit more general). Henry > > re "other ways": > > In the past I've found that test time reductions could be had by > looking at the longest running tests for flaws. Often a test will set > a session time of 30seconds and wait for expiration, or sleep for some > long/unnecessary period of time. I'll typically refactor the test to > improve the runtime. In past releases I've made significant > improvements using this method (perhaps mined out?) > > Another option is to restart the server(s) less frequently. This can > be done by starting the service once for all tests in a class, rather > than for each test method. (non-optimal though) > > Others would probably point out that what we call "unit tests" are > pretty much system tests and should be moved out. That seems unlikely > at this time however. > > Given that tests typically increase in scope (and time) and not > decrease we might want to consider moving to the approach that Pig and > some other projects have. They have test targets that run a subset of > the test suite. For example in Pig "test" takes 6-8 hrs, however they > have a "test-commit" which only takes 20min or so. We could do similar > for ZK. This is easy to do using "exclude files". (see pig build.xml) > > IMO long term we should categorize our tests, asking ppl to run > "test-commit" (short subset) prior to committing, whereas "test" (full > suite) would run as part of the patch testing, nightly testing, > release testing, etc... > > Patrick > -- Henry Robinson Software Engineer Cloudera 415-994-6679
[jira] [Created] (ZOOKEEPER-1363) Categorise unit tests by 'test-commit', 'full-test' etc
Categorise unit tests by 'test-commit', 'full-test' etc --- Key: ZOOKEEPER-1363 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1363 Project: ZooKeeper Issue Type: Improvement Reporter: Henry Robinson As discussed on the list, it would be good to split the Java test suite into categories so that it's easy to run a small set of unit tests against a patch, and to leave Jenkins to run the full suite of stress tests etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1294) One of the zookeeper server is not accepting any requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187483#comment-13187483 ] kavita sharma commented on ZOOKEEPER-1294: -- Thanks Henry. > One of the zookeeper server is not accepting any requests > - > > Key: ZOOKEEPER-1294 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1294 > Project: ZooKeeper > Issue Type: Bug > Components: server > Environment: 3 Zookeeper + 3 Observer with SuSe-11 >Reporter: amith >Assignee: kavita sharma > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1294-1.patch, ZOOKEEPER-1294-2.patch, > ZOOKEEPER-1294-3.patch, ZOOKEEPER-1294.patch > > > In zoo.cfg i have configured as > server.1 = XX.XX.XX.XX:65175:65173 > server.2 = XX.XX.XX.XX:65185:65183 > server.3 = XX.XX.XX.XX:65195:65193 > server.4 = XX.XX.XX.XX:65205:65203:observer > server.5 = XX.XX.XX.XX:65215:65213:observer > server.6 = XX.XX.XX.XX:65225:65223:observer > Like above I have configured 3 PARTICIPANTS and 3 OBSERVERS > in the cluster of 6 zookeepers > Steps to reproduce the defect > 1. Start all the 3 participant zookeeper > 2. Stop all the participant zookeeper > 3. Start zookeeper 1(Participant) > 4. Start zookeeper 2(Participant) > 5. Start zookeeper 4(Observer) > 6. Create a persistent node with external client and close it > 7. Stop the zookeeper 1(Participant neo quorum is unstable) > 8. Create a new client and try to find the node created b4 using exists api > (will fail since quorum not statisfied) > 9. Start the Zookeeper 1 (Participant stabilise the quorum) > Now check the observer using 4 letter word (Server.4) > linux-216:/home/amith/CI/source/install/zookeeper/zookeeper2/bin # echo stat > | netcat localhost 65200 > Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT > Clients: > /127.0.0.1:46370[0](queued=0,recved=1,sent=0) > Latency min/avg/max: 0/0/0 > Received: 1 > Sent: 0 > Outstanding: 0 > Zxid: 0x10003 > Mode: observer > Node count: 5 > check the participant 2 with 4 letter word > Latency min/avg/max: 22/48/83 > Received: 39 > Sent: 3 > Outstanding: 35 > Zxid: 0x10003 > Mode: leader > Node count: 5 > linux-216:/home/amith/CI/source/install/zookeeper/zookeeper2/bin # > check the participant 1 with 4 letter word > linux-216:/home/amith/CI/source/install/zookeeper/zookeeper2/bin # echo stat > | netcat localhost 65170 > This ZooKeeper instance is not currently serving requests > We can see the participant1 logs filled with > 2011-11-08 15:49:51,360 - WARN > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:65170:NIOServerCnxn@642] - Exception > causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not > running > Problem here is participent1 is not responding / accepting any requests -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (ZOOKEEPER-706) large numbers of watches can cause session re-establishment to fail
[ https://issues.apache.org/jira/browse/ZOOKEEPER-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kavita sharma reassigned ZOOKEEPER-706: --- Assignee: (was: kavita sharma) > large numbers of watches can cause session re-establishment to fail > --- > > Key: ZOOKEEPER-706 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-706 > Project: ZooKeeper > Issue Type: Bug > Components: c client, java client >Affects Versions: 3.1.2, 3.2.2, 3.3.0 >Reporter: Patrick Hunt >Priority: Critical > Fix For: 3.5.0 > > > If a client sets a large number of watches the "set watches" operation during > session re-establishment can fail. > for example: > WARN [NIOServerCxn.Factory:22801:NIOServerCnxn@417] - Exception causing > close of session 0xe727001201a4ee7c due to java.io.IOException: Len error > 4348380 > in this case the client was a web monitoring app and had set both data and > child watches on > 32k znodes. > there are two issues I see here we need to fix: > 1) handle this case properly (split up the set watches into multiple calls I > guess...) > 2) the session should have expired after the "timeout". however we seem to > consider any message from the client as re-setting the expiration on the > server side. Probably we should only consider messages from the client that > are sent during an established session, otherwise we can see this situation > where the session is not established however the session is not expired > either. Perhaps we should create another JIRA for this particular issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (ZOOKEEPER-706) large numbers of watches can cause session re-establishment to fail
[ https://issues.apache.org/jira/browse/ZOOKEEPER-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kavita sharma reassigned ZOOKEEPER-706: --- Assignee: kavita sharma > large numbers of watches can cause session re-establishment to fail > --- > > Key: ZOOKEEPER-706 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-706 > Project: ZooKeeper > Issue Type: Bug > Components: c client, java client >Affects Versions: 3.1.2, 3.2.2, 3.3.0 >Reporter: Patrick Hunt >Assignee: kavita sharma >Priority: Critical > Fix For: 3.5.0 > > > If a client sets a large number of watches the "set watches" operation during > session re-establishment can fail. > for example: > WARN [NIOServerCxn.Factory:22801:NIOServerCnxn@417] - Exception causing > close of session 0xe727001201a4ee7c due to java.io.IOException: Len error > 4348380 > in this case the client was a web monitoring app and had set both data and > child watches on > 32k znodes. > there are two issues I see here we need to fix: > 1) handle this case properly (split up the set watches into multiple calls I > guess...) > 2) the session should have expired after the "timeout". however we seem to > consider any message from the client as re-setting the expiration on the > server side. Probably we should only consider messages from the client that > are sent during an established session, otherwise we can see this situation > where the session is not established however the session is not expired > either. Perhaps we should create another JIRA for this particular issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1337) multi's "Transaction" class is missing tests.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187386#comment-13187386 ] Hadoop QA commented on ZOOKEEPER-1337: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510788/zk-1337-update.patch against trunk revision 1231809. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/909//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/909//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/909//console This message is automatically generated. > multi's "Transaction" class is missing tests. > - > > Key: ZOOKEEPER-1337 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1337 > Project: ZooKeeper > Issue Type: Test > Components: java client >Affects Versions: 3.4.0 >Reporter: Patrick Hunt >Assignee: Patrick Hunt >Priority: Minor > Fix For: 3.4.3, 3.5.0 > > Attachments: ZOOKEEPER-1337.patch, zk-1337-update.patch > > > Add tests for zookeeper client transaction() method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Success: ZOOKEEPER-1337 PreCommit Build #909
Jira: https://issues.apache.org/jira/browse/ZOOKEEPER-1337 Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/909/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 148062 lines...] [exec] BUILD SUCCESSFUL [exec] Total time: 0 seconds [exec] [exec] [exec] [exec] [exec] +1 overall. Here are the results of testing the latest attachment [exec] http://issues.apache.org/jira/secure/attachment/12510788/zk-1337-update.patch [exec] against trunk revision 1231809. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/909//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/909//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/909//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] k74WH7VM03 logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] BUILD SUCCESSFUL Total time: 24 minutes 25 seconds Archiving artifacts Recording test results Description set: ZOOKEEPER-1337 Email was triggered for: Success Sending email for trigger: Success ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Updated] (ZOOKEEPER-1337) multi's "Transaction" class is missing tests.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Camille Fournier updated ZOOKEEPER-1337: Attachment: zk-1337-update.patch update after changing other tests that overlap in this class > multi's "Transaction" class is missing tests. > - > > Key: ZOOKEEPER-1337 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1337 > Project: ZooKeeper > Issue Type: Test > Components: java client >Affects Versions: 3.4.0 >Reporter: Patrick Hunt >Assignee: Patrick Hunt >Priority: Minor > Fix For: 3.4.3, 3.5.0 > > Attachments: ZOOKEEPER-1337.patch, zk-1337-update.patch > > > Add tests for zookeeper client transaction() method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Reducing the unit test running time
On Sun, Jan 15, 2012 at 11:39 PM, Henry Robinson wrote: > Hi - > > The unit tests are taking longer and longer to run, particularly locally. I > was poking about looking for some easy wins, and I noticed that a lot of > the time is spent waiting for servers to come up, which is heavily > dependent on the tick time. Lo and behold, dropping the tick time on (for > example) QuorumPeerMainTest from 4s to 100ms made the test suite quicker by > 30s. > > On builds.apache.org it's not a great idea to reduce the tick time too far > because it generally runs on more contended hardware so timeouts get hit, > but what if we just increase the session expiration time commensurately? We > could set a 500ms tick time with a 30s (or more) max session expiration > time. Latencies due to waiting for servers to start should be lower, but > the tests should remain as stable. > > Any thoughts? Any other ways we can tighten up the test suite runtime? I'd be concerned that we were testing with a different setting than most users set. Would we be more or less likely to find issues by setting this lower? re "other ways": In the past I've found that test time reductions could be had by looking at the longest running tests for flaws. Often a test will set a session time of 30seconds and wait for expiration, or sleep for some long/unnecessary period of time. I'll typically refactor the test to improve the runtime. In past releases I've made significant improvements using this method (perhaps mined out?) Another option is to restart the server(s) less frequently. This can be done by starting the service once for all tests in a class, rather than for each test method. (non-optimal though) Others would probably point out that what we call "unit tests" are pretty much system tests and should be moved out. That seems unlikely at this time however. Given that tests typically increase in scope (and time) and not decrease we might want to consider moving to the approach that Pig and some other projects have. They have test targets that run a subset of the test suite. For example in Pig "test" takes 6-8 hrs, however they have a "test-commit" which only takes 20min or so. We could do similar for ZK. This is easy to do using "exclude files". (see pig build.xml) IMO long term we should categorize our tests, asking ppl to run "test-commit" (short subset) prior to committing, whereas "test" (full suite) would run as part of the patch testing, nightly testing, release testing, etc... Patrick
[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187353#comment-13187353 ] Hadoop QA commented on ZOOKEEPER-1355: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510782/loadbalancing-more-details.pdf against trunk revision 1231809. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/908//console This message is automatically generated. > Add zk.updateServerList(newServerList) > --- > > Key: ZOOKEEPER-1355 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355 > Project: ZooKeeper > Issue Type: New Feature > Components: java client >Reporter: Alexander Shraer >Assignee: Alexander Shraer > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, > ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, > ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf > > > When the set of servers changes, we would like to update the server list > stored by clients without restarting the clients. > Moreover, assuming that the number of clients per server is the same (in > expectation) in the old configuration (as guaranteed by the current list > shuffling for example), we would like to re-balance client connections across > the new set of servers in a way that a) the number of clients per server is > the same for all servers (in expectation) and b) there is no > excessive/unnecessary client migration. > It is simple to achieve (a) without (b) - just re-shuffle the new list of > servers at every client. But this would create unnecessary migration, which > we'd like to avoid. > We propose a simple probabilistic migration scheme that achieves (a) and (b) > - each client locally decides whether and where to migrate when the list of > servers changes. The attached document describes the scheme and shows an > evaluation of it in Zookeeper. We also implemented re-balancing through a > consistent-hashing scheme and show a comparison. We derived the probabilistic > migration rules from a simple formula that we can also provide, if someone's > interested in the proof. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Failed: ZOOKEEPER-1355 PreCommit Build #908
Jira: https://issues.apache.org/jira/browse/ZOOKEEPER-1355 Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/908/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 64 lines...] [exec] == [exec] == [exec] Applying patch. [exec] == [exec] == [exec] [exec] [exec] /usr/bin/patch: Only garbage was found in the patch input. [exec] PATCH APPLICATION FAILED [exec] [exec] [exec] [exec] [exec] -1 overall. Here are the results of testing the latest attachment [exec] http://issues.apache.org/jira/secure/attachment/12510782/loadbalancing-more-details.pdf [exec] against trunk revision 1231809. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] -1 patch. The patch command could not apply the patch. [exec] [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/908//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] HT7EL435ak logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/build.xml:1567: exec returned: 1 Total time: 42 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Description set: ZOOKEEPER-1355 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Shraer updated ZOOKEEPER-1355: Attachment: loadbalancing-more-details.pdf some more details added to the design document. Would be great if more committers comment on this patch. thanks a lot, alex > Add zk.updateServerList(newServerList) > --- > > Key: ZOOKEEPER-1355 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355 > Project: ZooKeeper > Issue Type: New Feature > Components: java client >Reporter: Alexander Shraer >Assignee: Alexander Shraer > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, > ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, > ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf > > > When the set of servers changes, we would like to update the server list > stored by clients without restarting the clients. > Moreover, assuming that the number of clients per server is the same (in > expectation) in the old configuration (as guaranteed by the current list > shuffling for example), we would like to re-balance client connections across > the new set of servers in a way that a) the number of clients per server is > the same for all servers (in expectation) and b) there is no > excessive/unnecessary client migration. > It is simple to achieve (a) without (b) - just re-shuffle the new list of > servers at every client. But this would create unnecessary migration, which > we'd like to avoid. > We propose a simple probabilistic migration scheme that achieves (a) and (b) > - each client locally decides whether and where to migrate when the list of > servers changes. The attached document describes the scheme and shows an > evaluation of it in Zookeeper. We also implemented re-balancing through a > consistent-hashing scheme and show a comparison. We derived the probabilistic > migration rules from a simple formula that we can also provide, if someone's > interested in the proof. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1362) ZooDefs.Ids ACL lists not immutable
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187327#comment-13187327 ] Tassos Souris commented on ZOOKEEPER-1362: -- ignore the above.. rushed to answer... can't delete it though > ZooDefs.Ids ACL lists not immutable > --- > > Key: ZOOKEEPER-1362 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1362 > Project: ZooKeeper > Issue Type: Improvement > Components: java client >Reporter: Tassos Souris >Priority: Trivial > > In org.apache.zookeeper: > 1) ZooDefs.Ids.OPEN_ACL_UNSAFE > 2) ZooDefs.Ids.CREATOR_ALL_ACL > 3) ZooDefs.Ids.READ_ALL_ACL > are not immutable lists. Unlikely but the client could alter them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1362) ZooDefs.Ids ACL lists not immutable
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187312#comment-13187312 ] Tassos Souris commented on ZOOKEEPER-1362: -- or keep the ArrayList and use unmodifiableList: public final ArrayList OPEN_ACL_UNSAFE = Collections.unmodifiableList( new ArrayList( Collections.singletonList(new ACL(Perms.ALL, ANYONE_ID_UNSAFE)) )); > ZooDefs.Ids ACL lists not immutable > --- > > Key: ZOOKEEPER-1362 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1362 > Project: ZooKeeper > Issue Type: Improvement > Components: java client >Reporter: Tassos Souris >Priority: Trivial > > In org.apache.zookeeper: > 1) ZooDefs.Ids.OPEN_ACL_UNSAFE > 2) ZooDefs.Ids.CREATOR_ALL_ACL > 3) ZooDefs.Ids.READ_ALL_ACL > are not immutable lists. Unlikely but the client could alter them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1362) ZooDefs.Ids ACL lists not immutable
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187212#comment-13187212 ] Edward Ribeiro commented on ZOOKEEPER-1362: --- All three fields are declared as follows: public final ArrayList OPEN_ACL_UNSAFE = new ArrayList( Collections.singletonList(new ACL(Perms.ALL, ANYONE_ID_UNSAFE))); The change would be a matter of just removing the ArrayList and declaring the field as a List as Collections.singletonList() already returns an immutable collection: public final List OPEN_ACL_UNSAFE = Collections.singletonList(new ACL(Perms.ALL, ANYONE_ID_UNSAFE)); Unfortunately, they are public fields, so I suppose it could pontentially broke many client apps that are referencing this field as ArrayList perms = ZooDefs.Ids.OPEN_ACL_UNSAFE; I still think this change is doable, but I would like to hear what other people think. :) > ZooDefs.Ids ACL lists not immutable > --- > > Key: ZOOKEEPER-1362 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1362 > Project: ZooKeeper > Issue Type: Improvement > Components: java client >Reporter: Tassos Souris >Priority: Trivial > > In org.apache.zookeeper: > 1) ZooDefs.Ids.OPEN_ACL_UNSAFE > 2) ZooDefs.Ids.CREATOR_ALL_ACL > 3) ZooDefs.Ids.READ_ALL_ACL > are not immutable lists. Unlikely but the client could alter them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1178) Add eclipse target for supporting Apache IvyDE
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Warren Turkal updated ZOOKEEPER-1178: - Attachment: (was: ZOOKEEPER-1178.patch) > Add eclipse target for supporting Apache IvyDE > -- > > Key: ZOOKEEPER-1178 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1178 > Project: ZooKeeper > Issue Type: Improvement > Components: build > Environment: Mac OS X w/ Eclipse 3.7. However, I believe this will > work in any Eclipse environment. >Reporter: Warren Turkal >Assignee: Warren Turkal >Priority: Minor > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1178.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This patch adds support for Eclipse with Apache IvyDE, which is the extension > that integrates Ivy support into Eclipse. This allows the creation of what > appear to be fully portable .eclipse and .classpath files. I will be posting > a patch shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1178) Add eclipse target for supporting Apache IvyDE
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Warren Turkal updated ZOOKEEPER-1178: - Attachment: ZOOKEEPER-1178.patch Fixed the location of build class files. > Add eclipse target for supporting Apache IvyDE > -- > > Key: ZOOKEEPER-1178 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1178 > Project: ZooKeeper > Issue Type: Improvement > Components: build > Environment: Mac OS X w/ Eclipse 3.7. However, I believe this will > work in any Eclipse environment. >Reporter: Warren Turkal >Assignee: Warren Turkal >Priority: Minor > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1178.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This patch adds support for Eclipse with Apache IvyDE, which is the extension > that integrates Ivy support into Eclipse. This allows the creation of what > appear to be fully portable .eclipse and .classpath files. I will be posting > a patch shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (ZOOKEEPER-1362) ZooDefs.Ids ACL lists not immutable
ZooDefs.Ids ACL lists not immutable --- Key: ZOOKEEPER-1362 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1362 Project: ZooKeeper Issue Type: Improvement Components: java client Reporter: Tassos Souris Priority: Trivial In org.apache.zookeeper: 1) ZooDefs.Ids.OPEN_ACL_UNSAFE 2) ZooDefs.Ids.CREATOR_ALL_ACL 3) ZooDefs.Ids.READ_ALL_ACL are not immutable lists. Unlikely but the client could alter them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1178) Add eclipse target for supporting Apache IvyDE
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187075#comment-13187075 ] Warren Turkal commented on ZOOKEEPER-1178: -- Please give this a new look when you get a chance. It's slightly improved from the last version, and it really improves the eclipse workflow when compared to the current way that the build.xml has to generate the eclipse metadata. > Add eclipse target for supporting Apache IvyDE > -- > > Key: ZOOKEEPER-1178 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1178 > Project: ZooKeeper > Issue Type: Improvement > Components: build > Environment: Mac OS X w/ Eclipse 3.7. However, I believe this will > work in any Eclipse environment. >Reporter: Warren Turkal >Assignee: Warren Turkal >Priority: Minor > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1178.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This patch adds support for Eclipse with Apache IvyDE, which is the extension > that integrates Ivy support into Eclipse. This allows the creation of what > appear to be fully portable .eclipse and .classpath files. I will be posting > a patch shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1178) Add eclipse target for supporting Apache IvyDE
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Warren Turkal updated ZOOKEEPER-1178: - Attachment: (was: ZOOKEEPER-1178.patch) > Add eclipse target for supporting Apache IvyDE > -- > > Key: ZOOKEEPER-1178 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1178 > Project: ZooKeeper > Issue Type: Improvement > Components: build > Environment: Mac OS X w/ Eclipse 3.7. However, I believe this will > work in any Eclipse environment. >Reporter: Warren Turkal >Assignee: Warren Turkal >Priority: Minor > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1178.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This patch adds support for Eclipse with Apache IvyDE, which is the extension > that integrates Ivy support into Eclipse. This allows the creation of what > appear to be fully portable .eclipse and .classpath files. I will be posting > a patch shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1178) Add eclipse target for supporting Apache IvyDE
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Warren Turkal updated ZOOKEEPER-1178: - Attachment: ZOOKEEPER-1178.patch This patch makes the eclipse-with-ivyde rule depend on building the generated code. > Add eclipse target for supporting Apache IvyDE > -- > > Key: ZOOKEEPER-1178 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1178 > Project: ZooKeeper > Issue Type: Improvement > Components: build > Environment: Mac OS X w/ Eclipse 3.7. However, I believe this will > work in any Eclipse environment. >Reporter: Warren Turkal >Assignee: Warren Turkal >Priority: Minor > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1178.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This patch adds support for Eclipse with Apache IvyDE, which is the extension > that integrates Ivy support into Eclipse. This allows the creation of what > appear to be fully portable .eclipse and .classpath files. I will be posting > a patch shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1321) Add number of client connections metric in JMX and srvr
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187069#comment-13187069
]
Henry Robinson commented on ZOOKEEPER-1321:
---
+1 looks good! Only weirdness to my eyes is the following:
{code}
+public int getNumAliveConnections() {
+int numConnections;
+synchronized(cnxns) {
+numConnections = cnxns.size();
+}
+return numConnections;
+}
{code}
It's perfectly legal to return inside a synchronized block, so it might be more
concise to have:
{code}
+public int getNumAliveConnections() {
+synchronized(cnxns) {
+return cnxns.size();
+}
+}
{code}
If you fix this nit I'm happy for you to commit this without another review
pass.
> Add number of client connections metric in JMX and srvr
> ---
>
> Key: ZOOKEEPER-1321
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1321
> Project: ZooKeeper
> Issue Type: Improvement
>Affects Versions: 3.3.4, 3.4.2
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
> Labels: patch
> Attachments: ZOOKEEPER-1321_3.4.patch, ZOOKEEPER-1321_trunk.patch,
> ZOOKEEPER-1321_trunk.patch, zk-1321-cleanup, zookeeper-1321-trunk-v2.patch
>
>
> The related conversation on the zookeeper user mailing list is here -
> http://apache.markmail.org/message/4jjcmooniowwugu2?q=+list:org.apache.hadoop.zookeeper-user
> It is useful to be able to monitor the number of disconnect operations on a
> client. This is generally indicative of a client going through large number
> of GC and hence disconnecting way too often from a zookeeper cluster.
> Today, this information is only indirectly exposed as part of the stat
> command which requires counting the results. That's alot of work for the
> server to do just to get connection count.
> For monitoring purposes, it will be useful to have this exposed through JMX
> and 4lw srvr.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1321) Add number of client connections metric in JMX and srvr
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186975#comment-13186975 ] Camille Fournier commented on ZOOKEEPER-1321: - If one of the other committers wants take a quick glance at the cleanup patch that would be great, I can then check it in with your ok. > Add number of client connections metric in JMX and srvr > --- > > Key: ZOOKEEPER-1321 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1321 > Project: ZooKeeper > Issue Type: Improvement >Affects Versions: 3.3.4, 3.4.2 >Reporter: Neha Narkhede >Assignee: Neha Narkhede > Labels: patch > Attachments: ZOOKEEPER-1321_3.4.patch, ZOOKEEPER-1321_trunk.patch, > ZOOKEEPER-1321_trunk.patch, zk-1321-cleanup, zookeeper-1321-trunk-v2.patch > > > The related conversation on the zookeeper user mailing list is here - > http://apache.markmail.org/message/4jjcmooniowwugu2?q=+list:org.apache.hadoop.zookeeper-user > It is useful to be able to monitor the number of disconnect operations on a > client. This is generally indicative of a client going through large number > of GC and hence disconnecting way too often from a zookeeper cluster. > Today, this information is only indirectly exposed as part of the stat > command which requires counting the results. That's alot of work for the > server to do just to get connection count. > For monitoring purposes, it will be useful to have this exposed through JMX > and 4lw srvr. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
ZooKeeper_branch34_solaris - Build # 81 - Failure
See https://builds.apache.org/job/ZooKeeper_branch34_solaris/81/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 139349 lines...] [junit] 2012-01-16 07:59:29,336 [myid:] - INFO [main:JMXEnv@105] - expect:InMemoryDataTree [junit] 2012-01-16 07:59:29,337 [myid:] - INFO [main:JMXEnv@108] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree [junit] 2012-01-16 07:59:29,337 [myid:] - INFO [main:JMXEnv@105] - expect:StandaloneServer_port [junit] 2012-01-16 07:59:29,338 [myid:] - INFO [main:JMXEnv@108] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port-1 [junit] 2012-01-16 07:59:29,338 [myid:] - INFO [main:ClientBase@398] - STOPPING server [junit] 2012-01-16 07:59:29,340 [myid:] - INFO [main:ZooKeeperServer@420] - shutting down [junit] 2012-01-16 07:59:29,340 [myid:] - INFO [main:SessionTrackerImpl@220] - Shutting down [junit] 2012-01-16 07:59:29,341 [myid:] - INFO [main:PrepRequestProcessor@733] - Shutting down [junit] 2012-01-16 07:59:29,342 [myid:] - INFO [main:SyncRequestProcessor@173] - Shutting down [junit] 2012-01-16 07:59:29,342 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@135] - PrepRequestProcessor exited loop! [junit] 2012-01-16 07:59:29,342 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited! [junit] 2012-01-16 07:59:29,343 [myid:] - INFO [main:FinalRequestProcessor@423] - shutdown of request processor complete [junit] 2012-01-16 07:59:29,344 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11221 [junit] 2012-01-16 07:59:29,345 [myid:] - INFO [main:JMXEnv@133] - ensureOnly:[] [junit] 2012-01-16 07:59:29,347 [myid:] - INFO [main:ClientBase@391] - STARTING server [junit] 2012-01-16 07:59:29,348 [myid:] - INFO [main:ZooKeeperServer@168] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/trunk/build/test/tmp/test2948013652293734866.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/trunk/build/test/tmp/test2948013652293734866.junit.dir/version-2 [junit] 2012-01-16 07:59:29,349 [myid:] - INFO [main:NIOServerCnxnFactory@110] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2012-01-16 07:59:29,350 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/trunk/build/test/tmp/test2948013652293734866.junit.dir/version-2/snapshot.b [junit] 2012-01-16 07:59:29,352 [myid:] - INFO [main:FileTxnSnapLog@237] - Snapshotting: b [junit] 2012-01-16 07:59:29,354 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11221 [junit] 2012-01-16 07:59:29,355 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@213] - Accepted socket connection from /127.0.0.1:45689 [junit] 2012-01-16 07:59:29,356 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@820] - Processing stat command from /127.0.0.1:45689 [junit] 2012-01-16 07:59:29,357 [myid:] - INFO [Thread-5:NIOServerCnxn$StatCommand@655] - Stat command output [junit] 2012-01-16 07:59:29,357 [myid:] - INFO [Thread-5:NIOServerCnxn@1000] - Closed socket connection for client /127.0.0.1:45689 (no session established for client) [junit] 2012-01-16 07:59:29,358 [myid:] - INFO [main:JMXEnv@133] - ensureOnly:[InMemoryDataTree, StandaloneServer_port] [junit] 2012-01-16 07:59:29,360 [myid:] - INFO [main:JMXEnv@105] - expect:InMemoryDataTree [junit] 2012-01-16 07:59:29,360 [myid:] - INFO [main:JMXEnv@108] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree [junit] 2012-01-16 07:59:29,361 [myid:] - INFO [main:JMXEnv@105] - expect:StandaloneServer_port [junit] 2012-01-16 07:59:29,362 [myid:] - INFO [main:JMXEnv@108] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port-1 [junit] 2012-01-16 07:59:29,363 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD testQuota [junit] 2012-01-16 07:59:29,363 [myid:] - INFO [main:ClientBase@428] - tearDown starting [junit] 2012-01-16 07:59:29,425 [myid:] - INFO [main:ZooKeeper@679] - Session: 0x134e5872f84 closed [junit] 2012-01-16 07:59:29,425 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@511] - EventThread shut down [junit] 2012-01-16 07:59:29,425 [myid:] - INFO [main:ClientBase@398] - STOPPING server [junit] 2012-01-16 07:59:29,427 [myid:] - INFO [main:ZooKeeperServer@420]
