Success: ZOOKEEPER- PreCommit Build #1401
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1401/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 78.26 MB...] [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +0 tests included. The patch appears to be a documentation patch that doesn't require tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1401//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1401//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1401//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] e95a33f713774d93ebf2f62a27ded2d851f44a0b logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 18 minutes 16 seconds Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-1580 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (ZOOKEEPER-1580) QuorumPeer.setRunning is not used
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317852#comment-16317852 ] Hadoop QA commented on ZOOKEEPER-1580: -- +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1401//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1401//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1401//console This message is automatically generated. > QuorumPeer.setRunning is not used > - > > Key: ZOOKEEPER-1580 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1580 > Project: ZooKeeper > Issue Type: Bug >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira >Priority: Minor > > setRunning is a public method and a search did not indicate that it is used > anywhere, not even in tests. In fact, I believe we should not change > "running" freely and we should only do it when calling shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (ZOOKEEPER-2964) "Conf" command returns dataDir and dataLogDir opposingly
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ZOOKEEPER-2964: - Summary: "Conf" command returns dataDir and dataLogDir opposingly (was: dataDir and dataLogDir are printed opposingly) > "Conf" command returns dataDir and dataLogDir opposingly > > > Key: ZOOKEEPER-2964 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2964 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.3 >Reporter: Qihong Xu >Priority: Minor > Attachments: ZOOKEEPER-2964.patch > > > I foung a bug that "conf" command would return dataDir and dataLogDir > opposingly. > This bug only exists in versions newer than 3.5. I only found dumpConf in > [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L188] > prints these two paths opposingly. Unlike ZOOKEEPER-2960, the actual paths > are not affected and server function is ok. > I made a small patch to fix this bug. Any review is appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper pull request #446: ZOOKEEPER-1580:QuorumPeer.setRunning is not use...
GitHub user maoling opened a pull request: https://github.com/apache/zookeeper/pull/446 ZOOKEEPER-1580:QuorumPeer.setRunning is not used more details in [JIRA:ZOOKEEPER-1580](https://issues.apache.org/jira/browse/ZOOKEEPER-1580) You can merge this pull request into a Git repository by running: $ git pull https://github.com/maoling/zookeeper ZOOKEEPER-1580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/446.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #446 commit b86ae0e6258b1e9319b75530e4bd917517c380c2 Author: maoling Date: 2018-01-09T06:22:04Z ZOOKEEPER-1580:QuorumPeer.setRunning is not used ---
[jira] [Commented] (ZOOKEEPER-1580) QuorumPeer.setRunning is not used
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317821#comment-16317821 ] ASF GitHub Bot commented on ZOOKEEPER-1580: --- GitHub user maoling opened a pull request: https://github.com/apache/zookeeper/pull/446 ZOOKEEPER-1580:QuorumPeer.setRunning is not used more details in [JIRA:ZOOKEEPER-1580](https://issues.apache.org/jira/browse/ZOOKEEPER-1580) You can merge this pull request into a Git repository by running: $ git pull https://github.com/maoling/zookeeper ZOOKEEPER-1580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/446.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #446 commit b86ae0e6258b1e9319b75530e4bd917517c380c2 Author: maoling Date: 2018-01-09T06:22:04Z ZOOKEEPER-1580:QuorumPeer.setRunning is not used > QuorumPeer.setRunning is not used > - > > Key: ZOOKEEPER-1580 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1580 > Project: ZooKeeper > Issue Type: Bug >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira >Priority: Minor > > setRunning is a public method and a search did not indicate that it is used > anywhere, not even in tests. In fact, I believe we should not change > "running" freely and we should only do it when calling shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper pull request #438: ZOOKEEPER-2959: ignore accepted epoch and LEADE...
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/438#discussion_r160286100
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Leader.java ---
@@ -1183,8 +1183,10 @@ public long getEpochToPropose(long sid, long
lastAcceptedEpoch) throws Interrupt
if (lastAcceptedEpoch >= epoch) {
epoch = lastAcceptedEpoch+1;
}
-connectingFollowers.add(sid);
QuorumVerifier verifier = self.getQuorumVerifier();
+if(verifier.getVotingMembers().containsKey(sid)) {
--- End diff --
I'm wondering if this logic is best suited for the `QuorumVerifier`. In
other words, the quorum verifier should be able to determine if a quorum is
present from a set of ids while taking into account which sids represent voting
members.
---
[jira] [Commented] (ZOOKEEPER-2959) ignore accepted epoch and LEADERINFO ack from observers when a newly elected leader computes new epoch
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317361#comment-16317361
]
ASF GitHub Bot commented on ZOOKEEPER-2959:
---
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/438#discussion_r160286100
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Leader.java ---
@@ -1183,8 +1183,10 @@ public long getEpochToPropose(long sid, long
lastAcceptedEpoch) throws Interrupt
if (lastAcceptedEpoch >= epoch) {
epoch = lastAcceptedEpoch+1;
}
-connectingFollowers.add(sid);
QuorumVerifier verifier = self.getQuorumVerifier();
+if(verifier.getVotingMembers().containsKey(sid)) {
--- End diff --
I'm wondering if this logic is best suited for the `QuorumVerifier`. In
other words, the quorum verifier should be able to determine if a quorum is
present from a set of ids while taking into account which sids represent voting
members.
> ignore accepted epoch and LEADERINFO ack from observers when a newly elected
> leader computes new epoch
> --
>
> Key: ZOOKEEPER-2959
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2959
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.4.10, 3.5.3
>Reporter: xiangyq000
>
> Once the ZooKeeper cluster finishes the election for new leader, all learners
> report their accepted epoch to the leader for the computation of new cluster
> epoch.
> org.apache.zookeeper.server.quorum.Leader#getEpochToPropose
> {code:java}
> private final HashSet connectingFollowers = new HashSet();
> public long getEpochToPropose(long sid, long lastAcceptedEpoch) throws
> InterruptedException, IOException {
> synchronized(connectingFollowers) {
> if (!waitingForNewEpoch) {
> return epoch;
> }
> if (lastAcceptedEpoch >= epoch) {
> epoch = lastAcceptedEpoch+1;
> }
> connectingFollowers.add(sid);
> QuorumVerifier verifier = self.getQuorumVerifier();
> if (connectingFollowers.contains(self.getId()) &&
>
> verifier.containsQuorum(connectingFollowers)) {
> waitingForNewEpoch = false;
> self.setAcceptedEpoch(epoch);
> connectingFollowers.notifyAll();
> } else {
> long start = Time.currentElapsedTime();
> long cur = start;
> long end = start + self.getInitLimit()*self.getTickTime();
> while(waitingForNewEpoch && cur < end) {
> connectingFollowers.wait(end - cur);
> cur = Time.currentElapsedTime();
> }
> if (waitingForNewEpoch) {
> throw new InterruptedException("Timeout while waiting for
> epoch from quorum");
> }
> }
> return epoch;
> }
> }
> {code}
> The computation will get an outcome once :
> # The leader has call method "getEpochToPropose"
> # The number of all reporters is greater than half of participants.
> The problem is, an observer server will also send its accepted epoch to the
> leader, while this procedure treat observers as participants.
> Supposed that the cluster consists of 1 leader, 2 followers and 1 observer,
> and now the leader and the observer have reported their accepted epochs while
> neither of the followers has. Thus, the connectingFollowers set consists of
> two elements, resulting in a size of 2, which is greater than half quorum,
> namely, 2. Then QuorumVerifier#containsQuorum will return true, because it
> does not check whether the elements of the parameter are participants.
> The same flaw exists in
> org.apache.zookeeper.server.quorum.Leader#waitForEpochAck
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
ZooKeeper-trunk - Build # 3685 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk/3685/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 139.78 KB...] [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.41 sec, Thread: 4, Class: org.apache.zookeeper.test.ServerCnxnTest [junit] Running org.apache.zookeeper.test.StandaloneTest in thread 4 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.634 sec, Thread: 4, Class: org.apache.zookeeper.test.StandaloneTest [junit] Running org.apache.zookeeper.test.StatTest in thread 4 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.473 sec, Thread: 4, Class: org.apache.zookeeper.test.StatTest [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.561 sec, Thread: 6, Class: org.apache.zookeeper.test.QuorumZxidSyncTest [junit] Running org.apache.zookeeper.test.StaticHostProviderTest in thread 4 [junit] Running org.apache.zookeeper.test.StringUtilTest in thread 6 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.078 sec, Thread: 6, Class: org.apache.zookeeper.test.StringUtilTest [junit] Running org.apache.zookeeper.test.SyncCallTest in thread 6 [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.326 sec, Thread: 4, Class: org.apache.zookeeper.test.StaticHostProviderTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.075 sec, Thread: 6, Class: org.apache.zookeeper.test.SyncCallTest [junit] Running org.apache.zookeeper.test.TruncateTest in thread 4 [junit] Running org.apache.zookeeper.test.WatchEventWhenAutoResetTest in thread 6 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.395 sec, Thread: 4, Class: org.apache.zookeeper.test.TruncateTest [junit] Tests run: 14, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 85.446 sec, Thread: 1, Class: org.apache.zookeeper.test.QuorumTest [junit] Running org.apache.zookeeper.test.WatchedEventTest in thread 4 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec, Thread: 4, Class: org.apache.zookeeper.test.WatchedEventTest [junit] Running org.apache.zookeeper.test.WatcherFuncTest in thread 4 [junit] Running org.apache.zookeeper.test.WatcherTest in thread 1 [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.007 sec, Thread: 4, Class: org.apache.zookeeper.test.WatcherFuncTest [junit] Running org.apache.zookeeper.test.X509AuthTest in thread 4 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.099 sec, Thread: 4, Class: org.apache.zookeeper.test.X509AuthTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.887 sec, Thread: 3, Class: org.apache.zookeeper.test.SessionUpgradeTest [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest in thread 4 [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.612 sec, Thread: 3, Class: org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.591 sec, Thread: 4, Class: org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.995 sec, Thread: 7, Class: org.apache.zookeeper.test.SessionTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.003 sec, Thread: 6, Class: org.apache.zookeeper.test.WatchEventWhenAutoResetTest [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.816 sec, Thread: 1, Class: org.apache.zookeeper.test.WatcherTest [junit] Tests run: 105, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 354.774 sec, Thread: 2, Class: org.apache.zookeeper.test.NettyNettySuiteTest [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 256.142 sec, Thread: 8, Class: org.apache.zookeeper.test.ReconfigTest [junit] Tests run: 105, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 363.95 sec, Thread: 5, Class: org.apache.zookeeper.test.NioNettySuiteTest fail.build.on.test.failure: BUILD FAILED /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/build.xml:1395: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/build.xml:1276: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/build.xml:1280: Tests failed! Total time: 9 minutes 23 seconds Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE [WARNINGS] Skipping publisher since build result is FAILURE Archiving artifacts Setting JDK_1_7_LATEST__HOME=
[jira] [Commented] (ZOOKEEPER-1621) ZooKeeper does not recover from crash when disk was full
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317193#comment-16317193 ] ASF GitHub Bot commented on ZOOKEEPER-1621: --- Github user afine commented on the issue: https://github.com/apache/zookeeper/pull/439 @abhishekrai Looking through the JIRA I found: > This has been a recurring problem for us in production because our app's operating environment occasionally causes a Zookeeper server's disk to become full. After that, the server invariably runs into this problem - perhaps because there's something else that deterministically triggers a log rotation when the previous txn log throws an IOException due to disk full? Do we have evidence that the log roll is being triggered "deterministically"? It would be great to know for sure that we are handling the disk filling up appropriately all the time rather than just a work around for special cases. > ZooKeeper does not recover from crash when disk was full > > > Key: ZOOKEEPER-1621 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1621 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.3 > Environment: Ubuntu 12.04, Amazon EC2 instance >Reporter: David Arthur >Assignee: Michi Mutsuzaki > Fix For: 3.5.4, 3.6.0 > > Attachments: ZOOKEEPER-1621.2.patch, ZOOKEEPER-1621.patch, > zookeeper.log.gz > > > The disk that ZooKeeper was using filled up. During a snapshot write, I got > the following exception > 2013-01-16 03:11:14,098 - ERROR [SyncThread:0:SyncRequestProcessor@151] - > Severe unrecoverable error, exiting > java.io.IOException: No space left on device > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:282) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at > org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:309) > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:306) > at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:484) > at > org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:162) > at > org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101) > Then many subsequent exceptions like: > 2013-01-16 15:02:23,984 - ERROR [main:Util@239] - Last transaction was > partial. > 2013-01-16 15:02:23,985 - ERROR [main:ZooKeeperServerMain@63] - Unexpected > exception, exiting abnormally > java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) > at > org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64) > at > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:558) > at > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:577) > at > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:543) > at > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625) > at > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:529) > at > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.(FileTxnLog.java:504) > at > org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:341) > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:130) > at > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) > at > org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:259) > at > org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:386) > at > org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:138) > at > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112) > at > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86) > at > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:
[GitHub] zookeeper issue #439: ZOOKEEPER-1621: Delete and skip txn log with incomplet...
Github user afine commented on the issue: https://github.com/apache/zookeeper/pull/439 @abhishekrai Looking through the JIRA I found: > This has been a recurring problem for us in production because our app's operating environment occasionally causes a Zookeeper server's disk to become full. After that, the server invariably runs into this problem - perhaps because there's something else that deterministically triggers a log rotation when the previous txn log throws an IOException due to disk full? Do we have evidence that the log roll is being triggered "deterministically"? It would be great to know for sure that we are handling the disk filling up appropriately all the time rather than just a work around for special cases. ---
[jira] [Commented] (ZOOKEEPER-2939) Deal with maxbuffer as it relates to proposals
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317019#comment-16317019
]
ASF GitHub Bot commented on ZOOKEEPER-2939:
---
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/440#discussion_r159987502
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/ProposalStats.java ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import com.codahale.metrics.ExponentiallyDecayingReservoir;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.JmxReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.Reservoir;
+import com.codahale.metrics.Snapshot;
+import org.apache.zookeeper.jmx.CommonNames;
+import org.apache.zookeeper.jmx.MBeanRegistry;
+
+import static com.codahale.metrics.MetricRegistry.name;
+
+/**
+ * Provides real-time metrics on Leader's proposal size.
+ * The class uses a histogram included in Dropwizard metrics library with
ExponentiallyDecayingReservoir.
+ * It provides stats of proposal sizes from the last 5 minutes with
acceptable cpu/memory footprint optimized for streaming data.
+ */
+public class ProposalStats {
+private final Histogram proposalSizes;
+
+ProposalStats() {
+final MetricRegistry metrics = new MetricRegistry();
+Reservoir reservoir = new ExponentiallyDecayingReservoir();
--- End diff --
I won't pretend to know much about exponentially decaying reservoirs. I'm
curious what the behavior is with minimum and maximum values. Are these
guaranteed to always be exact and for what time period?
> Deal with maxbuffer as it relates to proposals
> --
>
> Key: ZOOKEEPER-2939
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2939
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: jute, server
>Reporter: Andor Molnar
>Assignee: Andor Molnar
> Fix For: 3.5.4, 3.6.0
>
>
> Monitor real-time Jute buffer usage as it relates to proposals.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2939) Deal with maxbuffer as it relates to proposals
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317017#comment-16317017 ] ASF GitHub Bot commented on ZOOKEEPER-2939: --- Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r159986909 --- Diff: src/java/main/org/apache/zookeeper/server/quorum/ProposalStats.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.quorum; + +import com.codahale.metrics.ExponentiallyDecayingReservoir; --- End diff -- I don't see much being pulled in from dropwizard, only codahale. Is codahale.metrics available independently? It doesn't look like it but I only checked briefly. > Deal with maxbuffer as it relates to proposals > -- > > Key: ZOOKEEPER-2939 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2939 > Project: ZooKeeper > Issue Type: Sub-task > Components: jute, server >Reporter: Andor Molnar >Assignee: Andor Molnar > Fix For: 3.5.4, 3.6.0 > > > Monitor real-time Jute buffer usage as it relates to proposals. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2939) Deal with maxbuffer as it relates to proposals
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317022#comment-16317022
]
ASF GitHub Bot commented on ZOOKEEPER-2939:
---
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/440#discussion_r160249680
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/ProposalStats.java ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import com.codahale.metrics.ExponentiallyDecayingReservoir;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.JmxReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.Reservoir;
+import com.codahale.metrics.Snapshot;
+import org.apache.zookeeper.jmx.CommonNames;
+import org.apache.zookeeper.jmx.MBeanRegistry;
+
+import static com.codahale.metrics.MetricRegistry.name;
+
+/**
+ * Provides real-time metrics on Leader's proposal size.
+ * The class uses a histogram included in Dropwizard metrics library with
ExponentiallyDecayingReservoir.
+ * It provides stats of proposal sizes from the last 5 minutes with
acceptable cpu/memory footprint optimized for streaming data.
+ */
+public class ProposalStats {
+private final Histogram proposalSizes;
+
+ProposalStats() {
+final MetricRegistry metrics = new MetricRegistry();
+Reservoir reservoir = new ExponentiallyDecayingReservoir();
--- End diff --
Perhaps a user configurable SlidingTimeWindowReservoir may be more
appropriate?
> Deal with maxbuffer as it relates to proposals
> --
>
> Key: ZOOKEEPER-2939
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2939
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: jute, server
>Reporter: Andor Molnar
>Assignee: Andor Molnar
> Fix For: 3.5.4, 3.6.0
>
>
> Monitor real-time Jute buffer usage as it relates to proposals.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2939) Deal with maxbuffer as it relates to proposals
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317020#comment-16317020 ] ASF GitHub Bot commented on ZOOKEEPER-2939: --- Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r160236050 --- Diff: src/java/test/org/apache/zookeeper/server/quorum/LeaderBeanTest.java --- @@ -0,0 +1,119 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.quorum; + +import org.apache.jute.OutputArchive; --- End diff -- nit: there are a couple unused imports here > Deal with maxbuffer as it relates to proposals > -- > > Key: ZOOKEEPER-2939 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2939 > Project: ZooKeeper > Issue Type: Sub-task > Components: jute, server >Reporter: Andor Molnar >Assignee: Andor Molnar > Fix For: 3.5.4, 3.6.0 > > > Monitor real-time Jute buffer usage as it relates to proposals. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2939) Deal with maxbuffer as it relates to proposals
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317018#comment-16317018 ] ASF GitHub Bot commented on ZOOKEEPER-2939: --- Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r159985176 --- Diff: build.xml --- @@ -198,7 +198,7 @@ xmlns:cs="antlib:com.puppycrawl.tools.checkstyle.ant"> - + --- End diff -- what is the reason for the downgrade? > Deal with maxbuffer as it relates to proposals > -- > > Key: ZOOKEEPER-2939 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2939 > Project: ZooKeeper > Issue Type: Sub-task > Components: jute, server >Reporter: Andor Molnar >Assignee: Andor Molnar > Fix For: 3.5.4, 3.6.0 > > > Monitor real-time Jute buffer usage as it relates to proposals. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2939) Deal with maxbuffer as it relates to proposals
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317023#comment-16317023 ] ASF GitHub Bot commented on ZOOKEEPER-2939: --- Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r160233203 --- Diff: ivy.xml --- @@ -133,6 +133,12 @@ + + + Deal with maxbuffer as it relates to proposals > -- > > Key: ZOOKEEPER-2939 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2939 > Project: ZooKeeper > Issue Type: Sub-task > Components: jute, server >Reporter: Andor Molnar >Assignee: Andor Molnar > Fix For: 3.5.4, 3.6.0 > > > Monitor real-time Jute buffer usage as it relates to proposals. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2939) Deal with maxbuffer as it relates to proposals
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317021#comment-16317021
]
ASF GitHub Bot commented on ZOOKEEPER-2939:
---
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/440#discussion_r160236436
--- Diff:
src/java/test/org/apache/zookeeper/server/quorum/LeaderBeanTest.java ---
@@ -0,0 +1,119 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import org.apache.jute.OutputArchive;
+import org.apache.jute.Record;
+import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.ZKDatabase;
+import org.apache.zookeeper.server.persistence.FileTxnSnapLog;
+import org.apache.zookeeper.server.quorum.flexible.QuorumVerifier;
+import org.apache.zookeeper.server.util.SerializeUtils;
+import org.apache.zookeeper.test.ClientBase;
+import org.apache.zookeeper.txn.TxnHeader;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.io.File;
+import java.io.IOException;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+
+public class LeaderBeanTest {
+private Leader leader;
+private LeaderBean leaderBean;
+private FileTxnSnapLog fileTxnSnapLog;
+private LeaderZooKeeperServer zks;
+private QuorumPeer qp;
+
+@Before
+public void setUp() throws IOException {
+qp = new QuorumPeer();
+QuorumVerifier quorumVerifierMock = mock(QuorumVerifier.class);
+qp.setQuorumVerifier(quorumVerifierMock, false);
+File tmpDir = ClientBase.createEmptyTestDir();
+fileTxnSnapLog = new FileTxnSnapLog(new File(tmpDir, "data"),
+new File(tmpDir, "data_txnlog"));
+ZKDatabase zkDb = new ZKDatabase(fileTxnSnapLog);
+
+zks = new LeaderZooKeeperServer(fileTxnSnapLog, qp, zkDb);
+leader = new Leader(qp, zks);
+leaderBean = new LeaderBean(leader, zks);
+}
+
+@After
+public void tearDown() throws IOException {
+fileTxnSnapLog.close();
+}
+
+@Test
+public void testGetName() {
+assertEquals("Leader", leaderBean.getName());
+}
+
+@Test
+public void testGetCurrentZxid() {
+// Arrange
+zks.setZxid(1);
+
+// Assert
+assertEquals("0x1", leaderBean.getCurrentZxid());
+}
+
+@Test
+public void testGetElectionTimeTaken() {
+// Arrange
+qp.setElectionTimeTaken(1);
+
+// Assert
+assertEquals(1, leaderBean.getElectionTimeTaken());
+}
+
+private Request createMockRequest() throws IOException {
--- End diff --
is this ever used?
> Deal with maxbuffer as it relates to proposals
> --
>
> Key: ZOOKEEPER-2939
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2939
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: jute, server
>Reporter: Andor Molnar
>Assignee: Andor Molnar
> Fix For: 3.5.4, 3.6.0
>
>
> Monitor real-time Jute buffer usage as it relates to proposals.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] zookeeper pull request #440: ZOOKEEPER-2939 Deal with maxbuffer as it relate...
Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r160233203 --- Diff: ivy.xml --- @@ -133,6 +133,12 @@ + + +
[GitHub] zookeeper pull request #440: ZOOKEEPER-2939 Deal with maxbuffer as it relate...
Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r159986909 --- Diff: src/java/main/org/apache/zookeeper/server/quorum/ProposalStats.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.quorum; + +import com.codahale.metrics.ExponentiallyDecayingReservoir; --- End diff -- I don't see much being pulled in from dropwizard, only codahale. Is codahale.metrics available independently? It doesn't look like it but I only checked briefly. ---
[GitHub] zookeeper pull request #440: ZOOKEEPER-2939 Deal with maxbuffer as it relate...
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/440#discussion_r160236436
--- Diff:
src/java/test/org/apache/zookeeper/server/quorum/LeaderBeanTest.java ---
@@ -0,0 +1,119 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import org.apache.jute.OutputArchive;
+import org.apache.jute.Record;
+import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.ZKDatabase;
+import org.apache.zookeeper.server.persistence.FileTxnSnapLog;
+import org.apache.zookeeper.server.quorum.flexible.QuorumVerifier;
+import org.apache.zookeeper.server.util.SerializeUtils;
+import org.apache.zookeeper.test.ClientBase;
+import org.apache.zookeeper.txn.TxnHeader;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.io.File;
+import java.io.IOException;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+
+public class LeaderBeanTest {
+private Leader leader;
+private LeaderBean leaderBean;
+private FileTxnSnapLog fileTxnSnapLog;
+private LeaderZooKeeperServer zks;
+private QuorumPeer qp;
+
+@Before
+public void setUp() throws IOException {
+qp = new QuorumPeer();
+QuorumVerifier quorumVerifierMock = mock(QuorumVerifier.class);
+qp.setQuorumVerifier(quorumVerifierMock, false);
+File tmpDir = ClientBase.createEmptyTestDir();
+fileTxnSnapLog = new FileTxnSnapLog(new File(tmpDir, "data"),
+new File(tmpDir, "data_txnlog"));
+ZKDatabase zkDb = new ZKDatabase(fileTxnSnapLog);
+
+zks = new LeaderZooKeeperServer(fileTxnSnapLog, qp, zkDb);
+leader = new Leader(qp, zks);
+leaderBean = new LeaderBean(leader, zks);
+}
+
+@After
+public void tearDown() throws IOException {
+fileTxnSnapLog.close();
+}
+
+@Test
+public void testGetName() {
+assertEquals("Leader", leaderBean.getName());
+}
+
+@Test
+public void testGetCurrentZxid() {
+// Arrange
+zks.setZxid(1);
+
+// Assert
+assertEquals("0x1", leaderBean.getCurrentZxid());
+}
+
+@Test
+public void testGetElectionTimeTaken() {
+// Arrange
+qp.setElectionTimeTaken(1);
+
+// Assert
+assertEquals(1, leaderBean.getElectionTimeTaken());
+}
+
+private Request createMockRequest() throws IOException {
--- End diff --
is this ever used?
---
[GitHub] zookeeper pull request #440: ZOOKEEPER-2939 Deal with maxbuffer as it relate...
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/440#discussion_r159987502
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/ProposalStats.java ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import com.codahale.metrics.ExponentiallyDecayingReservoir;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.JmxReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.Reservoir;
+import com.codahale.metrics.Snapshot;
+import org.apache.zookeeper.jmx.CommonNames;
+import org.apache.zookeeper.jmx.MBeanRegistry;
+
+import static com.codahale.metrics.MetricRegistry.name;
+
+/**
+ * Provides real-time metrics on Leader's proposal size.
+ * The class uses a histogram included in Dropwizard metrics library with
ExponentiallyDecayingReservoir.
+ * It provides stats of proposal sizes from the last 5 minutes with
acceptable cpu/memory footprint optimized for streaming data.
+ */
+public class ProposalStats {
+private final Histogram proposalSizes;
+
+ProposalStats() {
+final MetricRegistry metrics = new MetricRegistry();
+Reservoir reservoir = new ExponentiallyDecayingReservoir();
--- End diff --
I won't pretend to know much about exponentially decaying reservoirs. I'm
curious what the behavior is with minimum and maximum values. Are these
guaranteed to always be exact and for what time period?
---
[GitHub] zookeeper pull request #440: ZOOKEEPER-2939 Deal with maxbuffer as it relate...
Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r159985176 --- Diff: build.xml --- @@ -198,7 +198,7 @@ xmlns:cs="antlib:com.puppycrawl.tools.checkstyle.ant"> - + --- End diff -- what is the reason for the downgrade? ---
[GitHub] zookeeper pull request #440: ZOOKEEPER-2939 Deal with maxbuffer as it relate...
Github user afine commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/440#discussion_r160236050 --- Diff: src/java/test/org/apache/zookeeper/server/quorum/LeaderBeanTest.java --- @@ -0,0 +1,119 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.quorum; + +import org.apache.jute.OutputArchive; --- End diff -- nit: there are a couple unused imports here ---
[GitHub] zookeeper pull request #440: ZOOKEEPER-2939 Deal with maxbuffer as it relate...
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/440#discussion_r160249680
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/ProposalStats.java ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import com.codahale.metrics.ExponentiallyDecayingReservoir;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.JmxReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.Reservoir;
+import com.codahale.metrics.Snapshot;
+import org.apache.zookeeper.jmx.CommonNames;
+import org.apache.zookeeper.jmx.MBeanRegistry;
+
+import static com.codahale.metrics.MetricRegistry.name;
+
+/**
+ * Provides real-time metrics on Leader's proposal size.
+ * The class uses a histogram included in Dropwizard metrics library with
ExponentiallyDecayingReservoir.
+ * It provides stats of proposal sizes from the last 5 minutes with
acceptable cpu/memory footprint optimized for streaming data.
+ */
+public class ProposalStats {
+private final Histogram proposalSizes;
+
+ProposalStats() {
+final MetricRegistry metrics = new MetricRegistry();
+Reservoir reservoir = new ExponentiallyDecayingReservoir();
--- End diff --
Perhaps a user configurable SlidingTimeWindowReservoir may be more
appropriate?
---
ZooKeeper-trunk-openjdk7 - Build # 1761 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-openjdk7/1761/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 114.08 KB...] [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.121 sec, Thread: 3, Class: org.apache.zookeeper.test.SaslClientTest [junit] Running org.apache.zookeeper.test.SaslSuperUserTest in thread 7 [junit] Running org.apache.zookeeper.test.ServerCnxnTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.713 sec, Thread: 7, Class: org.apache.zookeeper.test.SaslSuperUserTest [junit] Running org.apache.zookeeper.test.SessionInvalidationTest in thread 7 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.842 sec, Thread: 7, Class: org.apache.zookeeper.test.SessionInvalidationTest [junit] Running org.apache.zookeeper.test.SessionTest in thread 7 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.129 sec, Thread: 3, Class: org.apache.zookeeper.test.ServerCnxnTest [junit] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.499 sec, Thread: 2, Class: org.apache.zookeeper.test.ReadOnlyModeTest [junit] Running org.apache.zookeeper.test.SessionTrackerCheckTest in thread 3 [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.156 sec, Thread: 3, Class: org.apache.zookeeper.test.SessionTrackerCheckTest [junit] Running org.apache.zookeeper.test.SessionUpgradeTest in thread 3 [junit] Running org.apache.zookeeper.test.StandaloneTest in thread 2 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.705 sec, Thread: 2, Class: org.apache.zookeeper.test.StandaloneTest [junit] Running org.apache.zookeeper.test.StatTest in thread 2 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.614 sec, Thread: 2, Class: org.apache.zookeeper.test.StatTest [junit] Running org.apache.zookeeper.test.StaticHostProviderTest in thread 2 [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.412 sec, Thread: 4, Class: org.apache.zookeeper.test.QuorumZxidSyncTest [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.246 sec, Thread: 2, Class: org.apache.zookeeper.test.StaticHostProviderTest [junit] Running org.apache.zookeeper.test.StringUtilTest in thread 2 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.102 sec, Thread: 2, Class: org.apache.zookeeper.test.StringUtilTest [junit] Running org.apache.zookeeper.test.TruncateTest in thread 2 [junit] Running org.apache.zookeeper.test.SyncCallTest in thread 4 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.537 sec, Thread: 4, Class: org.apache.zookeeper.test.SyncCallTest [junit] Running org.apache.zookeeper.test.WatchEventWhenAutoResetTest in thread 4 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.346 sec, Thread: 2, Class: org.apache.zookeeper.test.TruncateTest [junit] Running org.apache.zookeeper.test.WatchedEventTest in thread 2 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec, Thread: 2, Class: org.apache.zookeeper.test.WatchedEventTest [junit] Running org.apache.zookeeper.test.WatcherFuncTest in thread 2 [junit] Tests run: 14, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 92.975 sec, Thread: 5, Class: org.apache.zookeeper.test.QuorumTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.069 sec, Thread: 2, Class: org.apache.zookeeper.test.WatcherFuncTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.226 sec, Thread: 3, Class: org.apache.zookeeper.test.SessionUpgradeTest [junit] Running org.apache.zookeeper.test.WatcherTest in thread 5 [junit] Running org.apache.zookeeper.test.X509AuthTest in thread 2 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.166 sec, Thread: 2, Class: org.apache.zookeeper.test.X509AuthTest [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest in thread 3 [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest in thread 2 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.379 sec, Thread: 2, Class: org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.16 sec, Thread: 7, Class: org.apache.zookeeper.test.SessionTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.271 sec, Thread: 3, Class: org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.716 sec, Thread: 4, Class: org.apache.zookeeper.test.WatchEventWhenAutoResetTest [ju
[jira] [Commented] (ZOOKEEPER-1621) ZooKeeper does not recover from crash when disk was full
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316984#comment-16316984
]
ASF GitHub Bot commented on ZOOKEEPER-1621:
---
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160245528
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
--- End diff --
logFiles & numTransactions variables are not being used in the rest of this
test.
> ZooKeeper does not recover from crash when disk was full
>
>
> Key: ZOOKEEPER-1621
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1621
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.3
> Environment: Ubuntu 12.04, Amazon EC2 instance
>Reporter: David Arthur
>Assignee: Michi Mutsuzaki
> Fix For: 3.5.4, 3.6.0
>
> Attachments: ZOOKEEPER-1621.2.patch, ZOOKEEPER-1621.patch,
> zookeeper.log.gz
>
>
> The disk that ZooKeeper was using filled up. During a snapshot write, I got
> the following exception
> 2013-01-16 03:11:14,098 - ERROR [SyncThread:0:SyncRequestProcessor@151] -
> Severe unrecoverable error, exiting
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:282)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:309)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:306)
> at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:484)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:162)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
> Then many subsequent exceptions like:
> 2013-01-16 15:02:23,984 - ERROR [main:Util@239] - Last transaction was
> partial.
> 2013-01-16 15:02:23,985 - ERROR [main:ZooKeeperServerMain@63] - Unexpected
> exception, exiting abnormally
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at
> org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:558)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:577)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:543)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(
[GitHub] zookeeper pull request #439: ZOOKEEPER-1621: Delete and skip txn log with in...
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160245528
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
--- End diff --
logFiles & numTransactions variables are not being used in the rest of this
test.
---
[GitHub] zookeeper pull request #439: ZOOKEEPER-1621: Delete and skip txn log with in...
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160243418
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
+while (fileItr.next()) {
+}
+
+// Verify that the truncated log file does not exist anymore.
+Assert.assertFalse("Verify truncated log file has been deleted",
+ lastLogFile.exists());
+}
+
+/**
+ * Verifies that FileTxnIterator throws CorruptedStreamException if the
+ * magic number is corrupted.
+ */
+@Test(expected = StreamCorruptedException.class)
+public void testCorruptMagicNumber() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+
+// Corrupt the magic number.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+RandomAccessFile file = new RandomAccessFile(lastLogFile, "rw");
+file.seek(0);
+file.writeByte(123);
+file.close();
+
+// This should throw CorruptedStreamException.
+while (fileItr.next()) {
+}
+}
+
+/**
+ * Starts a standalone server and create znodes.
+ */
+public void loadDatabase(File dataDir, int numEntries) throws
Exception {
--- End diff --
This method could be private.
---
[jira] [Commented] (ZOOKEEPER-1621) ZooKeeper does not recover from crash when disk was full
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316969#comment-16316969
]
ASF GitHub Bot commented on ZOOKEEPER-1621:
---
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160243418
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
+while (fileItr.next()) {
+}
+
+// Verify that the truncated log file does not exist anymore.
+Assert.assertFalse("Verify truncated log file has been deleted",
+ lastLogFile.exists());
+}
+
+/**
+ * Verifies that FileTxnIterator throws CorruptedStreamException if the
+ * magic number is corrupted.
+ */
+@Test(expected = StreamCorruptedException.class)
+public void testCorruptMagicNumber() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+
+// Corrupt the magic number.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+RandomAccessFile file = new RandomAccessFile(lastLogFile, "rw");
+file.seek(0);
+file.writeByte(123);
+file.close();
+
+// This should throw CorruptedStreamException.
+while (fileItr.next()) {
+}
+}
+
+/**
+ * Starts a standalone server and create znodes.
+ */
+public void loadDatabase(File dataDir, int numEntries) throws
Exception {
--- End diff --
This method could be private.
> ZooKeeper does not recover from crash when disk was full
>
>
> Key: ZOOKEEPER-1621
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1621
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.3
> Environment: Ubuntu 12.04, Amazon EC2 instance
>Reporter: David Arthur
>Assignee: Michi Mutsuzaki
> Fix For: 3.5.4, 3.6.0
>
> Attachments: ZOOKEEPER-1621.2.patch, ZOOKEEPER-1621.patch,
> zookeeper.log.gz
>
>
> The disk that ZooKeeper was using filled up. During a snapshot write, I got
> the following exception
> 2013-01-16 03:11:14,098 - ERROR [SyncThread:0:SyncRequestProcessor@151] -
> Severe unrecoverable error, exiting
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:282)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStr
[jira] [Commented] (ZOOKEEPER-1621) ZooKeeper does not recover from crash when disk was full
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316972#comment-16316972
]
ASF GitHub Bot commented on ZOOKEEPER-1621:
---
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160243609
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
+while (fileItr.next()) {
+}
+
+// Verify that the truncated log file does not exist anymore.
+Assert.assertFalse("Verify truncated log file has been deleted",
+ lastLogFile.exists());
+}
+
+/**
+ * Verifies that FileTxnIterator throws CorruptedStreamException if the
+ * magic number is corrupted.
+ */
+@Test(expected = StreamCorruptedException.class)
+public void testCorruptMagicNumber() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+
+// Corrupt the magic number.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+RandomAccessFile file = new RandomAccessFile(lastLogFile, "rw");
+file.seek(0);
+file.writeByte(123);
+file.close();
+
+// This should throw CorruptedStreamException.
+while (fileItr.next()) {
+}
+}
+
+/**
+ * Starts a standalone server and create znodes.
+ */
+public void loadDatabase(File dataDir, int numEntries) throws
Exception {
+final String hostPort = HOST + PortAssignment.unique();
+// setup a single server cluster
+ZooKeeperServer zks = new ZooKeeperServer(dataDir, dataDir, 3000);
+SyncRequestProcessor.setSnapCount(100);
+final int PORT = Integer.parseInt(hostPort.split(":")[1]);
+ServerCnxnFactory f = ServerCnxnFactory.createFactory(PORT, -1);
+f.startup(zks);
+Assert.assertTrue("waiting for server being up ",
+ClientBase.waitForServerUp(hostPort,CONNECTION_TIMEOUT));
+ZooKeeper zk = ClientBase.createZKClient(hostPort);
--- End diff --
Down to this line the logic is already implemented in the base class.
Please consider re-using it in your tests.
> ZooKeeper does not recover from crash when disk was full
>
>
> Key: ZOOKEEPER-1621
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1621
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.3
> Environment
[jira] [Commented] (ZOOKEEPER-1621) ZooKeeper does not recover from crash when disk was full
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316973#comment-16316973
]
ASF GitHub Bot commented on ZOOKEEPER-1621:
---
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160244289
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -38,10 +40,16 @@
import org.slf4j.LoggerFactory;
import java.io.File;
+import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.StreamCorruptedException;
+import java.nio.channels.FileChannel;
+import java.util.List;
public class LoadFromLogTest extends ClientBase {
private static final int NUM_MESSAGES = 300;
+private static final String HOST = "127.0.0.1:";
--- End diff --
It's already in the base class.
> ZooKeeper does not recover from crash when disk was full
>
>
> Key: ZOOKEEPER-1621
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1621
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.3
> Environment: Ubuntu 12.04, Amazon EC2 instance
>Reporter: David Arthur
>Assignee: Michi Mutsuzaki
> Fix For: 3.5.4, 3.6.0
>
> Attachments: ZOOKEEPER-1621.2.patch, ZOOKEEPER-1621.patch,
> zookeeper.log.gz
>
>
> The disk that ZooKeeper was using filled up. During a snapshot write, I got
> the following exception
> 2013-01-16 03:11:14,098 - ERROR [SyncThread:0:SyncRequestProcessor@151] -
> Severe unrecoverable error, exiting
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:282)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:309)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:306)
> at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:484)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:162)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
> Then many subsequent exceptions like:
> 2013-01-16 15:02:23,984 - ERROR [main:Util@239] - Last transaction was
> partial.
> 2013-01-16 15:02:23,985 - ERROR [main:ZooKeeperServerMain@63] - Unexpected
> exception, exiting abnormally
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at
> org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:558)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:577)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:543)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:529)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.(FileTxnLog.java:504)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:341)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:130)
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> at
> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:259)
> at
> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:386)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:138)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
[jira] [Commented] (ZOOKEEPER-1621) ZooKeeper does not recover from crash when disk was full
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316970#comment-16316970
]
ASF GitHub Bot commented on ZOOKEEPER-1621:
---
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160244035
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
+while (fileItr.next()) {
+}
+
+// Verify that the truncated log file does not exist anymore.
+Assert.assertFalse("Verify truncated log file has been deleted",
+ lastLogFile.exists());
+}
+
+/**
+ * Verifies that FileTxnIterator throws CorruptedStreamException if the
+ * magic number is corrupted.
+ */
+@Test(expected = StreamCorruptedException.class)
+public void testCorruptMagicNumber() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+
+// Corrupt the magic number.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+RandomAccessFile file = new RandomAccessFile(lastLogFile, "rw");
+file.seek(0);
+file.writeByte(123);
+file.close();
+
+// This should throw CorruptedStreamException.
+while (fileItr.next()) {
+}
+}
+
+/**
+ * Starts a standalone server and create znodes.
+ */
+public void loadDatabase(File dataDir, int numEntries) throws
Exception {
+final String hostPort = HOST + PortAssignment.unique();
+// setup a single server cluster
+ZooKeeperServer zks = new ZooKeeperServer(dataDir, dataDir, 3000);
+SyncRequestProcessor.setSnapCount(100);
+final int PORT = Integer.parseInt(hostPort.split(":")[1]);
+ServerCnxnFactory f = ServerCnxnFactory.createFactory(PORT, -1);
+f.startup(zks);
+Assert.assertTrue("waiting for server being up ",
+ClientBase.waitForServerUp(hostPort,CONNECTION_TIMEOUT));
+ZooKeeper zk = ClientBase.createZKClient(hostPort);
+
+// Generate some transactions that will get logged.
+try {
+for (int i = 0; i < numEntries; i++) {
+zk.create("/load-database-" + i, new byte[0],
+ Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+}
+} finally {
+zk.close();
+}
+f.shutdown();
--- End diff --
Starting the server is already implemented in base class' setUp() method
and shutdown
[jira] [Commented] (ZOOKEEPER-1621) ZooKeeper does not recover from crash when disk was full
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316971#comment-16316971
]
ASF GitHub Bot commented on ZOOKEEPER-1621:
---
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160244385
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
--- End diff --
Startup / shutdown logic should be in setUp() / tearDown() methods.
> ZooKeeper does not recover from crash when disk was full
>
>
> Key: ZOOKEEPER-1621
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1621
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.3
> Environment: Ubuntu 12.04, Amazon EC2 instance
>Reporter: David Arthur
>Assignee: Michi Mutsuzaki
> Fix For: 3.5.4, 3.6.0
>
> Attachments: ZOOKEEPER-1621.2.patch, ZOOKEEPER-1621.patch,
> zookeeper.log.gz
>
>
> The disk that ZooKeeper was using filled up. During a snapshot write, I got
> the following exception
> 2013-01-16 03:11:14,098 - ERROR [SyncThread:0:SyncRequestProcessor@151] -
> Severe unrecoverable error, exiting
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:282)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:309)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:306)
> at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:484)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:162)
> at
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
> Then many subsequent exceptions like:
> 2013-01-16 15:02:23,984 - ERROR [main:Util@239] - Last transaction was
> partial.
> 2013-01-16 15:02:23,985 - ERROR [main:ZooKeeperServerMain@63] - Unexpected
> exception, exiting abnormally
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at
> org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:558)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:577)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:543)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:625)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:529)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.(FileTxnLog.java:504)
> at
> org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:341)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:130)
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> at
> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:259)
> at
> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:386)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:138)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:112)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
> at
> org.apach
[GitHub] zookeeper pull request #439: ZOOKEEPER-1621: Delete and skip txn log with in...
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160244385
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
--- End diff --
Startup / shutdown logic should be in setUp() / tearDown() methods.
---
[GitHub] zookeeper pull request #439: ZOOKEEPER-1621: Delete and skip txn log with in...
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160244035
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
+while (fileItr.next()) {
+}
+
+// Verify that the truncated log file does not exist anymore.
+Assert.assertFalse("Verify truncated log file has been deleted",
+ lastLogFile.exists());
+}
+
+/**
+ * Verifies that FileTxnIterator throws CorruptedStreamException if the
+ * magic number is corrupted.
+ */
+@Test(expected = StreamCorruptedException.class)
+public void testCorruptMagicNumber() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+
+// Corrupt the magic number.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+RandomAccessFile file = new RandomAccessFile(lastLogFile, "rw");
+file.seek(0);
+file.writeByte(123);
+file.close();
+
+// This should throw CorruptedStreamException.
+while (fileItr.next()) {
+}
+}
+
+/**
+ * Starts a standalone server and create znodes.
+ */
+public void loadDatabase(File dataDir, int numEntries) throws
Exception {
+final String hostPort = HOST + PortAssignment.unique();
+// setup a single server cluster
+ZooKeeperServer zks = new ZooKeeperServer(dataDir, dataDir, 3000);
+SyncRequestProcessor.setSnapCount(100);
+final int PORT = Integer.parseInt(hostPort.split(":")[1]);
+ServerCnxnFactory f = ServerCnxnFactory.createFactory(PORT, -1);
+f.startup(zks);
+Assert.assertTrue("waiting for server being up ",
+ClientBase.waitForServerUp(hostPort,CONNECTION_TIMEOUT));
+ZooKeeper zk = ClientBase.createZKClient(hostPort);
+
+// Generate some transactions that will get logged.
+try {
+for (int i = 0; i < numEntries; i++) {
+zk.create("/load-database-" + i, new byte[0],
+ Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+}
+} finally {
+zk.close();
+}
+f.shutdown();
--- End diff --
Starting the server is already implemented in base class' setUp() method
and shutdown is already in tearDown(). It's safer to let junit's infrastructure
to deal with startup / shutdown methods which makes sure the server(s) gets
shutdown even if some error occurs.
---
[GitHub] zookeeper pull request #439: ZOOKEEPER-1621: Delete and skip txn log with in...
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160243609
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -307,4 +315,104 @@ public void testReloadSnapshotWithMissingParent()
throws Exception {
startServer();
}
+
+/**
+ * Verify that FileTxnIterator doesn't throw an EOFException when the
+ * transaction log header is incomplete.
+ */
+@Test
+public void testIncompleteHeader() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+int numTransactions = 0;
+while (fileItr.next()) {
+numTransactions++;
+}
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+Assert.assertTrue("Verify the number of transactions",
+ numTransactions >= NUM_MESSAGES);
+
+// Truncate the last log file.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+FileChannel channel = new
FileOutputStream(lastLogFile).getChannel();
+channel.truncate(0);
+channel.close();
+
+// This shouldn't thow Exception.
+fileItr = new FileTxnLog.FileTxnIterator(logDir, 0);
+logFiles = fileItr.getStoredFiles();
+numTransactions = 0;
+while (fileItr.next()) {
+}
+
+// Verify that the truncated log file does not exist anymore.
+Assert.assertFalse("Verify truncated log file has been deleted",
+ lastLogFile.exists());
+}
+
+/**
+ * Verifies that FileTxnIterator throws CorruptedStreamException if the
+ * magic number is corrupted.
+ */
+@Test(expected = StreamCorruptedException.class)
+public void testCorruptMagicNumber() throws Exception {
+ClientBase.setupTestEnv();
+File dataDir = ClientBase.createTmpDir();
+loadDatabase(dataDir, NUM_MESSAGES);
+
+File logDir = new File(dataDir, FileTxnSnapLog.version +
+FileTxnSnapLog.VERSION);
+FileTxnLog.FileTxnIterator fileItr = new
FileTxnLog.FileTxnIterator(logDir, 0);
+List logFiles = fileItr.getStoredFiles();
+Assert.assertTrue("Verify the number of log files",
+ logFiles.size() > 0);
+
+// Corrupt the magic number.
+File lastLogFile = logFiles.get(logFiles.size() - 1);
+RandomAccessFile file = new RandomAccessFile(lastLogFile, "rw");
+file.seek(0);
+file.writeByte(123);
+file.close();
+
+// This should throw CorruptedStreamException.
+while (fileItr.next()) {
+}
+}
+
+/**
+ * Starts a standalone server and create znodes.
+ */
+public void loadDatabase(File dataDir, int numEntries) throws
Exception {
+final String hostPort = HOST + PortAssignment.unique();
+// setup a single server cluster
+ZooKeeperServer zks = new ZooKeeperServer(dataDir, dataDir, 3000);
+SyncRequestProcessor.setSnapCount(100);
+final int PORT = Integer.parseInt(hostPort.split(":")[1]);
+ServerCnxnFactory f = ServerCnxnFactory.createFactory(PORT, -1);
+f.startup(zks);
+Assert.assertTrue("waiting for server being up ",
+ClientBase.waitForServerUp(hostPort,CONNECTION_TIMEOUT));
+ZooKeeper zk = ClientBase.createZKClient(hostPort);
--- End diff --
Down to this line the logic is already implemented in the base class.
Please consider re-using it in your tests.
---
[GitHub] zookeeper pull request #439: ZOOKEEPER-1621: Delete and skip txn log with in...
Github user anmolnar commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/439#discussion_r160244289
--- Diff: src/java/test/org/apache/zookeeper/test/LoadFromLogTest.java ---
@@ -38,10 +40,16 @@
import org.slf4j.LoggerFactory;
import java.io.File;
+import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.io.StreamCorruptedException;
+import java.nio.channels.FileChannel;
+import java.util.List;
public class LoadFromLogTest extends ClientBase {
private static final int NUM_MESSAGES = 300;
+private static final String HOST = "127.0.0.1:";
--- End diff --
It's already in the base class.
---
[jira] [Commented] (ZOOKEEPER-2959) ignore accepted epoch and LEADERINFO ack from observers when a newly elected leader computes new epoch
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316906#comment-16316906
]
ASF GitHub Bot commented on ZOOKEEPER-2959:
---
Github user anmolnar commented on the issue:
https://github.com/apache/zookeeper/pull/438
Hi @xyq000
Thanks for the contribution. I think fixing this issues makes sense, would
you please add at least one unit test to reproduce the problem?
> ignore accepted epoch and LEADERINFO ack from observers when a newly elected
> leader computes new epoch
> --
>
> Key: ZOOKEEPER-2959
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2959
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.4.10, 3.5.3
>Reporter: xiangyq000
>
> Once the ZooKeeper cluster finishes the election for new leader, all learners
> report their accepted epoch to the leader for the computation of new cluster
> epoch.
> org.apache.zookeeper.server.quorum.Leader#getEpochToPropose
> {code:java}
> private final HashSet connectingFollowers = new HashSet();
> public long getEpochToPropose(long sid, long lastAcceptedEpoch) throws
> InterruptedException, IOException {
> synchronized(connectingFollowers) {
> if (!waitingForNewEpoch) {
> return epoch;
> }
> if (lastAcceptedEpoch >= epoch) {
> epoch = lastAcceptedEpoch+1;
> }
> connectingFollowers.add(sid);
> QuorumVerifier verifier = self.getQuorumVerifier();
> if (connectingFollowers.contains(self.getId()) &&
>
> verifier.containsQuorum(connectingFollowers)) {
> waitingForNewEpoch = false;
> self.setAcceptedEpoch(epoch);
> connectingFollowers.notifyAll();
> } else {
> long start = Time.currentElapsedTime();
> long cur = start;
> long end = start + self.getInitLimit()*self.getTickTime();
> while(waitingForNewEpoch && cur < end) {
> connectingFollowers.wait(end - cur);
> cur = Time.currentElapsedTime();
> }
> if (waitingForNewEpoch) {
> throw new InterruptedException("Timeout while waiting for
> epoch from quorum");
> }
> }
> return epoch;
> }
> }
> {code}
> The computation will get an outcome once :
> # The leader has call method "getEpochToPropose"
> # The number of all reporters is greater than half of participants.
> The problem is, an observer server will also send its accepted epoch to the
> leader, while this procedure treat observers as participants.
> Supposed that the cluster consists of 1 leader, 2 followers and 1 observer,
> and now the leader and the observer have reported their accepted epochs while
> neither of the followers has. Thus, the connectingFollowers set consists of
> two elements, resulting in a size of 2, which is greater than half quorum,
> namely, 2. Then QuorumVerifier#containsQuorum will return true, because it
> does not check whether the elements of the parameter are participants.
> The same flaw exists in
> org.apache.zookeeper.server.quorum.Leader#waitForEpochAck
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] zookeeper issue #438: ZOOKEEPER-2959: ignore accepted epoch and LEADERINFO a...
Github user anmolnar commented on the issue: https://github.com/apache/zookeeper/pull/438 Hi @xyq000 Thanks for the contribution. I think fixing this issues makes sense, would you please add at least one unit test to reproduce the problem? ---
ZooKeeper-trunk-windows-cmake - Build # 2607 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-windows-cmake/2607/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7.81 KB...] [ivy:retrieve] found org.slf4j#slf4j-api;1.7.25 in maven2 [ivy:retrieve] found org.slf4j#slf4j-log4j12;1.7.25 in maven2 [ivy:retrieve] found commons-cli#commons-cli;1.2 in maven2 [ivy:retrieve] found log4j#log4j;1.2.17 in maven2 [ivy:retrieve] found org.apache.yetus#audience-annotations;0.5.0 in maven2 [ivy:retrieve] found io.netty#netty;3.10.6.Final in maven2 [ivy:retrieve] :: resolution report :: resolve 687ms :: artifacts dl 23ms - | |modules|| artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| - | default | 16 | 0 | 0 | 0 || 16 | 0 | - [ivy:retrieve] :: retrieving :: org.apache.zookeeper#zookeeper [ivy:retrieve] confs: [default] [ivy:retrieve] 16 artifacts copied, 0 already retrieved (4344kB/1028ms) ivy-retrieve-javacc: [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\javacc\lib [ivy:retrieve] :: resolving dependencies :: org.apache.zookeeper#zookeeper;3.6.0-SNAPSHOT [ivy:retrieve] confs: [javacc] [ivy:retrieve] found net.java.dev.javacc#javacc;5.0 in maven2 [ivy:retrieve] :: resolution report :: resolve 55ms :: artifacts dl 1ms - | |modules|| artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| - | javacc | 1 | 0 | 0 | 0 || 1 | 0 | - [ivy:retrieve] :: retrieving :: org.apache.zookeeper#zookeeper [ivy:retrieve] confs: [javacc] [ivy:retrieve] 1 artifacts copied, 0 already retrieved (291kB/17ms) generate_jute_parser: [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\jute_compiler\org\apache\jute\compiler\generated [ivy:artifactproperty] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:artifactproperty] :: loading settings :: file = f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\ivysettings.xml [move] Moving 1 file to f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\javacc\lib [javacc] Java Compiler Compiler Version 5.0 (Parser Generator) [javacc] (type "javacc" with no arguments for help) [javacc] Reading from file f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\src\java\main\org\apache\jute\compiler\generated\rcc.jj . . . [javacc] File "TokenMgrError.java" does not exist. Will create one. [javacc] File "ParseException.java" does not exist. Will create one. [javacc] File "Token.java" does not exist. Will create one. [javacc] File "SimpleCharStream.java" does not exist. Will create one. [javacc] Parser generated successfully. jute: BUILD FAILED f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build.xml:317: Unable to find a javac compiler; com.sun.tools.javac.Main is not on the classpath. Perhaps JAVA_HOME does not point to the JDK. It is currently set to "C:\Program Files\Java\jre-9" Total time: 4 seconds Build step 'Invoke Ant' marked build as failure Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
ZooKeeper_branch34_openjdk7 - Build # 1778 - Failure
See https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/1778/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 38.87 KB...] [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.778 sec [junit] Running org.apache.zookeeper.test.SaslAuthDesignatedClientTest [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.666 sec [junit] Running org.apache.zookeeper.test.SaslAuthDesignatedServerTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.588 sec [junit] Running org.apache.zookeeper.test.SaslAuthFailDesignatedClientTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.828 sec [junit] Running org.apache.zookeeper.test.SaslAuthFailNotifyTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.535 sec [junit] Running org.apache.zookeeper.test.SaslAuthFailTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.616 sec [junit] Running org.apache.zookeeper.test.SaslAuthMissingClientConfigTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.597 sec [junit] Running org.apache.zookeeper.test.SaslClientTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.08 sec [junit] Running org.apache.zookeeper.test.SessionInvalidationTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.687 sec [junit] Running org.apache.zookeeper.test.SessionTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.046 sec [junit] Running org.apache.zookeeper.test.StandaloneTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.869 sec [junit] Running org.apache.zookeeper.test.StatTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.905 sec [junit] Running org.apache.zookeeper.test.StaticHostProviderTest [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.249 sec [junit] Running org.apache.zookeeper.test.SyncCallTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.628 sec [junit] Running org.apache.zookeeper.test.TruncateTest [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.363 sec [junit] Running org.apache.zookeeper.test.UpgradeTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.316 sec [junit] Running org.apache.zookeeper.test.WatchedEventTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec [junit] Running org.apache.zookeeper.test.WatcherFuncTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.416 sec [junit] Running org.apache.zookeeper.test.WatcherTest [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.844 sec [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.812 sec [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.705 sec fail.build.on.test.failure: BUILD FAILED /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build.xml:1382: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build.xml:1385: Tests failed! Total time: 38 minutes 25 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Recording test results Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes Error Message: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit. Stack Trace: junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
ZooKeeper_branch35_jdk8 - Build # 803 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_jdk8/803/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 59.09 KB...] [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.667 sec, Thread: 7, Class: org.apache.zookeeper.test.SaslAuthMissingClientConfigTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec, Thread: 3, Class: org.apache.zookeeper.test.SaslClientTest [junit] Running org.apache.zookeeper.test.SaslSuperUserTest in thread 3 [junit] Running org.apache.zookeeper.test.ServerCnxnTest in thread 7 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.796 sec, Thread: 3, Class: org.apache.zookeeper.test.SaslSuperUserTest [junit] Running org.apache.zookeeper.test.SessionInvalidationTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.884 sec, Thread: 3, Class: org.apache.zookeeper.test.SessionInvalidationTest [junit] Running org.apache.zookeeper.test.SessionTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.445 sec, Thread: 7, Class: org.apache.zookeeper.test.ServerCnxnTest [junit] Running org.apache.zookeeper.test.SessionTrackerCheckTest in thread 7 [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.09 sec, Thread: 7, Class: org.apache.zookeeper.test.SessionTrackerCheckTest [junit] Running org.apache.zookeeper.test.SessionUpgradeTest in thread 7 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.877 sec, Thread: 8, Class: org.apache.zookeeper.test.RestoreCommittedLogTest [junit] Running org.apache.zookeeper.test.StandaloneTest in thread 8 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.607 sec, Thread: 8, Class: org.apache.zookeeper.test.StandaloneTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.727 sec, Thread: 1, Class: org.apache.zookeeper.test.RecoveryTest [junit] Running org.apache.zookeeper.test.StatTest in thread 8 [junit] Running org.apache.zookeeper.test.StaticHostProviderTest in thread 1 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.741 sec, Thread: 8, Class: org.apache.zookeeper.test.StatTest [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.104 sec, Thread: 1, Class: org.apache.zookeeper.test.StaticHostProviderTest [junit] Running org.apache.zookeeper.test.StringUtilTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.092 sec, Thread: 8, Class: org.apache.zookeeper.test.StringUtilTest [junit] Running org.apache.zookeeper.test.SyncCallTest in thread 1 [junit] Running org.apache.zookeeper.test.TruncateTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.729 sec, Thread: 1, Class: org.apache.zookeeper.test.SyncCallTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.69 sec, Thread: 7, Class: org.apache.zookeeper.test.SessionUpgradeTest [junit] Running org.apache.zookeeper.test.WatchEventWhenAutoResetTest in thread 1 [junit] Running org.apache.zookeeper.test.WatchedEventTest in thread 7 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.197 sec, Thread: 7, Class: org.apache.zookeeper.test.WatchedEventTest [junit] Running org.apache.zookeeper.test.WatcherFuncTest in thread 7 [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.477 sec, Thread: 7, Class: org.apache.zookeeper.test.WatcherFuncTest [junit] Running org.apache.zookeeper.test.WatcherTest in thread 7 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.66 sec, Thread: 8, Class: org.apache.zookeeper.test.TruncateTest [junit] Running org.apache.zookeeper.test.X509AuthTest in thread 8 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.12 sec, Thread: 8, Class: org.apache.zookeeper.test.X509AuthTest [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest in thread 8 [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.906 sec, Thread: 3, Class: org.apache.zookeeper.test.SessionTest [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.068 sec, Thread: 3, Class: org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.422 sec, Thread: 8, Class: org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.679 sec, Thread: 1, Class: org.apache.zookeeper.test.WatchEventWhenAutoRe
ZooKeeper-trunk-jdk8 - Build # 1332 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/1332/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 68.75 KB...] [junit] Running org.apache.zookeeper.test.SessionInvalidationTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.276 sec, Thread: 8, Class: org.apache.zookeeper.test.SessionInvalidationTest [junit] Running org.apache.zookeeper.test.SessionTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.618 sec, Thread: 3, Class: org.apache.zookeeper.test.ServerCnxnTest [junit] Running org.apache.zookeeper.test.SessionTrackerCheckTest in thread 3 [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124 sec, Thread: 3, Class: org.apache.zookeeper.test.SessionTrackerCheckTest [junit] Running org.apache.zookeeper.test.SessionUpgradeTest in thread 3 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.063 sec, Thread: 3, Class: org.apache.zookeeper.test.SessionUpgradeTest [junit] Running org.apache.zookeeper.test.StandaloneTest in thread 3 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.725 sec, Thread: 3, Class: org.apache.zookeeper.test.StandaloneTest [junit] Running org.apache.zookeeper.test.StatTest in thread 3 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.915 sec, Thread: 3, Class: org.apache.zookeeper.test.StatTest [junit] Running org.apache.zookeeper.test.StaticHostProviderTest in thread 3 [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.514 sec, Thread: 8, Class: org.apache.zookeeper.test.SessionTest [junit] Running org.apache.zookeeper.test.StringUtilTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.097 sec, Thread: 8, Class: org.apache.zookeeper.test.StringUtilTest [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.258 sec, Thread: 3, Class: org.apache.zookeeper.test.StaticHostProviderTest [junit] Running org.apache.zookeeper.test.SyncCallTest in thread 8 [junit] Running org.apache.zookeeper.test.TruncateTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.185 sec, Thread: 8, Class: org.apache.zookeeper.test.SyncCallTest [junit] Running org.apache.zookeeper.test.WatchEventWhenAutoResetTest in thread 8 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.246 sec, Thread: 3, Class: org.apache.zookeeper.test.TruncateTest [junit] Running org.apache.zookeeper.test.WatchedEventTest in thread 3 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 sec, Thread: 3, Class: org.apache.zookeeper.test.WatchedEventTest [junit] Running org.apache.zookeeper.test.WatcherFuncTest in thread 3 [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.248 sec, Thread: 3, Class: org.apache.zookeeper.test.WatcherFuncTest [junit] Running org.apache.zookeeper.test.WatcherTest in thread 3 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.338 sec, Thread: 8, Class: org.apache.zookeeper.test.WatchEventWhenAutoResetTest [junit] Running org.apache.zookeeper.test.X509AuthTest in thread 8 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.12 sec, Thread: 8, Class: org.apache.zookeeper.test.X509AuthTest [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.929 sec, Thread: 8, Class: org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.168 sec, Thread: 8, Class: org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 105, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 535.06 sec, Thread: 4, Class: org.apache.zookeeper.test.NettyNettySuiteTest [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.926 sec, Thread: 3, Class: org.apache.zookeeper.test.WatcherTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 706.766 sec, Thread: 6, Class: org.apache.zookeeper.test.DisconnectedWatcherTest [junit] Tests run: 105, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 557.746 sec, Thread: 2, Class: org.apache.zookeeper.test.NioNettySuiteTest [junit] Running org.apache.zookeeper.server.quorum.StandaloneDisabledTest in thread 5 [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec, Thread: 5, Class: org.apache.zookeeper.server.quorum.StandaloneDisabledTest [junit] Test org.apache.zookeeper.ser
Success: ZOOKEEPER- PreCommit Build #1400
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1400/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 39.62 MB...] [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +0 tests included. The patch appears to be a documentation patch that doesn't require tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1400//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1400//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1400//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] 1c0af1661ccc7e863a12117031643b4fa045ef1a logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 36 minutes 44 seconds Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-2955 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (ZOOKEEPER-2955) Enable Clover code coverage report
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316053#comment-16316053 ] Hadoop QA commented on ZOOKEEPER-2955: -- +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1400//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1400//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1400//console This message is automatically generated. > Enable Clover code coverage report > -- > > Key: ZOOKEEPER-2955 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2955 > Project: ZooKeeper > Issue Type: Test > Components: tests >Affects Versions: 3.5.3, 3.4.11 >Reporter: Mark Fenes >Assignee: Mark Fenes > > We have limited code coverage support in ZK. Clover for Java was running in > the past but was turned off. > Enable Clover code coverage report to make us more confident on the quality, > stability and compatibility of future ZK releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2955) Enable Clover code coverage report
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316028#comment-16316028 ] Hadoop QA commented on ZOOKEEPER-2955: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1399//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1399//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1399//console This message is automatically generated. > Enable Clover code coverage report > -- > > Key: ZOOKEEPER-2955 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2955 > Project: ZooKeeper > Issue Type: Test > Components: tests >Affects Versions: 3.5.3, 3.4.11 >Reporter: Mark Fenes >Assignee: Mark Fenes > > We have limited code coverage support in ZK. Clover for Java was running in > the past but was turned off. > Enable Clover code coverage report to make us more confident on the quality, > stability and compatibility of future ZK releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Failed: ZOOKEEPER- PreCommit Build #1399
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1399/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 71.40 MB...] [exec] +0 tests included. The patch appears to be a documentation patch that doesn't require tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1399//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1399//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1399//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] d94ce7b55aea40e64869987d73096e0710bbb5aa logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1732: exec returned: 1 Total time: 21 minutes 43 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-2955 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.StandaloneDisabledTest.startSingleServerTest Error Message: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. Stack Trace: junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. at java.lang.Thread.run(Thread.java:745)
Success: ZOOKEEPER- PreCommit Build #1398
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1398/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 77.46 MB...] [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +0 tests included. The patch appears to be a documentation patch that doesn't require tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1398//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1398//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1398//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] e8f047da997e9c527a778667bd227bfa20b61f28 logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 19 minutes 22 seconds Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-2955 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (ZOOKEEPER-2955) Enable Clover code coverage report
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316024#comment-16316024 ] Hadoop QA commented on ZOOKEEPER-2955: -- +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1398//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1398//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1398//console This message is automatically generated. > Enable Clover code coverage report > -- > > Key: ZOOKEEPER-2955 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2955 > Project: ZooKeeper > Issue Type: Test > Components: tests >Affects Versions: 3.5.3, 3.4.11 >Reporter: Mark Fenes >Assignee: Mark Fenes > > We have limited code coverage support in ZK. Clover for Java was running in > the past but was turned off. > Enable Clover code coverage report to make us more confident on the quality, > stability and compatibility of future ZK releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2960) The dataDir and dataLogDir are used opposingly
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316020#comment-16316020
]
Dan Milon commented on ZOOKEEPER-2960:
--
Thank you guys :)
> The dataDir and dataLogDir are used opposingly
> --
>
> Key: ZOOKEEPER-2960
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2960
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.11
> Environment: Not relevant.
>Reporter: Dan Milon
>Assignee: Andor Molnar
>Priority: Critical
> Fix For: 3.4.12
>
>
> _emphasized text_After upgrading from zookeeper 3.4.5, to 3.4.11, without
> editing {{zoo.cfg}}, the new version of the server tries to use the
> {{dataDir}} as the {{dataLogDir}}, and the {{dataLogDir}} as the {{dataDir}}.
> Or at least some parts of the server.
> Configuration file has:
> {noformat}
> $ grep -i data /etc/zookeeper/zoo.cfg
> dataLogDir=/var/lib/zookeeper/datalog
> dataDir=/var/lib/zookeeper/data
> {noformat}
> But runtime configuration has:
> {noformat}
> $ echo conf | nc localhost 2181 | grep -i data
> dataDir=/var/lib/zookeeper/datalog/version-2
> dataLogDir=/var/lib/zookeeper/data/version-2
> {noformat}
> Also, I got this in the debug logs, so clearly some parts of the server
> confuse things.
> {noformat}
> [PurgeTask:FileTxnSnapLog@79] - Opening datadir:/var/lib/zookeeper/datalog
> snapDir:/var/lib/zookeeper/data
> [main:FileTxnSnapLog@79] - Opening datadir:/var/lib/zookeeper/data
> snapDir:/var/lib/zookeeper/datalog
> {noformat}
> I tried to look in the code for wrong uses of the directories. I only found
> [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L227]
> is passing the arguments to {{FileTxnSnapLog}} in the wrong order, but the
> code comment says that this is legacy only for tests, so I assume it isn't
> the cause for my case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2955) Enable Clover code coverage report
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16315993#comment-16315993 ] ASF GitHub Bot commented on ZOOKEEPER-2955: --- GitHub user mfenes opened a pull request: https://github.com/apache/zookeeper/pull/445 ZOOKEEPER-2955: Enable Clover code coverage report ZOOKEEPER-2955: Enable Clover code coverage report This PR configures OpenClover to generate Java code coverage reports. To enable OpenClover run the following ant targets with -Drun.clover=true: ant -Drun.clover=true jar ant -Drun.clover=true test and then to generate the code coverage reports run: ant -Drun.clover=true generate-clover-reports The reports will be generated under the build/test/clover/reports directory in HTML and XML formats. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mfenes/zookeeper ZOOKEEPER-2955_3.4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/445.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #445 commit 7bda91c4dbaaabdb3df29730bc7af3855b065453 Author: Mark Fenes Date: 2018-01-05T13:21:19Z ZOOKEEPER-2955: Enable Clover code coverage report > Enable Clover code coverage report > -- > > Key: ZOOKEEPER-2955 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2955 > Project: ZooKeeper > Issue Type: Test > Components: tests >Affects Versions: 3.5.3, 3.4.11 >Reporter: Mark Fenes >Assignee: Mark Fenes > > We have limited code coverage support in ZK. Clover for Java was running in > the past but was turned off. > Enable Clover code coverage report to make us more confident on the quality, > stability and compatibility of future ZK releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper pull request #445: ZOOKEEPER-2955: Enable Clover code coverage rep...
GitHub user mfenes opened a pull request: https://github.com/apache/zookeeper/pull/445 ZOOKEEPER-2955: Enable Clover code coverage report ZOOKEEPER-2955: Enable Clover code coverage report This PR configures OpenClover to generate Java code coverage reports. To enable OpenClover run the following ant targets with -Drun.clover=true: ant -Drun.clover=true jar ant -Drun.clover=true test and then to generate the code coverage reports run: ant -Drun.clover=true generate-clover-reports The reports will be generated under the build/test/clover/reports directory in HTML and XML formats. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mfenes/zookeeper ZOOKEEPER-2955_3.4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/445.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #445 commit 7bda91c4dbaaabdb3df29730bc7af3855b065453 Author: Mark Fenes Date: 2018-01-05T13:21:19Z ZOOKEEPER-2955: Enable Clover code coverage report ---
[jira] [Commented] (ZOOKEEPER-2955) Enable Clover code coverage report
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16315990#comment-16315990 ] ASF GitHub Bot commented on ZOOKEEPER-2955: --- GitHub user mfenes opened a pull request: https://github.com/apache/zookeeper/pull/444 ZOOKEEPER-2955: Enable Clover code coverage report ZOOKEEPER-2955: Enable Clover code coverage report This PR configures OpenClover to generate Java code coverage reports. To enable OpenClover run the following ant targets with -Drun.clover=true: ant -Drun.clover=true jar ant -Drun.clover=true test and then to generate the code coverage reports run: ant -Drun.clover=true generate-clover-reports The reports will be generated under the build/test/clover/reports directory in HTML and XML formats. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mfenes/zookeeper ZOOKEEPER-2955_3.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/444.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #444 commit e12114ad5a6fa47b201852a82175a143439eb5db Author: Mark Fenes Date: 2018-01-05T13:21:19Z ZOOKEEPER-2955: Enable Clover code coverage report > Enable Clover code coverage report > -- > > Key: ZOOKEEPER-2955 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2955 > Project: ZooKeeper > Issue Type: Test > Components: tests >Affects Versions: 3.5.3, 3.4.11 >Reporter: Mark Fenes >Assignee: Mark Fenes > > We have limited code coverage support in ZK. Clover for Java was running in > the past but was turned off. > Enable Clover code coverage report to make us more confident on the quality, > stability and compatibility of future ZK releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper pull request #444: ZOOKEEPER-2955: Enable Clover code coverage rep...
GitHub user mfenes opened a pull request: https://github.com/apache/zookeeper/pull/444 ZOOKEEPER-2955: Enable Clover code coverage report ZOOKEEPER-2955: Enable Clover code coverage report This PR configures OpenClover to generate Java code coverage reports. To enable OpenClover run the following ant targets with -Drun.clover=true: ant -Drun.clover=true jar ant -Drun.clover=true test and then to generate the code coverage reports run: ant -Drun.clover=true generate-clover-reports The reports will be generated under the build/test/clover/reports directory in HTML and XML formats. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mfenes/zookeeper ZOOKEEPER-2955_3.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/444.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #444 commit e12114ad5a6fa47b201852a82175a143439eb5db Author: Mark Fenes Date: 2018-01-05T13:21:19Z ZOOKEEPER-2955: Enable Clover code coverage report ---
[jira] [Commented] (ZOOKEEPER-2955) Enable Clover code coverage report
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16315983#comment-16315983 ] ASF GitHub Bot commented on ZOOKEEPER-2955: --- GitHub user mfenes opened a pull request: https://github.com/apache/zookeeper/pull/443 ZOOKEEPER-2955: Enable Clover code coverage report ZOOKEEPER-2955: Enable Clover code coverage report This PR configures OpenClover to generate Java code coverage reports. To enable OpenClover run the following ant targets with -Drun.clover=true: ant -Drun.clover=true jar ant -Drun.clover=true test and then to generate the code coverage reports run: ant -Drun.clover=true generate-clover-reports The reports will be generated under the build/test/clover/reports directory in HTML and XML formats. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mfenes/zookeeper ZOOKEEPER-2955 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/443.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #443 commit f59dcf8ade02ffbe693b56b480d60aff821900de Author: Mark Fenes Date: 2018-01-05T13:21:19Z ZOOKEEPER-2955: Enable Clover code coverage report > Enable Clover code coverage report > -- > > Key: ZOOKEEPER-2955 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2955 > Project: ZooKeeper > Issue Type: Test > Components: tests >Affects Versions: 3.5.3, 3.4.11 >Reporter: Mark Fenes >Assignee: Mark Fenes > > We have limited code coverage support in ZK. Clover for Java was running in > the past but was turned off. > Enable Clover code coverage report to make us more confident on the quality, > stability and compatibility of future ZK releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper pull request #443: ZOOKEEPER-2955: Enable Clover code coverage rep...
GitHub user mfenes opened a pull request: https://github.com/apache/zookeeper/pull/443 ZOOKEEPER-2955: Enable Clover code coverage report ZOOKEEPER-2955: Enable Clover code coverage report This PR configures OpenClover to generate Java code coverage reports. To enable OpenClover run the following ant targets with -Drun.clover=true: ant -Drun.clover=true jar ant -Drun.clover=true test and then to generate the code coverage reports run: ant -Drun.clover=true generate-clover-reports The reports will be generated under the build/test/clover/reports directory in HTML and XML formats. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mfenes/zookeeper ZOOKEEPER-2955 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/443.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #443 commit f59dcf8ade02ffbe693b56b480d60aff821900de Author: Mark Fenes Date: 2018-01-05T13:21:19Z ZOOKEEPER-2955: Enable Clover code coverage report ---
Re: Bug in Zookeeper 3.4.11 release
Thanks Camille. I closed the pull request. Andor On Mon, Jan 8, 2018 at 2:32 AM, Camille Fournier wrote: > Yeah, this is bad. I reviewed and merged the PR to get this process > started. > > On Fri, Jan 5, 2018 at 5:25 PM, Abraham Fine wrote: > > > I left a comment on the JIRA and I'm not sure what the conventional way > to > > proceed would be either. > > > > I think we should work to get a release out as quickly as possible with > > this fix. In addition, I think we may want to add a check on startup that > > forces a failure if snapshots/logs are where they shouldn't be. That way > we > > prevent people that are running 3.4.11 from experiencing data loss. > > > > Abe > > > > On Wed, Jan 3, 2018, at 03:07, Andor Molnar wrote: > > > Hi all, > > > > > > Looks like we've a pretty serious bug in 3.4.11: > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2960 > > > > > > In a nutshell: ZK uses dataDir and dataLogDir parameters in the wrong > > > order, meaning it tries to access logs in the snapshot folder and vica > > > versa. > > > > > > Not sure what's the process in such case, but I believe we've to > > deprecate > > > 3.4.11 and release a maintenance/hotfix/etc release asap. > > > > > > Please advise. > > > > > > In the meantime I'm creating a patch to address the issue on > branch-3.4. > > > (other branches are not affected) > > > > > > Thanks, > > > Andor > > >
[jira] [Resolved] (ZOOKEEPER-2960) The dataDir and dataLogDir are used opposingly
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andor Molnar resolved ZOOKEEPER-2960.
-
Resolution: Fixed
Fix Version/s: 3.4.12
> The dataDir and dataLogDir are used opposingly
> --
>
> Key: ZOOKEEPER-2960
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2960
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.11
> Environment: Not relevant.
>Reporter: Dan Milon
>Assignee: Andor Molnar
>Priority: Critical
> Fix For: 3.4.12
>
>
> _emphasized text_After upgrading from zookeeper 3.4.5, to 3.4.11, without
> editing {{zoo.cfg}}, the new version of the server tries to use the
> {{dataDir}} as the {{dataLogDir}}, and the {{dataLogDir}} as the {{dataDir}}.
> Or at least some parts of the server.
> Configuration file has:
> {noformat}
> $ grep -i data /etc/zookeeper/zoo.cfg
> dataLogDir=/var/lib/zookeeper/datalog
> dataDir=/var/lib/zookeeper/data
> {noformat}
> But runtime configuration has:
> {noformat}
> $ echo conf | nc localhost 2181 | grep -i data
> dataDir=/var/lib/zookeeper/datalog/version-2
> dataLogDir=/var/lib/zookeeper/data/version-2
> {noformat}
> Also, I got this in the debug logs, so clearly some parts of the server
> confuse things.
> {noformat}
> [PurgeTask:FileTxnSnapLog@79] - Opening datadir:/var/lib/zookeeper/datalog
> snapDir:/var/lib/zookeeper/data
> [main:FileTxnSnapLog@79] - Opening datadir:/var/lib/zookeeper/data
> snapDir:/var/lib/zookeeper/datalog
> {noformat}
> I tried to look in the code for wrong uses of the directories. I only found
> [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L227]
> is passing the arguments to {{FileTxnSnapLog}} in the wrong order, but the
> code comment says that this is legacy only for tests, so I assume it isn't
> the cause for my case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2960) The dataDir and dataLogDir are used opposingly
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16315953#comment-16315953
]
ASF GitHub Bot commented on ZOOKEEPER-2960:
---
Github user anmolnar closed the pull request at:
https://github.com/apache/zookeeper/pull/441
> The dataDir and dataLogDir are used opposingly
> --
>
> Key: ZOOKEEPER-2960
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2960
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.11
> Environment: Not relevant.
>Reporter: Dan Milon
>Assignee: Andor Molnar
>Priority: Critical
>
> _emphasized text_After upgrading from zookeeper 3.4.5, to 3.4.11, without
> editing {{zoo.cfg}}, the new version of the server tries to use the
> {{dataDir}} as the {{dataLogDir}}, and the {{dataLogDir}} as the {{dataDir}}.
> Or at least some parts of the server.
> Configuration file has:
> {noformat}
> $ grep -i data /etc/zookeeper/zoo.cfg
> dataLogDir=/var/lib/zookeeper/datalog
> dataDir=/var/lib/zookeeper/data
> {noformat}
> But runtime configuration has:
> {noformat}
> $ echo conf | nc localhost 2181 | grep -i data
> dataDir=/var/lib/zookeeper/datalog/version-2
> dataLogDir=/var/lib/zookeeper/data/version-2
> {noformat}
> Also, I got this in the debug logs, so clearly some parts of the server
> confuse things.
> {noformat}
> [PurgeTask:FileTxnSnapLog@79] - Opening datadir:/var/lib/zookeeper/datalog
> snapDir:/var/lib/zookeeper/data
> [main:FileTxnSnapLog@79] - Opening datadir:/var/lib/zookeeper/data
> snapDir:/var/lib/zookeeper/datalog
> {noformat}
> I tried to look in the code for wrong uses of the directories. I only found
> [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L227]
> is passing the arguments to {{FileTxnSnapLog}} in the wrong order, but the
> code comment says that this is legacy only for tests, so I assume it isn't
> the cause for my case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] zookeeper pull request #441: ZOOKEEPER-2960. The dataDir and dataLogDir are ...
Github user anmolnar closed the pull request at: https://github.com/apache/zookeeper/pull/441 ---
ZooKeeper-trunk-openjdk7 - Build # 1760 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-openjdk7/1760/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 60.60 KB...] [junit] Running org.apache.zookeeper.test.SaslClientTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.088 sec, Thread: 3, Class: org.apache.zookeeper.test.SaslClientTest [junit] Running org.apache.zookeeper.test.SaslSuperUserTest in thread 7 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.585 sec, Thread: 8, Class: org.apache.zookeeper.test.SaslAuthMissingClientConfigTest [junit] Running org.apache.zookeeper.test.ServerCnxnTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.665 sec, Thread: 7, Class: org.apache.zookeeper.test.SaslSuperUserTest [junit] Running org.apache.zookeeper.test.SessionInvalidationTest in thread 8 [junit] Running org.apache.zookeeper.test.SessionTest in thread 7 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.636 sec, Thread: 8, Class: org.apache.zookeeper.test.SessionInvalidationTest [junit] Running org.apache.zookeeper.test.SessionTrackerCheckTest in thread 8 [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124 sec, Thread: 8, Class: org.apache.zookeeper.test.SessionTrackerCheckTest [junit] Running org.apache.zookeeper.test.SessionUpgradeTest in thread 8 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.503 sec, Thread: 3, Class: org.apache.zookeeper.test.ServerCnxnTest [junit] Running org.apache.zookeeper.test.StandaloneTest in thread 3 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.516 sec, Thread: 3, Class: org.apache.zookeeper.test.StandaloneTest [junit] Running org.apache.zookeeper.test.StatTest in thread 3 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.77 sec, Thread: 3, Class: org.apache.zookeeper.test.StatTest [junit] Running org.apache.zookeeper.test.StaticHostProviderTest in thread 3 [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.158 sec, Thread: 3, Class: org.apache.zookeeper.test.StaticHostProviderTest [junit] Running org.apache.zookeeper.test.StringUtilTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.088 sec, Thread: 3, Class: org.apache.zookeeper.test.StringUtilTest [junit] Running org.apache.zookeeper.test.SyncCallTest in thread 3 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.717 sec, Thread: 3, Class: org.apache.zookeeper.test.SyncCallTest [junit] Running org.apache.zookeeper.test.TruncateTest in thread 3 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.032 sec, Thread: 3, Class: org.apache.zookeeper.test.TruncateTest [junit] Running org.apache.zookeeper.test.WatchEventWhenAutoResetTest in thread 3 [junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.001 sec, Thread: 2, Class: org.apache.zookeeper.test.QuorumZxidSyncTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.067 sec, Thread: 8, Class: org.apache.zookeeper.test.SessionUpgradeTest [junit] Running org.apache.zookeeper.test.WatchedEventTest in thread 2 [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.091 sec, Thread: 2, Class: org.apache.zookeeper.test.WatchedEventTest [junit] Running org.apache.zookeeper.test.WatcherFuncTest in thread 2 [junit] Running org.apache.zookeeper.test.WatcherTest in thread 8 [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.107 sec, Thread: 2, Class: org.apache.zookeeper.test.WatcherFuncTest [junit] Tests run: 14, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 82.759 sec, Thread: 6, Class: org.apache.zookeeper.test.QuorumTest [junit] Running org.apache.zookeeper.test.X509AuthTest in thread 2 [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec, Thread: 2, Class: org.apache.zookeeper.test.X509AuthTest [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest in thread 6 [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest in thread 2 [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.785 sec, Thread: 2, Class: org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.411 sec, Thread: 7, Class: org.apache.zookeeper.test.SessionTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.719 sec, Thread: 6, Class: org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.67
