ZooKeeper-trunk-windows-cmake - Build # 2946 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-windows-cmake/2946/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 9.79 KB...] [ivy:retrieve] confs: [javacc] [ivy:retrieve] found net.java.dev.javacc#javacc;5.0 in maven2 [ivy:retrieve] :: resolution report :: resolve 26ms :: artifacts dl 1ms - | |modules|| artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| - | javacc | 1 | 0 | 0 | 0 || 1 | 0 | - [ivy:retrieve] :: retrieving :: org.apache.zookeeper#zookeeper [ivy:retrieve] confs: [javacc] [ivy:retrieve] 1 artifacts copied, 0 already retrieved (291kB/8ms) generate_jute_parser: [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\jute_compiler\org\apache\jute\compiler\generated [ivy:artifactproperty] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:artifactproperty] :: loading settings :: file = f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\ivysettings.xml [move] Moving 1 file to f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\javacc\lib [javacc] Java Compiler Compiler Version 5.0 (Parser Generator) [javacc] (type "javacc" with no arguments for help) [javacc] Reading from file f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-jute\src\main\java\org\apache\jute\compiler\generated\rcc.jj . . . [javacc] File "TokenMgrError.java" does not exist. Will create one. [javacc] File "ParseException.java" does not exist. Will create one. [javacc] File "Token.java" does not exist. Will create one. [javacc] File "SimpleCharStream.java" does not exist. Will create one. [javacc] Parser generated successfully. jute: [javac] Compiling 39 source files to f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\classes [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.8 [javac] 1 warning compile_jute_uptodate: compile_jute: [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\src\java\generated [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-client\zookeeper-client-c\generated [java] ../../../zookeeper-jute/src/main/resources/zookeeper.jute Parsed Successfully [java] ../../../zookeeper-jute/src/main/resources/zookeeper.jute Parsed Successfully [touch] Creating f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\src\java\generated\.generated BUILD SUCCESSFUL Total time: 5 seconds [ZooKeeper-trunk-windows-cmake] $ cmd /c call F:\jenkins\jenkins-slave\temp\jenkins5492249363117310886.bat f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>cd src/c The system cannot find the path specified. f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>cmake . CMake Error: The source directory "F:/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-windows-cmake" does not appear to contain CMakeLists.txt. Specify --help for usage, or press the help button on the CMake GUI. f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>cmake --build . Error: could not load cache f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>exit 1 Build step 'Execute Windows batch command' marked build as failure Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
[GitHub] zookeeper issue #639: ZOOKEEPER-2122: add openssl to zookeeper c client to s...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/639 thanks for update, @roodkcab . will take a look at this. quick question, looks like you had prove it working [here](https://github.com/apache/zookeeper/pull/625#issuecomment-421330923). Do you think you can add a unit test? It might not be easy, but would be good if we can have a unit test around this feature for c client. ---
[jira] [Resolved] (ZOOKEEPER-3146) Limit the maximum client connections per IP in NettyServerCnxnFactory
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Han resolved ZOOKEEPER-3146. Resolution: Fixed Issue resolved by pull request 623 [https://github.com/apache/zookeeper/pull/623] > Limit the maximum client connections per IP in NettyServerCnxnFactory > -- > > Key: ZOOKEEPER-3146 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3146 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Assignee: Fangmin Lv >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > There is maximum connections per IP limit in NIOServerCnxnFactory > implementation, but not exist in Netty, this is useful to avoid spamming > happened on prod ensembles. > This Jira is going to add similar throttling logic in NettyServerCnxnFactory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zookeeper pull request #623: [ZOOKEEPER-3146] Limit the maximum client conne...
Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/623 ---
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/623 the latest jenkins failure is a known flaky test. merging this in.. ---
[GitHub] zookeeper pull request #620: ZOOKEEPER-2847: Cannot bind to client port when...
Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/620 ---
[jira] [Resolved] (ZOOKEEPER-2847) Cannot bind to client port when reconfig based on old static config
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Han resolved ZOOKEEPER-2847.
Resolution: Fixed
Fix Version/s: 3.6.0
Issue resolved by pull request 620
[https://github.com/apache/zookeeper/pull/620]
> Cannot bind to client port when reconfig based on old static config
> ---
>
> Key: ZOOKEEPER-2847
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2847
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.3, 3.6.0
>Reporter: Fangmin Lv
>Assignee: Yisong Yue
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When started the ensemble with old static config that the server string
> doesn't have client port, dynamically remove and add the same server from the
> ensemble will cause that server cannot bind to client port, and the ZooKeeper
> server cannot serve client requests anymore.
> From the code, we'll set the clientAddr to null when start up with old static
> config, and dynamic config forces to have part, which will
> trigger the following rebind code in QuorumPeer#processReconfig, and cause
> the address already in used issue.
> public boolean processReconfig(QuorumVerifier qv, Long suggestedLeaderId,
> Long zxid, boolean restartLE) {
> ...
> if (myNewQS != null && myNewQS.clientAddr != null
> && !myNewQS.clientAddr.equals(oldClientAddr)) {
> cnxnFactory.reconfigure(myNewQS.clientAddr);
> updateThreadName();
> }
> ...
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zookeeper issue #620: ZOOKEEPER-2847: Cannot bind to client port when reconf...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/620 it's really hard to get a green build today. the closest build is https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2237/, which has a failed C test. so i am merging this in. ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219643998
--- Diff: src/java/main/org/apache/zookeeper/server/watch/WatchManager.java
---
@@ -83,18 +97,21 @@ synchronized void removeWatcher(Watcher watcher) {
Set list = watchTable.get(p);
if (list != null) {
list.remove(watcher);
-if (list.size() == 0) {
+if (list.isEmpty()) {
watchTable.remove(p);
}
}
}
}
-Set triggerWatch(String path, EventType type) {
+@Override
+public WatcherOrBitSet triggerWatch(String path, EventType type) {
return triggerWatch(path, type, null);
}
-Set triggerWatch(String path, EventType type, Set
supress) {
+@Override
+public WatcherOrBitSet triggerWatch(
+String path, EventType type, WatcherOrBitSet supress) {
--- End diff --
`suppress ` instead of `supress`
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219640916
--- Diff: src/java/main/org/apache/zookeeper/server/DataTree.java ---
@@ -253,6 +259,14 @@ public DataTree() {
addConfigNode();
nodeDataSize.set(approximateDataSize());
+try {
+dataWatches = WatchManagerFactory.createWatchManager();
+childWatches = WatchManagerFactory.createWatchManager();
+} catch (Exception e) {
+LOG.error("Unexpected exception when creating WatchManager, " +
+"exiting abnormally", e);
--- End diff --
nit: use parameterized logging here.
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219661278
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/WatcherCleaner.java ---
@@ -0,0 +1,176 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.zookeeper.common.Time;
+import org.apache.zookeeper.server.RateLogger;
+import org.apache.zookeeper.server.WorkerService;
+import org.apache.zookeeper.server.WorkerService.WorkRequest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Thread used to lazily clean up the closed watcher, it will trigger the
+ * clean up when the dead watchers get certain number or some number of
+ * seconds has elapsed since last clean up.
+ *
+ * Cost of running it:
+ *
+ * - need to go through all the paths even if the watcher may only
+ * watching a single path
+ * - block in the path BitHashSet when we try to check the dead watcher
+ * which won't block other stuff
+ */
+public class WatcherCleaner extends Thread {
+
+private static final Logger LOG =
LoggerFactory.getLogger(WatcherCleaner.class);
+private final RateLogger RATE_LOGGER = new RateLogger(LOG);
+
+private volatile boolean stopped = false;
+private final Object cleanEvent = new Object();
+private final Random r = new Random(System.nanoTime());
+private final WorkerService cleaners;
+
+private final Set deadWatchers;
+private final DeadWatcherListener listener;
+private final int watcherCleanThreshold;
+private final int watcherCleanIntervalInSeconds;
+private final int maxInProcessingDeadWatchers;
+private final AtomicInteger totalDeadWatchers = new AtomicInteger();
+
+public WatcherCleaner(DeadWatcherListener listener) {
+this(listener,
+Integer.getInteger("zookeeper.watcherCleanThreshold", 1000),
+Integer.getInteger("zookeeper.watcherCleanIntervalInSeconds",
600),
+Integer.getInteger("zookeeper.watcherCleanThreadsNum", 2),
+Integer.getInteger("zookeeper.maxInProcessingDeadWatchers",
-1));
+}
+
+public WatcherCleaner(DeadWatcherListener listener,
+int watcherCleanThreshold, int watcherCleanIntervalInSeconds,
+int watcherCleanThreadsNum, int maxInProcessingDeadWatchers) {
+this.listener = listener;
+this.watcherCleanThreshold = watcherCleanThreshold;
+this.watcherCleanIntervalInSeconds = watcherCleanIntervalInSeconds;
+int suggestedMaxInProcessingThreshold =
+watcherCleanThreshold * watcherCleanThreadsNum;
+if (maxInProcessingDeadWatchers > 0 &&
+maxInProcessingDeadWatchers <
suggestedMaxInProcessingThreshold) {
+maxInProcessingDeadWatchers =
suggestedMaxInProcessingThreshold;
+LOG.info("The maxInProcessingDeadWatchers config is smaller " +
+"than the suggested one, change it to use {}",
+maxInProcessingDeadWatchers);
+}
+this.maxInProcessingDeadWatchers = maxInProcessingDeadWatchers;
+this.deadWatchers = new HashSet();
+this.cleaners = new WorkerService("DeadWatcherCleanner",
+watcherCleanThreadsNum, false);
+
+LOG.info("watcherCleanThreshold={},
watcherCleanIntervalInSeconds={}" +
+", watcherCleanThreadsNum={},
maxInProcessingDeadWatchers={}",
+watcherCleanThreshold, watcherCleanIntervalInSeconds,
+watcherCleanThreadsNum, maxInProcessingDeadWatchers);
+}
+
+public void addDeadWatcher(int watcherBit) {
+// Wait if
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219659808
--- Diff: src/java/main/org/apache/zookeeper/server/util/BitMap.java ---
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.util;
+
+import java.util.Map;
+import java.util.HashMap;
+import java.util.BitSet;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * This is a helper class to maintain the bit to specific value and the
+ * reversed value to bit mapping.
+ */
+public class BitMap {
+
+private final Map value2Bit = new HashMap();
+private final Map bit2Value = new HashMap();
+
+private final BitSet freedBitSet = new BitSet();
+private Integer nextBit = Integer.valueOf(0);
+
+private final ReentrantReadWriteLock rwLock = new
ReentrantReadWriteLock();
+
+public Integer add(T value) {
+Integer bit = getBit(value);
+if (bit != null) {
+return bit;
+}
--- End diff --
is it usual that we usually call this `add` method with same value over and
over? If that's the case, then this optimization is good, but if not, then this
adds unnecessary cost of acquiring and releasing the reader lock plus writer
lock per `add`, as opposed to just acquiring and releasing writer lock once per
`add`.
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219659171 --- Diff: src/java/main/org/apache/zookeeper/server/DumbWatcher.java --- @@ -0,0 +1,101 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.nio.ByteBuffer; +import java.security.cert.Certificate; + +import org.apache.jute.Record; +import org.apache.zookeeper.Watcher; +import org.apache.zookeeper.WatchedEvent; +import org.apache.zookeeper.proto.ReplyHeader; +import org.apache.zookeeper.server.ServerCnxn; +import org.apache.zookeeper.server.ServerStats; + +/** + * A empthy watcher implementation used in bench and unit test. --- End diff -- spelling: `empty` ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219660065 --- Diff: zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml --- @@ -1025,6 +1025,102 @@ server.3=zoo3:2888:3888 + + +watchManaggerName + + + (Java system property only: zookeeper.watchManaggerName) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 New watcher +manager WatchManagerOptimized is added to optimize the memory overhead in heavy watch use cases. This +config is used define which watch manager to be used. Currently, we only support WatchManager and +WatchManagerOptimized. + + + + +watcherCleanThreadsNum + + + (Java system property only: zookeeper.watcherCleanThreadsNum) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, this config is used to decide how +many thread is used in the WatcherCleaner. More thread usually means larger clean up throughput. The +default value is 2, which is good enough even for heavy and continuous session closing/receating cases. + + + + +watcherCleanThreshold + + + (Java system property only: zookeeper.watcherCleanThreshold) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, the clean up process is relatively +heavy, batch processing will reduce the cost and improve the performance. This setting is used to decide +the batch size. The default one is 1000, we don't need to change it if there is no memory or clean up +speed issue. + + + + +watcherCleanIntervalInSeconds + + + (Java system property only: zookeeper.watcherCleanIntervalInSeconds) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, the clean up process is relatively +heavy, batch processing will reduce the cost and improve the performance. Besides watcherCleanThreshold, +this setting is used to clean up the dead watchers after certain time even the dead watchers are not larger +than watcherCleanThreshold, so that we won't leave the dead watchers there for too long. The default setting +is 10 minutes, which usually don't need to be changed. + + + + +maxInProcessingDeadWatchers + + + (Java system property only: zookeeper.maxInProcessingDeadWatchers) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 This is used +to control how many backlog can we have in the WatcherCleaner, when it reaches this number, it will +slow down adding the dead watcher to WatcherCleaner, which will in turn slow down adding and closing +watchers, so that we can avoid OOM issue. By default there is no limit, you can set it to values like +watcherCleanThreshold * 1000. + + + + +bitHashCacheSize + + + (Java system property only: zookeeper.bitHashCacheSize) + + New 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 This is the +settin used to decide the HashSet cache size in the BitHashSet implementation. Without HashSet, we +need to to use O(N) time to get the elements, N is the bit numbers in elementBits. But we need to +keep the size small to make sure it doesn't cost too much in memory, there is a tradeoff between memory --- End diff -- space between `trade` and `off`. ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219645409 --- Diff: zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml --- @@ -1025,6 +1025,102 @@ server.3=zoo3:2888:3888 + + +watchManaggerName + + + (Java system property only: zookeeper.watchManaggerName) --- End diff -- this should be `zookeeper.watchManagerName`. ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219659161 --- Diff: src/java/main/org/apache/zookeeper/server/DumbWatcher.java --- @@ -0,0 +1,101 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.nio.ByteBuffer; +import java.security.cert.Certificate; + +import org.apache.jute.Record; +import org.apache.zookeeper.Watcher; --- End diff -- unused import. ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219641593 --- Diff: src/java/main/org/apache/zookeeper/server/watch/IWatchManager.java --- @@ -0,0 +1,144 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.watch; + +import java.io.PrintWriter; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; + +import org.apache.zookeeper.Watcher; +import org.apache.zookeeper.Watcher.Event.EventType; +import org.apache.zookeeper.server.ServerCnxn; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; --- End diff -- Remove all imports here except these three since rest of those were not used (my guess is this file was copied pasted?) ` import java.io.PrintWriter; import org.apache.zookeeper.Watcher; import org.apache.zookeeper.Watcher.Event.EventType; ` ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219659865
--- Diff: src/java/main/org/apache/zookeeper/server/util/BitMap.java ---
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.util;
+
+import java.util.Map;
+import java.util.HashMap;
+import java.util.BitSet;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * This is a helper class to maintain the bit to specific value and the
+ * reversed value to bit mapping.
+ */
+public class BitMap {
+
+private final Map value2Bit = new HashMap();
+private final Map bit2Value = new HashMap();
+
+private final BitSet freedBitSet = new BitSet();
+private Integer nextBit = Integer.valueOf(0);
+
+private final ReentrantReadWriteLock rwLock = new
ReentrantReadWriteLock();
+
+public Integer add(T value) {
+Integer bit = getBit(value);
+if (bit != null) {
+return bit;
+}
--- End diff --
I am also wondering, if this optimization is indeed useful, why not do the
same for the `remove` methods, that is, check and return early with a read lock
before trying to acquire a write lock.
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219659934 --- Diff: src/java/main/org/apache/zookeeper/server/watch/BitHashSet.java --- @@ -0,0 +1,156 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.watch; + +import java.util.BitSet; +import java.util.Set; +import java.util.HashSet; +import java.util.Iterator; +import java.lang.Iterable; + +import org.apache.zookeeper.server.util.BitMap; + +/** + * Using BitSet to store all the elements, and use HashSet to cache limited + * number of elements to find a balance between memory and time complexity. + * + * Without HashSet, we need to to use O(N) time to get the elements, N is + * the bit numbers in elementBits. But we need to keep the size small to make + * sure it doesn't cost too much in memory, there is a tradeoff between --- End diff -- nit `trade off` ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219661330
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/WatcherCleaner.java ---
@@ -0,0 +1,176 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.zookeeper.common.Time;
+import org.apache.zookeeper.server.RateLogger;
+import org.apache.zookeeper.server.WorkerService;
+import org.apache.zookeeper.server.WorkerService.WorkRequest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Thread used to lazily clean up the closed watcher, it will trigger the
+ * clean up when the dead watchers get certain number or some number of
+ * seconds has elapsed since last clean up.
+ *
+ * Cost of running it:
+ *
+ * - need to go through all the paths even if the watcher may only
+ * watching a single path
+ * - block in the path BitHashSet when we try to check the dead watcher
+ * which won't block other stuff
+ */
+public class WatcherCleaner extends Thread {
+
+private static final Logger LOG =
LoggerFactory.getLogger(WatcherCleaner.class);
+private final RateLogger RATE_LOGGER = new RateLogger(LOG);
+
+private volatile boolean stopped = false;
+private final Object cleanEvent = new Object();
+private final Random r = new Random(System.nanoTime());
+private final WorkerService cleaners;
+
+private final Set deadWatchers;
+private final DeadWatcherListener listener;
+private final int watcherCleanThreshold;
+private final int watcherCleanIntervalInSeconds;
+private final int maxInProcessingDeadWatchers;
+private final AtomicInteger totalDeadWatchers = new AtomicInteger();
+
+public WatcherCleaner(DeadWatcherListener listener) {
+this(listener,
+Integer.getInteger("zookeeper.watcherCleanThreshold", 1000),
+Integer.getInteger("zookeeper.watcherCleanIntervalInSeconds",
600),
+Integer.getInteger("zookeeper.watcherCleanThreadsNum", 2),
+Integer.getInteger("zookeeper.maxInProcessingDeadWatchers",
-1));
+}
+
+public WatcherCleaner(DeadWatcherListener listener,
+int watcherCleanThreshold, int watcherCleanIntervalInSeconds,
+int watcherCleanThreadsNum, int maxInProcessingDeadWatchers) {
+this.listener = listener;
+this.watcherCleanThreshold = watcherCleanThreshold;
+this.watcherCleanIntervalInSeconds = watcherCleanIntervalInSeconds;
+int suggestedMaxInProcessingThreshold =
+watcherCleanThreshold * watcherCleanThreadsNum;
+if (maxInProcessingDeadWatchers > 0 &&
+maxInProcessingDeadWatchers <
suggestedMaxInProcessingThreshold) {
+maxInProcessingDeadWatchers =
suggestedMaxInProcessingThreshold;
+LOG.info("The maxInProcessingDeadWatchers config is smaller " +
+"than the suggested one, change it to use {}",
+maxInProcessingDeadWatchers);
+}
+this.maxInProcessingDeadWatchers = maxInProcessingDeadWatchers;
+this.deadWatchers = new HashSet();
+this.cleaners = new WorkerService("DeadWatcherCleanner",
+watcherCleanThreadsNum, false);
+
+LOG.info("watcherCleanThreshold={},
watcherCleanIntervalInSeconds={}" +
+", watcherCleanThreadsNum={},
maxInProcessingDeadWatchers={}",
+watcherCleanThreshold, watcherCleanIntervalInSeconds,
+watcherCleanThreadsNum, maxInProcessingDeadWatchers);
+}
+
+public void addDeadWatcher(int watcherBit) {
+// Wait if
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219661253
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/WatcherCleaner.java ---
@@ -0,0 +1,176 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.zookeeper.common.Time;
+import org.apache.zookeeper.server.RateLogger;
+import org.apache.zookeeper.server.WorkerService;
+import org.apache.zookeeper.server.WorkerService.WorkRequest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Thread used to lazily clean up the closed watcher, it will trigger the
+ * clean up when the dead watchers get certain number or some number of
+ * seconds has elapsed since last clean up.
+ *
+ * Cost of running it:
+ *
+ * - need to go through all the paths even if the watcher may only
+ * watching a single path
+ * - block in the path BitHashSet when we try to check the dead watcher
+ * which won't block other stuff
+ */
+public class WatcherCleaner extends Thread {
+
+private static final Logger LOG =
LoggerFactory.getLogger(WatcherCleaner.class);
+private final RateLogger RATE_LOGGER = new RateLogger(LOG);
+
+private volatile boolean stopped = false;
+private final Object cleanEvent = new Object();
+private final Random r = new Random(System.nanoTime());
+private final WorkerService cleaners;
+
+private final Set deadWatchers;
+private final DeadWatcherListener listener;
+private final int watcherCleanThreshold;
+private final int watcherCleanIntervalInSeconds;
+private final int maxInProcessingDeadWatchers;
+private final AtomicInteger totalDeadWatchers = new AtomicInteger();
+
+public WatcherCleaner(DeadWatcherListener listener) {
+this(listener,
+Integer.getInteger("zookeeper.watcherCleanThreshold", 1000),
+Integer.getInteger("zookeeper.watcherCleanIntervalInSeconds",
600),
+Integer.getInteger("zookeeper.watcherCleanThreadsNum", 2),
+Integer.getInteger("zookeeper.maxInProcessingDeadWatchers",
-1));
+}
+
+public WatcherCleaner(DeadWatcherListener listener,
+int watcherCleanThreshold, int watcherCleanIntervalInSeconds,
+int watcherCleanThreadsNum, int maxInProcessingDeadWatchers) {
+this.listener = listener;
+this.watcherCleanThreshold = watcherCleanThreshold;
+this.watcherCleanIntervalInSeconds = watcherCleanIntervalInSeconds;
+int suggestedMaxInProcessingThreshold =
+watcherCleanThreshold * watcherCleanThreadsNum;
+if (maxInProcessingDeadWatchers > 0 &&
+maxInProcessingDeadWatchers <
suggestedMaxInProcessingThreshold) {
+maxInProcessingDeadWatchers =
suggestedMaxInProcessingThreshold;
+LOG.info("The maxInProcessingDeadWatchers config is smaller " +
+"than the suggested one, change it to use {}",
+maxInProcessingDeadWatchers);
+}
+this.maxInProcessingDeadWatchers = maxInProcessingDeadWatchers;
+this.deadWatchers = new HashSet();
+this.cleaners = new WorkerService("DeadWatcherCleanner",
+watcherCleanThreadsNum, false);
+
+LOG.info("watcherCleanThreshold={},
watcherCleanIntervalInSeconds={}" +
+", watcherCleanThreadsNum={},
maxInProcessingDeadWatchers={}",
+watcherCleanThreshold, watcherCleanIntervalInSeconds,
+watcherCleanThreadsNum, maxInProcessingDeadWatchers);
+}
+
+public void addDeadWatcher(int watcherBit) {
+// Wait if
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219659984
--- Diff: src/java/main/org/apache/zookeeper/server/watch/BitHashSet.java
---
@@ -0,0 +1,156 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.util.BitSet;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.lang.Iterable;
+
+import org.apache.zookeeper.server.util.BitMap;
+
+/**
+ * Using BitSet to store all the elements, and use HashSet to cache limited
+ * number of elements to find a balance between memory and time complexity.
+ *
+ * Without HashSet, we need to to use O(N) time to get the elements, N is
+ * the bit numbers in elementBits. But we need to keep the size small to
make
+ * sure it doesn't cost too much in memory, there is a tradeoff between
+ * memory and time complexity.
+ *
+ * Previously, was deciding to dynamically switch between SparseBitSet and
+ * HashSet based on the memory consumption, but it will take time to copy
+ * data over and may have some herd effect of keep copying data from one
+ * data structure to anther. The current solution can do a very good job
+ * given most of the paths have limited number of elements.
+ */
+public class BitHashSet implements Iterable {
+
+static final long serialVersionUID = 6382565447128283568L;
--- End diff --
why do we need this? `BitHashSet` is not implementing `Serializable` here...
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219660054 --- Diff: zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml --- @@ -1025,6 +1025,102 @@ server.3=zoo3:2888:3888 + + +watchManaggerName + + + (Java system property only: zookeeper.watchManaggerName) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 New watcher +manager WatchManagerOptimized is added to optimize the memory overhead in heavy watch use cases. This +config is used define which watch manager to be used. Currently, we only support WatchManager and +WatchManagerOptimized. + + + + +watcherCleanThreadsNum + + + (Java system property only: zookeeper.watcherCleanThreadsNum) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, this config is used to decide how +many thread is used in the WatcherCleaner. More thread usually means larger clean up throughput. The +default value is 2, which is good enough even for heavy and continuous session closing/receating cases. + + + + +watcherCleanThreshold + + + (Java system property only: zookeeper.watcherCleanThreshold) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, the clean up process is relatively +heavy, batch processing will reduce the cost and improve the performance. This setting is used to decide +the batch size. The default one is 1000, we don't need to change it if there is no memory or clean up +speed issue. + + + + +watcherCleanIntervalInSeconds + + + (Java system property only: zookeeper.watcherCleanIntervalInSeconds) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, the clean up process is relatively +heavy, batch processing will reduce the cost and improve the performance. Besides watcherCleanThreshold, +this setting is used to clean up the dead watchers after certain time even the dead watchers are not larger +than watcherCleanThreshold, so that we won't leave the dead watchers there for too long. The default setting +is 10 minutes, which usually don't need to be changed. + + + + +maxInProcessingDeadWatchers + + + (Java system property only: zookeeper.maxInProcessingDeadWatchers) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 This is used +to control how many backlog can we have in the WatcherCleaner, when it reaches this number, it will +slow down adding the dead watcher to WatcherCleaner, which will in turn slow down adding and closing +watchers, so that we can avoid OOM issue. By default there is no limit, you can set it to values like +watcherCleanThreshold * 1000. + + + + +bitHashCacheSize + + + (Java system property only: zookeeper.bitHashCacheSize) + + New 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 This is the +settin used to decide the HashSet cache size in the BitHashSet implementation. Without HashSet, we --- End diff -- spell check: `setting used to` ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219660630
--- Diff: src/java/main/org/apache/zookeeper/server/watch/BitHashSet.java
---
@@ -0,0 +1,156 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.util.BitSet;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.lang.Iterable;
+
+import org.apache.zookeeper.server.util.BitMap;
+
+/**
+ * Using BitSet to store all the elements, and use HashSet to cache limited
+ * number of elements to find a balance between memory and time complexity.
+ *
+ * Without HashSet, we need to to use O(N) time to get the elements, N is
+ * the bit numbers in elementBits. But we need to keep the size small to
make
+ * sure it doesn't cost too much in memory, there is a tradeoff between
+ * memory and time complexity.
+ *
+ * Previously, was deciding to dynamically switch between SparseBitSet and
+ * HashSet based on the memory consumption, but it will take time to copy
+ * data over and may have some herd effect of keep copying data from one
+ * data structure to anther. The current solution can do a very good job
+ * given most of the paths have limited number of elements.
+ */
+public class BitHashSet implements Iterable {
+
+static final long serialVersionUID = 6382565447128283568L;
+
+/**
+ * Change to SparseBitSet if we we want to optimize more, the number of
+ * elements on a single server is usually limited, so BitSet should be
+ * fine.
+ */
+private final BitSet elementBits = new BitSet();
+private final Set cache = new HashSet();
+
+private final int cacheSize;
+
+// To record how many elements in this set.
+private int elementCount = 0;
+
+public BitHashSet() {
+this(Integer.getInteger("zookeeper.bitHashCacheSize", 10));
+}
+
+public BitHashSet(int cacheSize) {
+this.cacheSize = cacheSize;
+}
+
+public synchronized boolean add(Integer elementBit) {
+if (elementBit == null || elementBits.get(elementBit)) {
+return false;
+}
+if (cache.size() < cacheSize) {
+cache.add(elementBit);
+}
+elementBits.set(elementBit);
+elementCount++;
+return true;
+}
+
+/**
+ * Remove the watches, and return the number of watches being removed.
+ */
+public synchronized int remove(Set bitSet, BitSet bits) {
+cache.removeAll(bitSet);
+elementBits.andNot(bits);
+int elementCountBefore = elementCount;
+elementCount = elementBits.cardinality();
+return elementCountBefore - elementCount;
+}
+
+public synchronized boolean remove(Integer elementBit) {
+if (elementBit == null || !elementBits.get(elementBit)) {
+return false;
+}
+
+cache.remove(elementBit);
+elementBits.clear(elementBit);
+elementCount--;
+return true;
+}
+
+public synchronized boolean contains(Integer elementBit) {
+if (elementBit == null) {
+return false;
+}
+return elementBits.get(elementBit);
+}
+
+public synchronized int size() {
+return elementCount;
+}
+
+/**
+ * This function is not thread-safe, need to synchronized when
+ * iterate through this set.
+ */
+@Override
+public Iterator iterator() {
--- End diff --
curious - what's this `iterator` is used for? I did not spot any usage of
it by just going over the diff..
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219644285
--- Diff: src/java/main/org/apache/zookeeper/server/watch/WatchManager.java
---
@@ -180,7 +192,8 @@ synchronized void dumpWatches(PrintWriter pwriter,
boolean byPath) {
*watcher object reference
* @return true if the watcher exists, false otherwise
*/
--- End diff --
we should remove this comment, which is already present in the
`IWatchManager`, also for consistency with other overrides in `WatchManager`.
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219642525
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/IWatchManager.java ---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.io.PrintWriter;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.zookeeper.Watcher;
+import org.apache.zookeeper.Watcher.Event.EventType;
+import org.apache.zookeeper.server.ServerCnxn;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface IWatchManager {
+
+/**
+ * Add watch to specific path.
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher added is not already present
+ */
+public boolean addWatch(String path, Watcher watcher);
+
+/**
+ * Checks the specified watcher exists for the given path
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher exists, false otherwise
+ */
+public boolean containsWatcher(String path, Watcher watcher);
+
+/**
+ * Removes the specified watcher for the given path
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher successfully removed, false otherwise
+ */
+public boolean removeWatcher(String path, Watcher watcher);
+
+/**
+ * The entry to remove the watcher when the cnxn is closed.
+ *
+ * @param watcher watcher object reference
+ */
+public void removeWatcher(Watcher watcher);
+
+/**
+ * Distribute the watch event for the given path.
+ *
+ * @param path znode path
+ * @param type the watch event type
+ *
+ * @return the watchers have been notified
+ */
+public WatcherOrBitSet triggerWatch(String path, EventType type);
+
+/**
+ * Distribute the watch event for the given path, but ignore those
+ * supressed ones.
--- End diff --
spell check: `suppressed` instead of`supressed`
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219642002
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/IWatchManager.java ---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.io.PrintWriter;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.zookeeper.Watcher;
+import org.apache.zookeeper.Watcher.Event.EventType;
+import org.apache.zookeeper.server.ServerCnxn;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface IWatchManager {
+
+/**
+ * Add watch to specific path.
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher added is not already present
+ */
+public boolean addWatch(String path, Watcher watcher);
+
+/**
+ * Checks the specified watcher exists for the given path
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher exists, false otherwise
+ */
+public boolean containsWatcher(String path, Watcher watcher);
+
+/**
+ * Removes the specified watcher for the given path
--- End diff --
nit: missing full stop at end of sentence.
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219641957
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/IWatchManager.java ---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.io.PrintWriter;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.zookeeper.Watcher;
+import org.apache.zookeeper.Watcher.Event.EventType;
+import org.apache.zookeeper.server.ServerCnxn;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface IWatchManager {
+
+/**
+ * Add watch to specific path.
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher added is not already present
+ */
+public boolean addWatch(String path, Watcher watcher);
+
+/**
+ * Checks the specified watcher exists for the given path
--- End diff --
nit: missing full stop at end of sentence.
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219660689
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/DeadWatcherListener.java ---
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.util.Set;
+
+/**
+ * Interface used to process the dead watchers related to closed cnxns.
+ */
+public interface DeadWatcherListener {
--- End diff --
would be good to rename this to `IDeadWatchListner`, which makes it obvious
this is an interface. We already do this for `IWatchManager`.
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219660120 --- Diff: src/java/main/org/apache/zookeeper/server/watch/BitHashSet.java --- @@ -0,0 +1,156 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.watch; + +import java.util.BitSet; +import java.util.Set; +import java.util.HashSet; +import java.util.Iterator; +import java.lang.Iterable; + +import org.apache.zookeeper.server.util.BitMap; + +/** + * Using BitSet to store all the elements, and use HashSet to cache limited + * number of elements to find a balance between memory and time complexity. + * + * Without HashSet, we need to to use O(N) time to get the elements, N is --- End diff -- nit `need to to` ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219645514 --- Diff: zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml --- @@ -1025,6 +1025,102 @@ server.3=zoo3:2888:3888 + + +watchManaggerName + + + (Java system property only: zookeeper.watchManaggerName) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 New watcher +manager WatchManagerOptimized is added to optimize the memory overhead in heavy watch use cases. This +config is used define which watch manager to be used. Currently, we only support WatchManager and --- End diff -- is used to define which watcher ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219660592
--- Diff: src/java/main/org/apache/zookeeper/server/watch/BitHashSet.java
---
@@ -0,0 +1,156 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.util.BitSet;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.lang.Iterable;
+
+import org.apache.zookeeper.server.util.BitMap;
+
+/**
+ * Using BitSet to store all the elements, and use HashSet to cache limited
+ * number of elements to find a balance between memory and time complexity.
+ *
+ * Without HashSet, we need to to use O(N) time to get the elements, N is
+ * the bit numbers in elementBits. But we need to keep the size small to
make
+ * sure it doesn't cost too much in memory, there is a tradeoff between
+ * memory and time complexity.
+ *
+ * Previously, was deciding to dynamically switch between SparseBitSet and
+ * HashSet based on the memory consumption, but it will take time to copy
+ * data over and may have some herd effect of keep copying data from one
+ * data structure to anther. The current solution can do a very good job
+ * given most of the paths have limited number of elements.
+ */
+public class BitHashSet implements Iterable {
+
+static final long serialVersionUID = 6382565447128283568L;
+
+/**
+ * Change to SparseBitSet if we we want to optimize more, the number of
+ * elements on a single server is usually limited, so BitSet should be
+ * fine.
+ */
+private final BitSet elementBits = new BitSet();
+private final Set cache = new HashSet();
+
+private final int cacheSize;
+
+// To record how many elements in this set.
+private int elementCount = 0;
+
+public BitHashSet() {
+this(Integer.getInteger("zookeeper.bitHashCacheSize", 10));
+}
+
+public BitHashSet(int cacheSize) {
+this.cacheSize = cacheSize;
+}
+
+public synchronized boolean add(Integer elementBit) {
+if (elementBit == null || elementBits.get(elementBit)) {
+return false;
+}
+if (cache.size() < cacheSize) {
+cache.add(elementBit);
+}
+elementBits.set(elementBit);
+elementCount++;
+return true;
+}
+
+/**
+ * Remove the watches, and return the number of watches being removed.
+ */
+public synchronized int remove(Set bitSet, BitSet bits) {
+cache.removeAll(bitSet);
+elementBits.andNot(bits);
+int elementCountBefore = elementCount;
+elementCount = elementBits.cardinality();
+return elementCountBefore - elementCount;
+}
+
+public synchronized boolean remove(Integer elementBit) {
+if (elementBit == null || !elementBits.get(elementBit)) {
+return false;
+}
+
+cache.remove(elementBit);
+elementBits.clear(elementBit);
+elementCount--;
+return true;
+}
+
+public synchronized boolean contains(Integer elementBit) {
+if (elementBit == null) {
+return false;
+}
+return elementBits.get(elementBit);
--- End diff --
should we look up `cache` first here? If not, what's the purpose of adding
`cache`?
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219660094 --- Diff: zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml --- @@ -1025,6 +1025,102 @@ server.3=zoo3:2888:3888 + + +watchManaggerName + + + (Java system property only: zookeeper.watchManaggerName) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 New watcher +manager WatchManagerOptimized is added to optimize the memory overhead in heavy watch use cases. This +config is used define which watch manager to be used. Currently, we only support WatchManager and +WatchManagerOptimized. + + + + +watcherCleanThreadsNum + + + (Java system property only: zookeeper.watcherCleanThreadsNum) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, this config is used to decide how +many thread is used in the WatcherCleaner. More thread usually means larger clean up throughput. The +default value is 2, which is good enough even for heavy and continuous session closing/receating cases. + + + + +watcherCleanThreshold + + + (Java system property only: zookeeper.watcherCleanThreshold) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, the clean up process is relatively +heavy, batch processing will reduce the cost and improve the performance. This setting is used to decide +the batch size. The default one is 1000, we don't need to change it if there is no memory or clean up +speed issue. + + + + +watcherCleanIntervalInSeconds + + + (Java system property only: zookeeper.watcherCleanIntervalInSeconds) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, the clean up process is relatively +heavy, batch processing will reduce the cost and improve the performance. Besides watcherCleanThreshold, +this setting is used to clean up the dead watchers after certain time even the dead watchers are not larger +than watcherCleanThreshold, so that we won't leave the dead watchers there for too long. The default setting +is 10 minutes, which usually don't need to be changed. + + + + +maxInProcessingDeadWatchers + + + (Java system property only: zookeeper.maxInProcessingDeadWatchers) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 This is used +to control how many backlog can we have in the WatcherCleaner, when it reaches this number, it will +slow down adding the dead watcher to WatcherCleaner, which will in turn slow down adding and closing +watchers, so that we can avoid OOM issue. By default there is no limit, you can set it to values like +watcherCleanThreshold * 1000. + + + + +bitHashCacheSize + + + (Java system property only: zookeeper.bitHashCacheSize) + + New 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 This is the +settin used to decide the HashSet cache size in the BitHashSet implementation. Without HashSet, we +need to to use O(N) time to get the elements, N is the bit numbers in elementBits. But we need to --- End diff -- `need to` instead of `need to to` ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219660892 --- Diff: zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml --- @@ -1025,6 +1025,102 @@ server.3=zoo3:2888:3888 + + +watchManaggerName + + + (Java system property only: zookeeper.watchManaggerName) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 New watcher +manager WatchManagerOptimized is added to optimize the memory overhead in heavy watch use cases. This +config is used define which watch manager to be used. Currently, we only support WatchManager and +WatchManagerOptimized. + + + + +watcherCleanThreadsNum + + + (Java system property only: zookeeper.watcherCleanThreadsNum) + + New in 3.6.0: Added in +https://issues.apache.org/jira/browse/ZOOKEEPER-1179;>ZOOKEEPER-1179 The new watcher +manager WatchManagerOptimized will clean up the dead watchers lazily, this config is used to decide how +many thread is used in the WatcherCleaner. More thread usually means larger clean up throughput. The +default value is 2, which is good enough even for heavy and continuous session closing/receating cases. --- End diff -- `closing/recreating` ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219659918 --- Diff: src/java/main/org/apache/zookeeper/server/watch/BitHashSet.java --- @@ -0,0 +1,156 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.watch; --- End diff -- is there a reason that this `BitHashSet` is part of `server.watch` package rather than part of `server.util` package, where a similar helper class `BitMap` sits? ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/590#discussion_r219642730
--- Diff:
src/java/main/org/apache/zookeeper/server/watch/IWatchManager.java ---
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.watch;
+
+import java.io.PrintWriter;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.zookeeper.Watcher;
+import org.apache.zookeeper.Watcher.Event.EventType;
+import org.apache.zookeeper.server.ServerCnxn;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface IWatchManager {
+
+/**
+ * Add watch to specific path.
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher added is not already present
+ */
+public boolean addWatch(String path, Watcher watcher);
+
+/**
+ * Checks the specified watcher exists for the given path
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher exists, false otherwise
+ */
+public boolean containsWatcher(String path, Watcher watcher);
+
+/**
+ * Removes the specified watcher for the given path
+ *
+ * @param path znode path
+ * @param watcher watcher object reference
+ *
+ * @return true if the watcher successfully removed, false otherwise
+ */
+public boolean removeWatcher(String path, Watcher watcher);
+
+/**
+ * The entry to remove the watcher when the cnxn is closed.
+ *
+ * @param watcher watcher object reference
+ */
+public void removeWatcher(Watcher watcher);
+
+/**
+ * Distribute the watch event for the given path.
+ *
+ * @param path znode path
+ * @param type the watch event type
+ *
+ * @return the watchers have been notified
+ */
+public WatcherOrBitSet triggerWatch(String path, EventType type);
+
+/**
+ * Distribute the watch event for the given path, but ignore those
+ * supressed ones.
+ *
+ * @param path znode path
+ * @param type the watch event type
+ * @param supress the supressed watcher set
+ *
+ * @return the watchers have been notified
+ */
+public WatcherOrBitSet triggerWatch(
+String path, EventType type, WatcherOrBitSet supress);
--- End diff --
similar spelling issue for `supress`
---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219659886 --- Diff: src/java/main/org/apache/zookeeper/server/watch/BitHashSet.java --- @@ -0,0 +1,156 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.watch; + +import java.util.BitSet; +import java.util.Set; +import java.util.HashSet; +import java.util.Iterator; +import java.lang.Iterable; + +import org.apache.zookeeper.server.util.BitMap; --- End diff -- I don't think this is used anywhere in this file. ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219659165 --- Diff: src/java/main/org/apache/zookeeper/server/DumbWatcher.java --- @@ -0,0 +1,101 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.nio.ByteBuffer; +import java.security.cert.Certificate; + +import org.apache.jute.Record; +import org.apache.zookeeper.Watcher; +import org.apache.zookeeper.WatchedEvent; +import org.apache.zookeeper.proto.ReplyHeader; +import org.apache.zookeeper.server.ServerCnxn; +import org.apache.zookeeper.server.ServerStats; --- End diff -- unused imports. ---
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user dineshappavoo commented on the issue: https://github.com/apache/zookeeper/pull/623 Hey @lvfangmin , This change looks good overall. I am trying to involve in the community. I had a question about this change. I thought I would ask here. Please bare with me if this is very basic which I do not know :). 1. What is the difference between NIOServerCnxnFactory rate limiting and this one? What will be the impact after this change is merged? Thanks. ---
ZooKeeper-trunk - Build # 199 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk/199/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 284.41 KB...]
[exec] : elapsed 1001 : OK
[exec] Zookeeper_simpleSystem::testLogCallbackClearLog Message Received:
[2018-09-21 23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1080: Client
environment:zookeeper.version=zookeeper C client 3.6.0]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1084: Client
environment:host.name=asf909.gq1.ygridcore.net]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1091: Client
environment:os.name=Linux]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1092: Client
environment:os.arch=3.13.0-153-generic]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1093: Client
environment:os.version=#203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1101: Client
environment:user.name=jenkins]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1109: Client
environment:user.home=/home/jenkins]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@log_env@1121: Client
environment:user.dir=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/build/test/test-cppunit]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04a39cf40):ZOO_INFO@zookeeper_init_internal@1167:
Initiating client connection, host=127.0.0.1:22181 sessionTimeout=1
watcher=0x4639e0 sessionId=0 sessionPasswd= context=0x7ffe8f65d300
flags=0]
[exec] Log Message Received: [2018-09-21
23:34:04,901:27284(0x2ae04c3ff700):ZOO_INFO@check_events@2454: initiated
connection to server 127.0.0.1:22181]
[exec] Log Message Received: [2018-09-21
23:34:04,913:27284(0x2ae04c3ff700):ZOO_INFO@check_events@2506: session
establishment complete on server 127.0.0.1:22181, sessionId=0x101807ebb41000f,
negotiated timeout=1 ]
[exec] : elapsed 1000 : OK
[exec] Zookeeper_simpleSystem::testAsyncWatcherAutoReset ZooKeeper server
started : elapsed 10511 : OK
[exec] Zookeeper_simpleSystem::testDeserializeString : elapsed 0 : OK
[exec] Zookeeper_simpleSystem::testFirstServerDown : elapsed 1001 : OK
[exec] Zookeeper_simpleSystem::testNonexistentHost : elapsed 1034 : OK
[exec] Zookeeper_simpleSystem::testNullData : elapsed 1033 : OK
[exec] Zookeeper_simpleSystem::testIPV6 : elapsed 1012 : OK
[exec] Zookeeper_simpleSystem::testCreate : elapsed 1018 : OK
[exec] Zookeeper_simpleSystem::testPath : elapsed 1049 : OK
[exec] Zookeeper_simpleSystem::testPathValidation : elapsed 1142 : OK
[exec] Zookeeper_simpleSystem::testPing : elapsed 17646 : OK
[exec] Zookeeper_simpleSystem::testAcl : elapsed 1027 : OK
[exec] Zookeeper_simpleSystem::testChroot : elapsed 3084 : OK
[exec] Zookeeper_simpleSystem::testAuth ZooKeeper server started ZooKeeper
server started : elapsed 31076 : OK
[exec] Zookeeper_simpleSystem::testHangingClient : elapsed 1042 : OK
[exec] Zookeeper_simpleSystem::testWatcherAutoResetWithGlobal ZooKeeper
server started ZooKeeper server started ZooKeeper server started : elapsed
15654 : OK
[exec] Zookeeper_simpleSystem::testWatcherAutoResetWithLocal ZooKeeper
server started ZooKeeper server started ZooKeeper server started : elapsed
15698 : OK
[exec] Zookeeper_simpleSystem::testGetChildren2 : elapsed 1070 : OK
[exec] Zookeeper_simpleSystem::testLastZxid : elapsed 4529 : OK
[exec] Zookeeper_simpleSystem::testRemoveWatchers ZooKeeper server started
: elapsed 4643 : OK
[exec] Zookeeper_readOnly::testReadOnly : elapsed 4134 : OK
[exec] Zookeeper_logClientEnv::testLogClientEnv : elapsed 1 : OK
[exec] OK (76)
[exec] FAIL: zktest-mt
[exec] ==
[exec] 1 of 2 tests failed
[exec] Please report to u...@zookeeper.apache.org
[exec] ==
[exec] make[1]: Leaving directory
`/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/build/test/test-cppunit'
[exec] *** Error in `./zktest-mt': free(): invalid pointer:
0x2ae04a388000 ***
[exec] /bin/bash: line 5: 27284 Aborted
ZKROOT=/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/zookeeper-client/zookeeper-client-c/../..
CLASSPATH=$CLASSPATH:$CLOVER_HOME/lib/clover.jar ${dir}$tst
[exec] make[1]: *** [check-TESTS] Error 1
[exec] make: *** [check-am] Error 2
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk/build.xml:1472: The
following error occurred while
[GitHub] zookeeper issue #620: ZOOKEEPER-2847: Cannot bind to client port when reconf...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/620 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2238/ ---
Failed: ZOOKEEPER- PreCommit Build #2238
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2238/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 80.69 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2238//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2238//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2238//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=B5DE78E96D822FEE28B2E0D1CD1C8B84. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 12 minutes 22 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-2847 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of a2bb8e084801391eec63cf3f1e91b79a5189ce48 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2238/ and message: 'FAILURE 1732 tests run, 1 skipped, 1 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2238/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.ReconfigLegacyTest.testReconfigRemoveClientFromStatic Error Message: null Stack Trace: junit.framework.AssertionFailedError at org.apache.zookeeper.server.quorum.ReconfigLegacyTest.testReconfigRemoveClientFromStatic(ReconfigLegacyTest.java:206) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
[GitHub] zookeeper issue #620: ZOOKEEPER-2847: Cannot bind to client port when reconf...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/620 retest this please ---
Failed: ZOOKEEPER- PreCommit Build #2237
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2237/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 79.66 MB...] [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2237//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2237//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2237//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=AAEF00CD04A38E423F0593807CCFD7A2. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 18 minutes 18 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3098 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
Failed: ZOOKEEPER- PreCommit Build #2235
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2235/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 73.43 MB...] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] -1 contrib tests. The patch failed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2235//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2235//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2235//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/src/java/test/bin/test-github-pr.sh: line 471: 22782 Killed $ANT_HOME/bin/ant -DZookeeperPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=yes test-contrib [exec] [exec] Error: No value specified for option "issue" [exec] Ignore unexpected error logging out. Error: Invalid request: java.net.ProtocolException: Server redirected too many times (20). [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 2 Total time: 10 minutes 9 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3098 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
Failed: ZOOKEEPER- PreCommit Build #2236
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2236/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 34.62 KB...] [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2236//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2236//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2236//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build@2/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build@2/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build@2/build.xml:1842: exec returned: 1 Total time: 2 minutes 40 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [Fast Archiver] Compressed 635.93 KB of artifacts by 20.1% relative to #2225 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3098 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## No tests ran.
Failed: ZOOKEEPER- PreCommit Build #2234
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2234/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 79.41 MB...] [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2234//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2234//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2234//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=A8B42CCDF4518A27256EC7935498AF61. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 14 minutes 5 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3098 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged Error Message: expected: but was: Stack Trace: junit.framework.AssertionFailedError: expected: but was: at org.apache.zookeeper.test.WatchEventWhenAutoResetTest$EventsWatcher.assertEvent(WatchEventWhenAutoResetTest.java:67) at org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged(WatchEventWhenAutoResetTest.java:117) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Failed: ZOOKEEPER- PreCommit Build #2233
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2233/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 76.06 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2233//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2233//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2233//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1945: exec returned: 1 Total time: 17 minutes 20 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3032 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of ac88ca4a4887029fb76423d17b238033b423c473 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2233/ and message: 'FAILURE 1731 tests run, 1 skipped, 1 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2233/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testLeaderElectionWithDisloyalVoter_stillHasMajority Error Message: Server 0 should have joined quorum by now Stack Trace: junit.framework.AssertionFailedError: Server 0 should have joined quorum by now at org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testLeaderElection(QuorumPeerMainTest.java:1486) at org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testLeaderElectionWithDisloyalVoter_stillHasMajority(QuorumPeerMainTest.java:1435) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
[GitHub] zookeeper issue #633: ZOOKEEPER-3032 - MAVEN MIGRATION - move java server, c...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/633 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2233/ ---
Failed: ZOOKEEPER- PreCommit Build #2232
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2232/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 80.54 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2232//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2232//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2232//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=5A08425BD1D98F506D25A96794B869B8. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 13 minutes 32 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3146 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of bf70f5bc5975cb5948bae2bf4ae8629101efac44 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2232/ and message: 'FAILURE 1734 tests run, 1 skipped, 1 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2232/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged Error Message: expected: but was: Stack Trace: junit.framework.AssertionFailedError: expected: but was: at org.apache.zookeeper.test.WatchEventWhenAutoResetTest$EventsWatcher.assertEvent(WatchEventWhenAutoResetTest.java:67) at org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged(WatchEventWhenAutoResetTest.java:117) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/623 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2232/ ---
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/623 retest this please ---
[GitHub] zookeeper pull request #605: [ZOOKEEPER-3125] Fixing pzxid consistent issue ...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/605#discussion_r219628462
--- Diff:
src/java/test/org/apache/zookeeper/server/quorum/FuzzySnapshotRelatedTest.java
---
@@ -162,6 +167,98 @@ public void process(String path) {
new String(zk[followerA].getData(node2, null, null)));
}
+/**
+ * It's possibel during SNAP sync, the parent is serialized before the
+ * child get deleted during sending the snapshot over.
+ *
+ * In which case, we need to make sure the pzxid get correctly updated
+ * when applying the txns received.
+ */
+@Test
+public void testPZxidUpdatedDuringSnapSyncing() throws Exception {
+LOG.info("Enable force snapshot sync");
+System.setProperty(LearnerHandler.FORCE_SNAP_SYNC, "true");
+
+final String parent = "/testPZxidUpdatedWhenDeletingNonExistNode";
+final String child = parent + "/child";
+createEmptyNode(zk[leaderId], parent);
+createEmptyNode(zk[leaderId], child);
+
+LOG.info("shutdown follower {}", followerA);
+mt[followerA].shutdown();
+QuorumPeerMainTest.waitForOne(zk[followerA], States.CONNECTING);
+
+LOG.info("Set up ZKDatabase to catch the node serializing in
DataTree");
+addSerializeListener(leaderId, parent, child);
+
+LOG.info("Restart follower A to trigger a SNAP sync with leader");
+mt[followerA].start();
+QuorumPeerMainTest.waitForOne(zk[followerA], States.CONNECTED);
+
+LOG.info("Check and make sure the pzxid of the parent is the same
" +
+"on leader and follower A");
+compareStat(parent, leaderId, followerA);
+}
+
+/**
+ * It's possible during taking fuzzy snapshot, the parent is serialized
+ * before the child get deleted in the fuzzy range.
+ *
+ * In which case, we need to make sure the pzxid get correctly updated
+ * when replaying the txns.
+ */
+@Test
+public void testPZxidUpdatedWhenLoadingSnapshot() throws Exception {
+
+final String parent = "/testPZxidUpdatedDuringTakingSnapshot";
+final String child = parent + "/child";
+createEmptyNode(zk[followerA], parent);
+createEmptyNode(zk[followerA], child);
+
+LOG.info("Set up ZKDatabase to catch the node serializing in
DataTree");
+addSerializeListener(followerA, parent, child);
+
+LOG.info("Take snapshot on follower A");
+ZooKeeperServer zkServer =
mt[followerA].main.quorumPeer.getActiveServer();
+zkServer.takeSnapshot(true);
+
+LOG.info("Restarting follower A to load snapshot");
+mt[followerA].shutdown();
+mt[followerA].start();
+QuorumPeerMainTest.waitForOne(zk[followerA], States.CONNECTED);
+
+LOG.info("Check and make sure the pzxid of the parent is the same
" +
+"on leader and follower A");
+compareStat(parent, leaderId, followerA);
+}
+
+private void addSerializeListener(int sid, String parent, String
child) {
+final ZooKeeper zkClient = zk[followerA];
+CustomDataTree dt =
+(CustomDataTree)
mt[sid].main.quorumPeer.getZkDb().getDataTree();
+dt.addListener(parent, new NodeSerializeListener() {
+@Override
+public void nodeSerialized(String path) {
+try {
+zkClient.delete(child, -1);
+LOG.info("Deleted the child node after the parent is
serialized");
+} catch (Exception e) {
+LOG.error("Error when deleting node {}", e);
+}
+}
+});
+}
+
+private void compareStat(String path, int sid, int compareWithSid)
throws Exception{
+Stat stat1 = new Stat();
+zk[sid].getData(path, null, stat1);
+
+Stat stat2 = new Stat();
+zk[compareWithSid].getData(path, null, stat2);
--- End diff --
something like this (we should make a utility out of this)
`for (int j = 0; j < 30; j++) {
try {
zk[compareWithSid].getData(path, null, stat2);
} catch (KeeperException.ConnectionLossException e) {
if (j < 29) {
Thread.sleep(1000);
} else {
Assert.fail("fail to get data ..");
}
}
}
`
---
[GitHub] zookeeper pull request #605: [ZOOKEEPER-3125] Fixing pzxid consistent issue ...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/605#discussion_r219628119
--- Diff:
src/java/test/org/apache/zookeeper/server/quorum/FuzzySnapshotRelatedTest.java
---
@@ -162,6 +167,98 @@ public void process(String path) {
new String(zk[followerA].getData(node2, null, null)));
}
+/**
+ * It's possibel during SNAP sync, the parent is serialized before the
+ * child get deleted during sending the snapshot over.
+ *
+ * In which case, we need to make sure the pzxid get correctly updated
+ * when applying the txns received.
+ */
+@Test
+public void testPZxidUpdatedDuringSnapSyncing() throws Exception {
+LOG.info("Enable force snapshot sync");
+System.setProperty(LearnerHandler.FORCE_SNAP_SYNC, "true");
+
+final String parent = "/testPZxidUpdatedWhenDeletingNonExistNode";
+final String child = parent + "/child";
+createEmptyNode(zk[leaderId], parent);
+createEmptyNode(zk[leaderId], child);
+
+LOG.info("shutdown follower {}", followerA);
+mt[followerA].shutdown();
+QuorumPeerMainTest.waitForOne(zk[followerA], States.CONNECTING);
+
+LOG.info("Set up ZKDatabase to catch the node serializing in
DataTree");
+addSerializeListener(leaderId, parent, child);
+
+LOG.info("Restart follower A to trigger a SNAP sync with leader");
+mt[followerA].start();
+QuorumPeerMainTest.waitForOne(zk[followerA], States.CONNECTED);
+
+LOG.info("Check and make sure the pzxid of the parent is the same
" +
+"on leader and follower A");
+compareStat(parent, leaderId, followerA);
+}
+
+/**
+ * It's possible during taking fuzzy snapshot, the parent is serialized
+ * before the child get deleted in the fuzzy range.
+ *
+ * In which case, we need to make sure the pzxid get correctly updated
+ * when replaying the txns.
+ */
+@Test
+public void testPZxidUpdatedWhenLoadingSnapshot() throws Exception {
+
+final String parent = "/testPZxidUpdatedDuringTakingSnapshot";
+final String child = parent + "/child";
+createEmptyNode(zk[followerA], parent);
+createEmptyNode(zk[followerA], child);
+
+LOG.info("Set up ZKDatabase to catch the node serializing in
DataTree");
+addSerializeListener(followerA, parent, child);
+
+LOG.info("Take snapshot on follower A");
+ZooKeeperServer zkServer =
mt[followerA].main.quorumPeer.getActiveServer();
+zkServer.takeSnapshot(true);
+
+LOG.info("Restarting follower A to load snapshot");
+mt[followerA].shutdown();
+mt[followerA].start();
+QuorumPeerMainTest.waitForOne(zk[followerA], States.CONNECTED);
+
+LOG.info("Check and make sure the pzxid of the parent is the same
" +
+"on leader and follower A");
+compareStat(parent, leaderId, followerA);
+}
+
+private void addSerializeListener(int sid, String parent, String
child) {
+final ZooKeeper zkClient = zk[followerA];
+CustomDataTree dt =
+(CustomDataTree)
mt[sid].main.quorumPeer.getZkDb().getDataTree();
+dt.addListener(parent, new NodeSerializeListener() {
+@Override
+public void nodeSerialized(String path) {
+try {
+zkClient.delete(child, -1);
+LOG.info("Deleted the child node after the parent is
serialized");
+} catch (Exception e) {
+LOG.error("Error when deleting node {}", e);
+}
+}
+});
+}
+
+private void compareStat(String path, int sid, int compareWithSid)
throws Exception{
+Stat stat1 = new Stat();
+zk[sid].getData(path, null, stat1);
+
+Stat stat2 = new Stat();
+zk[compareWithSid].getData(path, null, stat2);
--- End diff --
@lvfangmin we should add a try catch with retry here around
`ConnectionLossException`. I noticed this when checking the flaky test
dashboard. See one failure
[example](https://builds.apache.org/job/ZooKeeper-trunk/198/testReport/junit/org.apache.zookeeper.server.quorum/FuzzySnapshotRelatedTest/testPZxidUpdatedWhenLoadingSnapshot/)
---
[GitHub] zookeeper pull request #629: ZOOKEEPER-2641:AvgRequestLatency metric improve...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/629#discussion_r219627594
--- Diff: src/java/main/org/apache/zookeeper/server/ServerStats.java ---
@@ -53,10 +53,11 @@ public ServerStats(Provider provider) {
synchronized public long getMinLatency() {
return minLatency == Long.MAX_VALUE ? 0 : minLatency;
}
-
-synchronized public long getAvgLatency() {
+
+synchronized public double getAvgLatency() {
if (count != 0) {
-return totalLatency / count;
+//be account to four decimal places
+return (totalLatency * 1 / count) / 1.0;
--- End diff --
@maoling here is the link:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2198/artifact/build/test/findbugs/newPatchFindbugsWarnings.html
---
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/623 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2231/ ---
Failed: ZOOKEEPER- PreCommit Build #2231
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2231/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 80.59 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2231//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2231//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2231//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=8962E083BB72568C814964440502E2F9. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 13 minutes 19 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3146 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of bf70f5bc5975cb5948bae2bf4ae8629101efac44 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2231/ and message: 'FAILURE 1734 tests run, 1 skipped, 1 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2231/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.ReconfigFailureCasesTest.testLeaderTimesoutOnNewQuorum Error Message: expected:<[leaderelection]> but was:<[following]> Stack Trace: junit.framework.AssertionFailedError: expected:<[leaderelection]> but was:<[following]> at org.apache.zookeeper.server.quorum.ReconfigFailureCasesTest.testLeaderTimesoutOnNewQuorum(ReconfigFailureCasesTest.java:198) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Re: About Jenkins pre-commit job
>> It is the default config, I can drop. In BK sometimes it is useful in order to save resources for docs/website changes for instance We have separate repo for ZK web site and our doc artifacts only regenerate when we do a new release. So I think there is not much benefit of keeping this option, as we always want Jenkins as the gate before committing any code changes. >> But it seems to me that the 'Jenkins' line never goes green nor red, it stays yellow forever I see what you mean. I think "*Jenkins *Pending — Jenkins is validating pull request ..." should be removed, as it's not telling anything new. I don't know how though. On Fri, Sep 21, 2018 at 11:20 AM, Enrico Olivelli wrote: > Il ven 21 set 2018, 19:11 Michael Han ha scritto: > > > Thanks for the work, Enrico! Looks like it's working? > > > > >> if the description of the PR contains 'skip ci' validation won't be > > run from jenkins > > > > Do we need this option? > > > > > It is the default config, I can drop. In BK sometimes it is useful in order > to save resources for docs/website changes for instance > > > > > >> doc/wiki to update and if there is a common place for storing CI Jobs > > configurations > > > > jenkins job config is stored in the job itself. We don't version control > > it separately. > > With this update the "Jenkins" section is out of date: > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute > > Would you please update the page to reflect latest state of the jenkins - > > github integration? > > > > Ok > > > > > >> It seems that the first line with 'Jenkins' never goes away. > > > > Seems ok to me to keep two lines. The second line does not cover things > > other than test (like find bug checks, release audit checks, etc). > > > > TheI job execution is one, so the informative content is the same: red vs > green. > > But it seems to me that the 'Jenkins' line never goes green nor red, it > stays yellow forever > > > Enrico > > > > On Fri, Sep 21, 2018 at 5:51 AM, Enrico Olivelli > > wrote: > > > >> Too early... > >> It seems that the first line with 'Jenkins' never goes away. > >> > >> Patrick, do you have some hint ? > >> Otherwise I can ask to infra or bui...@apache.org > >> > >> Enrico > >> > >> Il giorno ven 21 set 2018 alle ore 14:45 Enrico Olivelli > >> ha scritto: > >> > > >> > Okay > >> > on github you see two lines, but there is only one job on Jenkins > >> > > >> > see > >> > https://github.com/apache/zookeeper/pull/638 > >> > > >> > @Michael Han please double check > >> > > >> > If all it is okay I will send a new clean summary in another email > >> thread > >> > The short version is: > >> > - an user sends a PR (against master, branch-3.5 and branch-3.4)> CI > >> kicks in an validates, on GitHub you see the status and the results > >> > - an "Apache Committer" (from any project I guess, as I am not a ZK > >> committer but it works, it is important that he is registered in Apache > >> Gihub 'organization') can add a comment with only this phrase 'retest > this > >> please' and the PR will be re-validated from CI > >> > - if the description of the PR contains 'skip ci' validation won't be > >> run from jenkins > >> > > >> > configuration is here: > >> > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ > >> > > >> > > >> > Please tell me if there is any doc/wiki to update and if there is a > >> common place for storing CI Jobs configurations > >> > > >> > > >> > Enrico > >> > > >> > Il giorno ven 21 set 2018 alle ore 11:56 Enrico Olivelli < > >> eolive...@gmail.com> ha scritto: > >> >> > >> >> > >> >> > >> >> Il giorno ven 21 set 2018 alle ore 11:27 Enrico Olivelli < > >> eolive...@gmail.com> ha scritto: > >> >>> > >> >>> I have set up the 'retest this please' trigger. > >> >>> But now we have two jobs running for each pull request. > >> >>> > >> >>> I can't find the flag which makes the validation named 'Jenkins' > start > >> >>> > >> https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr- > build/configure > >> >>> > >> >>> the name of the new one is 'default', and it maps to the same Job > >> >> > >> >> > >> >> Actually the result for the new 'default' validation looks better > >> >> see it here > >> >> https://github.com/apache/zookeeper/pull/637 > >> >> > >> >> The old one just told 'Looks like there's a problem with this pull > >> request' > >> >> the new one reports ' FAILURE 1731 tests run, 1 skipped, 2 failed' > >> >> > >> >> Enrico > >> >> > >> >> > >> >>> > >> >>> is there any other trigger ? > >> >>> I will continue to search > >> >>> > >> >>> Enrico > >> >>> > >> >>> Il giorno ven 21 set 2018 alle ore 09:33 Enrico Olivelli < > >> eolive...@gmail.com> ha scritto: > >> > >> I apologize if there will be some spam on the list > >> > >> When you enable this trigger asfgit starts to add comments like > this: > >> Can one of the admins verify this patch? > >> > >> I am looking for a solution. We had the same on BookKeeper
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/623 retest this please ---
Failed: ZOOKEEPER- PreCommit Build #2230
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2230/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 81.16 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2230//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2230//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2230//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=311F81ECAC5278DFB153A48DF4245369. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 14 minutes 2 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3146 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of bf70f5bc5975cb5948bae2bf4ae8629101efac44 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2230/ and message: 'FAILURE 1734 tests run, 1 skipped, 1 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2230/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.SaslAuthTest.testZKOperationsAfterClientSaslAuthFailure Error Message: Failed to connect to ZooKeeper server. Stack Trace: java.util.concurrent.TimeoutException: Failed to connect to ZooKeeper server. at org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForConnected(ClientBase.java:151) at org.apache.zookeeper.SaslAuthTest.testZKOperationsAfterClientSaslAuthFailure(SaslAuthTest.java:176) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/623 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2230/ ---
[GitHub] zookeeper issue #620: ZOOKEEPER-2847: Cannot bind to client port when reconf...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/620 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2229/ ---
Failed: ZOOKEEPER- PreCommit Build #2229
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2229/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 73.24 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2229//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2229//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2229//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=8A2080912BD094D29F0EA7951B012DD7. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 13 minutes 40 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-2847 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of a2bb8e084801391eec63cf3f1e91b79a5189ce48 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2229/ and message: 'FAILURE 1732 tests run, 1 skipped, 3 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2229/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 3 tests failed. FAILED: org.apache.zookeeper.server.quorum.FuzzySnapshotRelatedTest.testPZxidUpdatedWhenLoadingSnapshot Error Message: KeeperErrorCode = ConnectionLoss for /testPZxidUpdatedDuringTakingSnapshot Stack Trace: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /testPZxidUpdatedDuringTakingSnapshot at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2046) at org.apache.zookeeper.server.quorum.FuzzySnapshotRelatedTest.compareStat(FuzzySnapshotRelatedTest.java:260) at org.apache.zookeeper.server.quorum.FuzzySnapshotRelatedTest.testPZxidUpdatedWhenLoadingSnapshot(FuzzySnapshotRelatedTest.java:235) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) FAILED: org.apache.zookeeper.server.quorum.ReconfigLegacyTest.testReconfigRemoveClientFromStatic Error Message: null Stack Trace: junit.framework.AssertionFailedError at
[GitHub] zookeeper issue #623: [ZOOKEEPER-3146] Limit the maximum client connections ...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/623 retest this please ---
[GitHub] zookeeper issue #620: ZOOKEEPER-2847: Cannot bind to client port when reconf...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/620 retest this please ---
[jira] [Assigned] (ZOOKEEPER-2847) Cannot bind to client port when reconfig based on old static config
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yisong Yue reassigned ZOOKEEPER-2847:
-
Assignee: Yisong Yue
> Cannot bind to client port when reconfig based on old static config
> ---
>
> Key: ZOOKEEPER-2847
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2847
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.5.3, 3.6.0
>Reporter: Fangmin Lv
>Assignee: Yisong Yue
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When started the ensemble with old static config that the server string
> doesn't have client port, dynamically remove and add the same server from the
> ensemble will cause that server cannot bind to client port, and the ZooKeeper
> server cannot serve client requests anymore.
> From the code, we'll set the clientAddr to null when start up with old static
> config, and dynamic config forces to have part, which will
> trigger the following rebind code in QuorumPeer#processReconfig, and cause
> the address already in used issue.
> public boolean processReconfig(QuorumVerifier qv, Long suggestedLeaderId,
> Long zxid, boolean restartLE) {
> ...
> if (myNewQS != null && myNewQS.clientAddr != null
> && !myNewQS.clientAddr.equals(oldClientAddr)) {
> cnxnFactory.reconfigure(myNewQS.clientAddr);
> updateThreadName();
> }
> ...
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
ZooKeeper-trunk-windows-cmake - Build # 2945 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-windows-cmake/2945/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 9.75 KB...] [ivy:retrieve] confs: [javacc] [ivy:retrieve] found net.java.dev.javacc#javacc;5.0 in maven2 [ivy:retrieve] :: resolution report :: resolve 41ms :: artifacts dl 0ms - | |modules|| artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| - | javacc | 1 | 0 | 0 | 0 || 1 | 0 | - [ivy:retrieve] :: retrieving :: org.apache.zookeeper#zookeeper [ivy:retrieve] confs: [javacc] [ivy:retrieve] 1 artifacts copied, 0 already retrieved (291kB/14ms) generate_jute_parser: [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\jute_compiler\org\apache\jute\compiler\generated [ivy:artifactproperty] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:artifactproperty] :: loading settings :: file = f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\ivysettings.xml [move] Moving 1 file to f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\javacc\lib [javacc] Java Compiler Compiler Version 5.0 (Parser Generator) [javacc] (type "javacc" with no arguments for help) [javacc] Reading from file f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-jute\src\main\java\org\apache\jute\compiler\generated\rcc.jj . . . [javacc] File "TokenMgrError.java" does not exist. Will create one. [javacc] File "ParseException.java" does not exist. Will create one. [javacc] File "Token.java" does not exist. Will create one. [javacc] File "SimpleCharStream.java" does not exist. Will create one. [javacc] Parser generated successfully. jute: [javac] Compiling 39 source files to f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\classes [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.8 [javac] 1 warning compile_jute_uptodate: compile_jute: [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\src\java\generated [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-client\zookeeper-client-c\generated [java] ../../../zookeeper-jute/src/main/resources/zookeeper.jute Parsed Successfully [java] ../../../zookeeper-jute/src/main/resources/zookeeper.jute Parsed Successfully [touch] Creating f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\src\java\generated\.generated BUILD SUCCESSFUL Total time: 7 seconds [ZooKeeper-trunk-windows-cmake] $ cmd /c call F:\jenkins\jenkins-slave\temp\jenkins7520606458641278911.bat f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>cd src/c The system cannot find the path specified. f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>cmake . CMake Error: The source directory "F:/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-windows-cmake" does not appear to contain CMakeLists.txt. Specify --help for usage, or press the help button on the CMake GUI. f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>cmake --build . Error: could not load cache f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>exit 1 Build step 'Execute Windows batch command' marked build as failure Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
Re: About Jenkins pre-commit job
Il ven 21 set 2018, 19:11 Michael Han ha scritto: > Thanks for the work, Enrico! Looks like it's working? > > >> if the description of the PR contains 'skip ci' validation won't be > run from jenkins > > Do we need this option? > It is the default config, I can drop. In BK sometimes it is useful in order to save resources for docs/website changes for instance > > >> doc/wiki to update and if there is a common place for storing CI Jobs > configurations > > jenkins job config is stored in the job itself. We don't version control > it separately. > With this update the "Jenkins" section is out of date: > https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute > Would you please update the page to reflect latest state of the jenkins - > github integration? > Ok > > >> It seems that the first line with 'Jenkins' never goes away. > > Seems ok to me to keep two lines. The second line does not cover things > other than test (like find bug checks, release audit checks, etc). > TheI job execution is one, so the informative content is the same: red vs green. But it seems to me that the 'Jenkins' line never goes green nor red, it stays yellow forever Enrico > On Fri, Sep 21, 2018 at 5:51 AM, Enrico Olivelli > wrote: > >> Too early... >> It seems that the first line with 'Jenkins' never goes away. >> >> Patrick, do you have some hint ? >> Otherwise I can ask to infra or bui...@apache.org >> >> Enrico >> >> Il giorno ven 21 set 2018 alle ore 14:45 Enrico Olivelli >> ha scritto: >> > >> > Okay >> > on github you see two lines, but there is only one job on Jenkins >> > >> > see >> > https://github.com/apache/zookeeper/pull/638 >> > >> > @Michael Han please double check >> > >> > If all it is okay I will send a new clean summary in another email >> thread >> > The short version is: >> > - an user sends a PR (against master, branch-3.5 and branch-3.4)> CI >> kicks in an validates, on GitHub you see the status and the results >> > - an "Apache Committer" (from any project I guess, as I am not a ZK >> committer but it works, it is important that he is registered in Apache >> Gihub 'organization') can add a comment with only this phrase 'retest this >> please' and the PR will be re-validated from CI >> > - if the description of the PR contains 'skip ci' validation won't be >> run from jenkins >> > >> > configuration is here: >> > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ >> > >> > >> > Please tell me if there is any doc/wiki to update and if there is a >> common place for storing CI Jobs configurations >> > >> > >> > Enrico >> > >> > Il giorno ven 21 set 2018 alle ore 11:56 Enrico Olivelli < >> eolive...@gmail.com> ha scritto: >> >> >> >> >> >> >> >> Il giorno ven 21 set 2018 alle ore 11:27 Enrico Olivelli < >> eolive...@gmail.com> ha scritto: >> >>> >> >>> I have set up the 'retest this please' trigger. >> >>> But now we have two jobs running for each pull request. >> >>> >> >>> I can't find the flag which makes the validation named 'Jenkins' start >> >>> >> https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/configure >> >>> >> >>> the name of the new one is 'default', and it maps to the same Job >> >> >> >> >> >> Actually the result for the new 'default' validation looks better >> >> see it here >> >> https://github.com/apache/zookeeper/pull/637 >> >> >> >> The old one just told 'Looks like there's a problem with this pull >> request' >> >> the new one reports ' FAILURE 1731 tests run, 1 skipped, 2 failed' >> >> >> >> Enrico >> >> >> >> >> >>> >> >>> is there any other trigger ? >> >>> I will continue to search >> >>> >> >>> Enrico >> >>> >> >>> Il giorno ven 21 set 2018 alle ore 09:33 Enrico Olivelli < >> eolive...@gmail.com> ha scritto: >> >> I apologize if there will be some spam on the list >> >> When you enable this trigger asfgit starts to add comments like this: >> Can one of the admins verify this patch? >> >> I am looking for a solution. We had the same on BookKeeper jobs >> >> Enrico >> >> Il giorno ven 21 set 2018 alle ore 09:31 Enrico Olivelli < >> eolive...@gmail.com> ha scritto: >> > >> > >> > >> > Il giorno ven 21 set 2018 alle ore 03:44 Michael Han < >> h...@apache.org> ha scritto: >> >> >> >> Thanks Enrico for volunteering, it'll be cool if we can fix this. >> > >> > >> > Ok, working now on >> > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ >> > >> > Stay tuned >> > Enrico >> > >> >> >> >> >> >> On Wed, Sep 19, 2018 at 10:42 PM, Enrico Olivelli < >> eolive...@gmail.com> wrote: >> >>> >> >>> Il gio 20 set 2018, 01:17 Michael Han ha >> scritto: >> >>> >> >>> > We've seen lots of confusions recently around Jenkins >> pre-commit check >> >>> > status on github pull request as well as questions like how to >> trigger >> >>> > Jenkins job to get a green build. I added a section
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219023394
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Leader.java ---
@@ -1170,6 +1194,16 @@ synchronized public long
startForwarding(LearnerHandler handler,
return lastProposed;
}
+
+@Override
+public void waitForStartup() throws InterruptedException {
+synchronized(zk){
--- End diff --
nit: space after synchronized(zk)
---
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219581743
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/ObserverMaster.java ---
@@ -0,0 +1,513 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import org.apache.zookeeper.jmx.MBeanRegistry;
+import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.ZKDatabase;
+
+import java.io.BufferedInputStream;
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.SocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.zookeeper.server.quorum.auth.QuorumAuthServer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used by Followers to host Observers. This reduces the network load on
the Leader process by pushing
+ * the responsibility for keeping Observers in sync off the leading peer.
+ *
+ * It is expected that Observers will continue to perform the initial
vetting of clients and requests.
+ * Observers send the request to the follower where it is received by an
ObserverMaster.
+ *
+ * The ObserverMaster forwards a copy of the request to the ensemble
Leader and inserts it into its own
+ * request processor pipeline where it can be matched with the response
comes back. All commits received
+ * from the Leader will be forwarded along to every Learner connected to
the ObserverMaster.
+ *
+ * New Learners connecting to a Follower will receive a LearnerHandler
object and be party to its syncing logic
+ * to be brought up to date.
+ *
+ * The logic is quite a bit simpler than the corresponding logic in Leader
because it only hosts observers.
+ */
+public class ObserverMaster implements LearnerMaster, Runnable {
+private static final Logger LOG =
LoggerFactory.getLogger(ObserverMaster.class);
+
+//Follower counter
+private final AtomicLong followerCounter = new AtomicLong(-1);
+
+private QuorumPeer self;
+private FollowerZooKeeperServer zks;
+private int port;
+
+private Set activeObservers =
Collections.newSetFromMap(new ConcurrentHashMap());
+
+private final ConcurrentHashMap
connectionBeans = new ConcurrentHashMap<>();
+
+/**
+ * we want to keep a log of past txns so that observers can sync up
with us when we connect,
+ * but we can't keep everything in memory, so this limits how much
memory will be dedicated
+ * to keeping recent txns.
+ */
+private final static int PKTS_SIZE_LIMIT = 32 * 1024 * 1024;
+private static volatile int pktsSizeLimit =
Integer.getInteger("zookeeper.observerMaster.sizeLimit", PKTS_SIZE_LIMIT);
+private ConcurrentLinkedQueue proposedPkts = new
ConcurrentLinkedQueue<>();
+private ConcurrentLinkedQueue committedPkts = new
ConcurrentLinkedQueue<>();
+private int pktsSize = 0;
+
+private long lastProposedZxid;
+
+// ensure ordering of revalidations returned to this learner
+private final Object revalidateSessionLock = new Object();
+
+// Throttle when there are too many concurrent snapshots being sent to
observers
+private static final String MAX_CONCURRENT_SNAPSHOTS =
"zookeeper.leader.maxConcurrentSnapshots";
+private static final int maxConcurrentSnapshots;
+
+private static final
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219585427
--- Diff: src/java/test/org/apache/zookeeper/test/ObserverMasterTest.java
---
@@ -0,0 +1,764 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.test;
+
+import static org.apache.zookeeper.test.ClientBase.CONNECTION_TIMEOUT;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.zookeeper.AsyncCallback;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.jmx.CommonNames;
+import org.apache.zookeeper.jmx.MBeanRegistry;
+import org.apache.zookeeper.jmx.ZKMBeanInfo;
+import org.apache.zookeeper.server.admin.Commands;
+import org.apache.zookeeper.server.quorum.DelayRequestProcessor;
+import org.apache.zookeeper.server.quorum.FollowerZooKeeperServer;
+import org.apache.zookeeper.server.quorum.QuorumPeerConfig;
+import org.apache.zookeeper.server.util.PortForwarder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException.ConnectionLossException;
+import org.apache.zookeeper.PortAssignment;
+import org.apache.zookeeper.WatchedEvent;
+import org.apache.zookeeper.Watcher;
+import org.apache.zookeeper.Watcher.Event.KeeperState;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.ZooKeeper.States;
+import org.apache.zookeeper.admin.ZooKeeperAdmin;
+import org.apache.zookeeper.server.quorum.QuorumPeerTestBase;
+import org.junit.Assert;
+import org.junit.Test;
+
+import javax.management.Attribute;
+import javax.management.AttributeNotFoundException;
+import javax.management.InstanceNotFoundException;
+import javax.management.InvalidAttributeValueException;
+import javax.management.MBeanException;
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+import javax.management.ReflectionException;
+import javax.management.RuntimeMBeanException;
+
+@RunWith(Parameterized.class)
+public class ObserverMasterTest extends QuorumPeerTestBase implements
Watcher{
+protected static final Logger LOG =
LoggerFactory.getLogger(ObserverTest.class);
+
+public ObserverMasterTest(Boolean testObserverMaster) {
+this.testObserverMaster = testObserverMaster;
+}
+
+@Parameterized.Parameters
+public static List data() { return Arrays.asList(new Object
[][] {
+{Boolean.TRUE},
+{Boolean.FALSE}});
+}
+
+private Boolean testObserverMaster;
+
+private CountDownLatch latch;
+ZooKeeper zk;
+private WatchedEvent lastEvent = null;
+
+private int CLIENT_PORT_QP1;
+private int CLIENT_PORT_QP2;
+private int CLIENT_PORT_OBS;
+private int OM_PORT;
+private MainThread q1;
+private MainThread q2;
+private MainThread q3;
+
+private PortForwarder setUp(final int omProxyPort) throws IOException {
--- End diff --
I think we should re-use the quorum setup (LaunchServers) in
QuorumPeerMainTest, we have some discussion with @anmolnar in
https://github.com/lvfangmin/zookeeper/pull/1.
---
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219583415
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java
---
@@ -1920,7 +1937,75 @@ private void updateRemotePeerMXBeans(Map newMembers) {
}
}
- private boolean updateLearnerType(QuorumVerifier newQV) {
+private ArrayList observerMasters = new ArrayList<>();
+private void updateObserverMasterList() {
+if (observerMasterPort <= 0) {
+return; // observer masters not enabled
+}
+observerMasters.clear();
+StringBuilder sb = new StringBuilder();
+for (QuorumServer server :
quorumVerifier.getVotingMembers().values()) {
+InetSocketAddress addr = new
InetSocketAddress(server.addr.getAddress(), observerMasterPort);
+observerMasters.add(new QuorumServer(server.id, addr));
+sb.append(addr).append(",");
+}
+LOG.info("Updated learner master list to be {}", sb.toString());
+Collections.shuffle(observerMasters);
+}
+
+private boolean useObserverMasters() {
+return getLearnerType() == LearnerType.OBSERVER &&
observerMasters.size() > 0;
+}
+
+private int nextObserverMaster = 0;
+private QuorumServer nextObserverMaster() {
+if (nextObserverMaster >= observerMasters.size()) {
+nextObserverMaster = 0;
+}
+return observerMasters.get(nextObserverMaster++);
+}
+
+QuorumServer findLearnerMaster(QuorumServer leader) {
+return useObserverMasters() ? nextObserverMaster() : leader;
+}
+
+
--- End diff --
Remove extra empty line here.
---
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219582907
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/ObserverMaster.java ---
@@ -0,0 +1,513 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import org.apache.zookeeper.jmx.MBeanRegistry;
+import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.ZKDatabase;
+
+import java.io.BufferedInputStream;
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.SocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.zookeeper.server.quorum.auth.QuorumAuthServer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used by Followers to host Observers. This reduces the network load on
the Leader process by pushing
+ * the responsibility for keeping Observers in sync off the leading peer.
+ *
+ * It is expected that Observers will continue to perform the initial
vetting of clients and requests.
+ * Observers send the request to the follower where it is received by an
ObserverMaster.
+ *
+ * The ObserverMaster forwards a copy of the request to the ensemble
Leader and inserts it into its own
+ * request processor pipeline where it can be matched with the response
comes back. All commits received
+ * from the Leader will be forwarded along to every Learner connected to
the ObserverMaster.
+ *
+ * New Learners connecting to a Follower will receive a LearnerHandler
object and be party to its syncing logic
+ * to be brought up to date.
+ *
+ * The logic is quite a bit simpler than the corresponding logic in Leader
because it only hosts observers.
+ */
+public class ObserverMaster implements LearnerMaster, Runnable {
+private static final Logger LOG =
LoggerFactory.getLogger(ObserverMaster.class);
+
+//Follower counter
+private final AtomicLong followerCounter = new AtomicLong(-1);
+
+private QuorumPeer self;
+private FollowerZooKeeperServer zks;
+private int port;
+
+private Set activeObservers =
Collections.newSetFromMap(new ConcurrentHashMap());
+
+private final ConcurrentHashMap
connectionBeans = new ConcurrentHashMap<>();
+
+/**
+ * we want to keep a log of past txns so that observers can sync up
with us when we connect,
+ * but we can't keep everything in memory, so this limits how much
memory will be dedicated
+ * to keeping recent txns.
+ */
+private final static int PKTS_SIZE_LIMIT = 32 * 1024 * 1024;
+private static volatile int pktsSizeLimit =
Integer.getInteger("zookeeper.observerMaster.sizeLimit", PKTS_SIZE_LIMIT);
+private ConcurrentLinkedQueue proposedPkts = new
ConcurrentLinkedQueue<>();
+private ConcurrentLinkedQueue committedPkts = new
ConcurrentLinkedQueue<>();
+private int pktsSize = 0;
+
+private long lastProposedZxid;
+
+// ensure ordering of revalidations returned to this learner
+private final Object revalidateSessionLock = new Object();
+
+// Throttle when there are too many concurrent snapshots being sent to
observers
+private static final String MAX_CONCURRENT_SNAPSHOTS =
"zookeeper.leader.maxConcurrentSnapshots";
+private static final int maxConcurrentSnapshots;
+
+private static final
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219312586
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/FollowerZooKeeperServer.java
---
@@ -113,13 +116,17 @@ public void commit(long zxid) {
}
synchronized public void sync(){
-if(pendingSyncs.size() ==0){
+if(pendingSyncs.size() == 0){
--- End diff --
format here as well.
---
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219583775
--- Diff:
src/java/test/org/apache/zookeeper/server/quorum/DelayRequestProcessor.java ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.RequestProcessor;
+
+import java.util.concurrent.LinkedBlockingQueue;
+
+public class DelayRequestProcessor implements RequestProcessor {
--- End diff --
Add a simple description for this class.
---
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219023427
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Leader.java ---
@@ -1170,6 +1194,16 @@ synchronized public long
startForwarding(LearnerHandler handler,
return lastProposed;
}
+
+@Override
+public void waitForStartup() throws InterruptedException {
+synchronized(zk){
+while(!zk.isRunning() &&
!Thread.currentThread().isInterrupted()){
--- End diff --
Add space here as well.
---
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219312930
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Leader.java ---
@@ -1465,4 +1508,107 @@ private boolean isRunning() {
private boolean isParticipant(long sid) {
return
self.getQuorumVerifier().getVotingMembers().containsKey(sid);
}
+
+@Override
+public int getCurrentTick() {
+return self.tick.get();
+}
+
+@Override
+public int syncTimeout() {
+return self.tickTime * self.syncLimit;
+}
+
+@Override
+public int getTickOfNextAckDeadline() {
+return self.tick.get() + self.syncLimit;
+}
+
+@Override
+public int getTickOfInitialAckDeadline() {
+return self.tick.get() + self.initLimit + self.syncLimit;
+}
+
+@Override
+public long getAndDecrementFollowerCounter() {
+return followerCounter.getAndDecrement();
+}
+
+@Override
+public void touch(long sess, int to) {
+zk.touch(sess, to);
+}
+
+@Override
+public void submitLearnerRequest(Request si) {
+zk.submitLearnerRequest(si);
+}
+
+@Override
+public long getQuorumVerifierVersion() {
+return self.getQuorumVerifier().getVersion();
+}
+
+@Override
+public String getPeerInfo(long sid) {
+QuorumPeer.QuorumServer server = self.getView().get(sid);
+return server == null ? "" : server.toString();
+}
+
+@Override
+public byte[] getQuorumVerifierBytes() {
+return self.getLastSeenQuorumVerifier().toString().getBytes();
+}
+
+@Override
+public QuorumAuthServer getQuorumAuthServer() {
+return (self == null) ? null : self.authServer;
+}
+
+@Override
+public void revalidateSession(QuorumPacket qp, LearnerHandler
learnerHandler) throws IOException {
+ByteArrayInputStream bis = new ByteArrayInputStream(qp.getData());
+DataInputStream dis = new DataInputStream(bis);
+long id = dis.readLong();
+int to = dis.readInt();
+ByteArrayOutputStream bos = new ByteArrayOutputStream();
+DataOutputStream dos = new DataOutputStream(bos);
+dos.writeLong(id);
+boolean valid = zk.checkIfValidGlobalSession(id, to);
+if (valid) {
+try {
+//set the session owner
--- End diff --
Add space after //, suggest to put these 3 line comments into a single line.
---
[GitHub] zookeeper pull request #628: ZOOKEEPER-3140: Allow Followers to host Observe...
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/628#discussion_r219313594
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java ---
@@ -246,12 +250,12 @@ protected void connectToLeader(InetSocketAddress
addr, String hostname)
sock.setSoTimeout(self.tickTime * self.initLimit);
int initLimitTime = self.tickTime * self.initLimit;
-int remainingInitLimitTime = initLimitTime;
+int remainingInitLimitTime;
long startNanoTime = nanoTime();
for (int tries = 0; tries < 5; tries++) {
try {
-// recalculate the init limit time because retries sleep
for 1000 milliseconds
+// recalculate the init limit time because retries sleep
for 500 milliseconds
--- End diff --
The comment about sleep time is not accurate anymore.
---
[jira] [Comment Edited] (ZOOKEEPER-3152) Port ZK netty stack to netty 4
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623921#comment-16623921 ] Ilya Maykov edited comment on ZOOKEEPER-3152 at 9/21/18 5:31 PM: - I'm not sure how to split it, since I don't think you can't make different parts (client, server) depend on different versions of Netty. Any suggestions for a splitting strategy are welcome. The change is actually not that big, the patch I have now is ~600 lines added and ~500 lines deleted with 11 files changed. was (Author: ilyam): I'm not sure how to split it, since I don't think you can't make different parts (client, server) depend on different version of Netty. Any suggestions are welcome. The change is actually not that big, the patch I have now is ~600 lines added and ~500 lines deleted with 11 files changed. > Port ZK netty stack to netty 4 > -- > > Key: ZOOKEEPER-3152 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3152 > Project: ZooKeeper > Issue Type: Improvement > Components: java client, server >Affects Versions: 3.6.0 >Reporter: Ilya Maykov >Priority: Minor > > Netty 3 is super old. Let's port ZK's netty stack to netty 4. I'm working on > a patch that I will put up as a pull request on github once we finish testing > it internally at Facebook, just getting the Jira ticket ready ahead of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3152) Port ZK netty stack to netty 4
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623921#comment-16623921 ] Ilya Maykov commented on ZOOKEEPER-3152: I'm not sure how to split it, since I don't think you can't make different parts (client, server) depend on different version of Netty. Any suggestions are welcome. The change is actually not that big, the patch I have now is ~600 lines added and ~500 lines deleted with 11 files changed. > Port ZK netty stack to netty 4 > -- > > Key: ZOOKEEPER-3152 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3152 > Project: ZooKeeper > Issue Type: Improvement > Components: java client, server >Affects Versions: 3.6.0 >Reporter: Ilya Maykov >Priority: Minor > > Netty 3 is super old. Let's port ZK's netty stack to netty 4. I'm working on > a patch that I will put up as a pull request on github once we finish testing > it internally at Facebook, just getting the Jira ticket ready ahead of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: About Jenkins pre-commit job
Thanks for the work, Enrico! Looks like it's working? >> if the description of the PR contains 'skip ci' validation won't be run from jenkins Do we need this option? >> doc/wiki to update and if there is a common place for storing CI Jobs configurations jenkins job config is stored in the job itself. We don't version control it separately. With this update the "Jenkins" section is out of date: https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute Would you please update the page to reflect latest state of the jenkins - github integration? >> It seems that the first line with 'Jenkins' never goes away. Seems ok to me to keep two lines. The second line does not cover things other than test (like find bug checks, release audit checks, etc). On Fri, Sep 21, 2018 at 5:51 AM, Enrico Olivelli wrote: > Too early... > It seems that the first line with 'Jenkins' never goes away. > > Patrick, do you have some hint ? > Otherwise I can ask to infra or bui...@apache.org > > Enrico > > Il giorno ven 21 set 2018 alle ore 14:45 Enrico Olivelli > ha scritto: > > > > Okay > > on github you see two lines, but there is only one job on Jenkins > > > > see > > https://github.com/apache/zookeeper/pull/638 > > > > @Michael Han please double check > > > > If all it is okay I will send a new clean summary in another email thread > > The short version is: > > - an user sends a PR (against master, branch-3.5 and branch-3.4)> CI > kicks in an validates, on GitHub you see the status and the results > > - an "Apache Committer" (from any project I guess, as I am not a ZK > committer but it works, it is important that he is registered in Apache > Gihub 'organization') can add a comment with only this phrase 'retest this > please' and the PR will be re-validated from CI > > - if the description of the PR contains 'skip ci' validation won't be > run from jenkins > > > > configuration is here: > > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ > > > > > > Please tell me if there is any doc/wiki to update and if there is a > common place for storing CI Jobs configurations > > > > > > Enrico > > > > Il giorno ven 21 set 2018 alle ore 11:56 Enrico Olivelli < > eolive...@gmail.com> ha scritto: > >> > >> > >> > >> Il giorno ven 21 set 2018 alle ore 11:27 Enrico Olivelli < > eolive...@gmail.com> ha scritto: > >>> > >>> I have set up the 'retest this please' trigger. > >>> But now we have two jobs running for each pull request. > >>> > >>> I can't find the flag which makes the validation named 'Jenkins' start > >>> https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr- > build/configure > >>> > >>> the name of the new one is 'default', and it maps to the same Job > >> > >> > >> Actually the result for the new 'default' validation looks better > >> see it here > >> https://github.com/apache/zookeeper/pull/637 > >> > >> The old one just told 'Looks like there's a problem with this pull > request' > >> the new one reports ' FAILURE 1731 tests run, 1 skipped, 2 failed' > >> > >> Enrico > >> > >> > >>> > >>> is there any other trigger ? > >>> I will continue to search > >>> > >>> Enrico > >>> > >>> Il giorno ven 21 set 2018 alle ore 09:33 Enrico Olivelli < > eolive...@gmail.com> ha scritto: > > I apologize if there will be some spam on the list > > When you enable this trigger asfgit starts to add comments like this: > Can one of the admins verify this patch? > > I am looking for a solution. We had the same on BookKeeper jobs > > Enrico > > Il giorno ven 21 set 2018 alle ore 09:31 Enrico Olivelli < > eolive...@gmail.com> ha scritto: > > > > > > > > Il giorno ven 21 set 2018 alle ore 03:44 Michael Han < > h...@apache.org> ha scritto: > >> > >> Thanks Enrico for volunteering, it'll be cool if we can fix this. > > > > > > Ok, working now on > > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ > > > > Stay tuned > > Enrico > > > >> > >> > >> On Wed, Sep 19, 2018 at 10:42 PM, Enrico Olivelli < > eolive...@gmail.com> wrote: > >>> > >>> Il gio 20 set 2018, 01:17 Michael Han ha > scritto: > >>> > >>> > We've seen lots of confusions recently around Jenkins pre-commit > check > >>> > status on github pull request as well as questions like how to > trigger > >>> > Jenkins job to get a green build. I added a section "Jenkins > Pre-commit > >>> > Check" to our wiki page about this matter: > >>> > > >>> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ > HowToContribute > >>> > > >>> > I suspect the permission issue is about the fact that apache > zookeeper > >>> > github is read only for most of us including committers. Only > asfbot has > >>> > write access. When a Jenkins job is triggered through "Rebuild" > button the > >>> > credential it's using is not asfbot thus the final job result > will fail to > >>>
[GitHub] zookeeper issue #590: [ZOOKEEPER-1177] Add the memory optimized watch manage...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/590 @anmolnar I'll sign this off by end of next Monday if no other issues. @lvfangmin great work and thanks for your patience! ---
[GitHub] zookeeper pull request #590: [ZOOKEEPER-1177] Add the memory optimized watch...
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/590#discussion_r219562472 --- Diff: src/test/java/bench/org/apache/zookeeper/BenchMain.java --- @@ -0,0 +1,12 @@ +package org.apache.zookeeper; --- End diff -- This file is missing apache license header. This triggers a -1 in last jenkins build. ---
Failed: ZOOKEEPER- PreCommit Build #2228
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2228/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 82.40 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2228//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2228//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2228//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=96248F21C1CE783507A69D0C16F8F891. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 18 minutes 25 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-2122 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of 121eba758b9f81d758f5092a07977a0de7243c20 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2228/ and message: 'FAILURE 1731 tests run, 1 skipped, 1 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2228/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.StandaloneDisabledTest.startSingleServerTest Error Message: test timed out after 60 milliseconds Stack Trace: org.junit.runners.model.TestTimedOutException: test timed out after 60 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1524) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1507) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1453) at org.apache.zookeeper.test.ReconfigTest.createZNode(ReconfigTest.java:192) at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:165) at org.apache.zookeeper.server.quorum.StandaloneDisabledTest.testReconfig(StandaloneDisabledTest.java:243) at org.apache.zookeeper.server.quorum.StandaloneDisabledTest.startSingleServerTest(StandaloneDisabledTest.java:113) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)
[GitHub] zookeeper issue #639: ZOOKEEPER-2122: add openssl to zookeeper c client to s...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/639 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2228/ ---
Re: Java version (Oracle - OpenJDK) support
Just throwing in a nice read :) "Java is still free" https://docs.google.com/document/d/1nFGazvrCvHMZJgFstlbzoHjpAVwv5DEdnaBr_5pKuHo/edit On Fri, Sep 14, 2018 at 6:38 PM Andor Molnar wrote: > > > > On 2018. Sep 13., at 19:29, Patrick Hunt wrote: > > > > On Thu, Sep 13, 2018 at 1:52 AM Andor Molnar an...@apache.org>> wrote: > > > >> Hi team, > >> > >> Let me quickly summarise what we’re trying to resolve on this thread. > >> > >> Problem #1: due to Oracle licensing changes people are expected to move > >> away from Oracle Java and support of other “open” Java implementations > will > >> become important for ZooKeeper. > >> > >> Problem #2: Java 8 support will end in September 2018, we have to add > >> support for Java 11 (LTS). Java 9 and 10 support are nice-to-haves, > because > >> those are not LTS releases. > >> > >> branch-3.4 (stable version): > >> From Jenkins this is currently the most stable version (kudos for the > hard > >> work with flaky tests). > >> Problem: Java 11 build is failing due to Kerberos tests. > >> Solution: Backport Kerberos tests from 3.5 > >> > >> I think there’s no need to change Java support on branch-3.4 or it’s not > >> scope of this conversation. > >> > >> branch-3.5 (upcoming stable version): > >> Problem: there’s no Java 11 build currently > >> Solution: create new Jenkins job > >> > >> branch-3.6 (master): > >> Problem: Java 11 build is constantly failing > >> Solution: no solution provided yet > >> > >> A few more thoughts: > >> - We don’t necessarily need to add Java 11 support for branch-3.4. I > think > >> it would be better to push people towards upgrading to 3.5 and focus on > >> making it stable as soon as possible. (See the other thread for details > - > >> we’re getting close) > >> > > > > Until the downstream projects (those using ZK) update to post-3.4 many > > users will be forced to keep 3.4. Given 3.4 is currently the stable > branch > > and it's unknown when 3.5 will reach the same level of maturity, and > Oracle > > forcing the issue re JDK licensing, I don't think this will be feasible > for > > many users. > > > > Patrick > > Understood. > Enrico already submitted a patch to make Kerberos testing JDK11 compatible. > We’ll be in a good shape with that for 3.4 > > Andor > > > > > > > > > > > > >> - We should add more pressure on the testing side of 3.5: there’re only > 3 > >> Jenkins job currently running on branch-3.5. Let’s add Oracle Java 11 > >> build, OpenJDK 8,9,10,11 whatever you think makes sense. > >> > >> Regards, > >> Andor > >> > >> > >> > >> > >> > >>> On 2018. Sep 13., at 8:38, Enrico Olivelli > wrote: > >>> > >>> Il gio 13 set 2018, 01:00 Patrick Hunt >> ph...@apache.org>> ha scritto: > >>> > Historically we've always defined a minimum version and let users > >> decide. > That seems to have worked pretty well. It provides the most > flexibility > >> and > hasn't really bothered us too much. It limits our use of new language > features of Java, but typically Java is ensuring b/w compat from a > >> runtime > perspective and as a result there's isn't much of a burden to say we > support 6 and laters vs alternately saying we support 6&8 exclusively. > > >>> > >>> The problem is only about a test case. We can copy the test case 3.5 > >> branch > >>> which is using Kerby and run that one in case of java >= 11 using junit > >>> 'assumptions'. > >>> > >>> Enrico > >>> > >>> > Patrick > > On Wed, Sep 12, 2018 at 7:16 AM Enrico Olivelli > wrote: > > > Il mer 12 set 2018, 11:37 Norbert Kalmar > >>> > ha > > scritto: > > > >> Thanks Enrico! > >> > >> Agree, as I mentioned, only JDK8 and 11 should be tested on 3.4 and > other > >> branches as well. > >> > >> I checked the Kerby problems, 3.4 does not have Kerby, it uses > Apache > >> directory server. > >> Maybe we should introduce also introduce Kerby on 3.4? Or just try > to > fix > >> the problems with directory server. > >> > > > > Directory Server supports jdk6 and Kerby not. > > If wr drop JDK6 we can switch to Kerby > > > > Enrico > > > > > > > >> Regards, > >> Norbert > >> > >> On Wed, Sep 12, 2018 at 11:22 AM Enrico Olivelli < > eolive...@gmail.com > >>> > >> wrote: > >> > >>> Il giorno mer 12 set 2018 alle ore 11:04 Norbert Kalmar > >>> ha scritto: > >>> > Hi all, > > Oracle8 will have it's support end in January. They changed there > >> release > drastically. > A good article on that: > https://dev.karakun.com/java/2018/06/25/java-releases.html > > Long story short: From January, no Oracle JDK version can be used > in > >> PROD > environment without a license. End every release, even LTS (next > one > > is > version 11) will only have a 6 month public update period. > >
ZooKeeper_branch34_openjdk7 - Build # 2062 - Failure
See https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/2062/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 41.34 KB...] [junit] Running org.apache.zookeeper.test.SaslAuthFailDesignatedClientTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.179 sec [junit] Running org.apache.zookeeper.test.SaslAuthFailNotifyTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.632 sec [junit] Running org.apache.zookeeper.test.SaslAuthFailTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.717 sec [junit] Running org.apache.zookeeper.test.SaslAuthMissingClientConfigTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.599 sec [junit] Running org.apache.zookeeper.test.SaslClientTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec [junit] Running org.apache.zookeeper.test.SessionInvalidationTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.593 sec [junit] Running org.apache.zookeeper.test.SessionTest [junit] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.545 sec [junit] Running org.apache.zookeeper.test.SessionTimeoutTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.76 sec [junit] Running org.apache.zookeeper.test.StandaloneTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.889 sec [junit] Running org.apache.zookeeper.test.StatTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.701 sec [junit] Running org.apache.zookeeper.test.StaticHostProviderTest [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.652 sec [junit] Running org.apache.zookeeper.test.SyncCallTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.757 sec [junit] Running org.apache.zookeeper.test.TruncateTest [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.376 sec [junit] Running org.apache.zookeeper.test.UpgradeTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.748 sec [junit] Running org.apache.zookeeper.test.WatchedEventTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.093 sec [junit] Running org.apache.zookeeper.test.WatcherFuncTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.123 sec [junit] Running org.apache.zookeeper.test.WatcherTest [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.852 sec [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.328 sec [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.72 sec [junit] Running org.apache.jute.BinaryInputArchiveTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.091 sec fail.build.on.test.failure: BUILD FAILED /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build.xml:1418: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build.xml:1421: Tests failed! Total time: 32 minutes 13 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Recording test results Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalFollowerRunWithDiff Error Message: null Stack Trace: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:588) at
[GitHub] zookeeper issue #639: ZOOKEEPER-2122: add openssl to zookeeper c client to s...
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/639 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2227/ ---
Failed: ZOOKEEPER- PreCommit Build #2227
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2227/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 80.12 MB...] [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2227//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2227//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2227//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=C134930D439BCB1AD18AD959200CE542. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 12 minutes 50 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-2122 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of 9a06457cf9f543bdd005fe5565f4a79c980ff3ea to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2227/ and message: 'FAILURE 1731 tests run, 1 skipped, 0 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2227/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
[GitHub] zookeeper pull request #639: ZOOKEEPER-2122: add openssl to zookeeper c clie...
GitHub user roodkcab opened a pull request: https://github.com/apache/zookeeper/pull/639 ZOOKEEPER-2122: add openssl to zookeeper c client to support ssl I have some problem with cmake, I was intend to find openssl as dependency using pkg_search_module, but it didn't work. so I have to set the include and lib directory in /opt/local, can anyone help me on this ? thanks. I made a wrong pull request to branch-3.5 #625, this one is a correction. You can merge this pull request into a Git repository by running: $ git pull https://github.com/roodkcab/zookeeper master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/639.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #639 commit 7ecbcbc14f99a43f1c7b226e05acc171f72e1951 Author: buaacss Date: 2018-09-17T16:31:49Z refine zhandle fd commit cedd176b0189a4a4ddf5252dcb4435011ec4e8dd Author: éç¡å® Date: 2018-09-19T09:55:51Z refine cli.c opt to getopt commit 9a06457cf9f543bdd005fe5565f4a79c980ff3ea Author: buaacss Date: 2018-09-21T14:40:13Z refine with getopt ---
[GitHub] zookeeper pull request #625: ZOOKEEPER-2122: add openssl to zookeeper c clie...
Github user roodkcab closed the pull request at: https://github.com/apache/zookeeper/pull/625 ---
[jira] [Commented] (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets
[ https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623686#comment-16623686 ] Yun Qin commented on ZOOKEEPER-900: --- I believe we have met an issue very similar to what's described in section b) of the original bug description. In our case, sid 1/2 weren't able to rejoin a quorum consisting of sid 3/4/5. The root cause was that QuorumCnxManager$Listener wasn't able to accept connection from sid 1/2 on sid 3/5. Restarting ZooKeeper on sid 3/5 solved the issue. I would really appreciate it if we can proceed with the fix, now that we have a preliminary patch and people willing to take over. Thanks. > FLE implementation should be improved to use non-blocking sockets > - > > Key: ZOOKEEPER-900 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900 > Project: ZooKeeper > Issue Type: Bug >Reporter: Vishal Kher >Assignee: Martin Kuchta >Priority: Critical > Fix For: 3.6.0, 3.5.5 > > Attachments: ZOOKEEPER-900-part2.patch, ZOOKEEPER-900.patch, > ZOOKEEPER-900.patch1, ZOOKEEPER-900.patch2 > > > From earlier email exchanges: > 1. Blocking connects and accepts: > a) The first problem is in manager.toSend(). This invokes connectOne(), which > does a blocking connect. While testing, I changed the code so that > connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() > does a socketChannel.connect(). After starting AsyncConnect, connectOne > starts a timer. connectOne continues with normal operations if the connection > is established before the timer expires, otherwise, when the timer expires it > interrupts AsyncConnect() thread and returns. In this way, I can have an > upper bound on the amount of time we need to wait for connect to succeed. Of > course, this was a quick fix for my testing. Ideally, we should use Selector > to do non-blocking connects/accepts. I am planning to do that later once we > at least have a quick fix for the problem and consensus from others for the > real fix (this problem is big blocker for us). Note that it is OK to do > blocking IO in SenderWorker and RecvWorker threads since they block IO to the > respective peer. > b) The blocking IO problem is not just restricted to connectOne(), but also > in receiveConnection(). The Listener thread calls receiveConnection() for > each incoming connection request. receiveConnection does blocking IO to get > peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the > peer that had sent the connection request. All of this is happening from the > Listener. In short, if a peer fails after initiating a connection, the > Listener thread won't be able to accept connections from other peers, because > it would be stuck in read() or connetOne(). Also the code has an inherent > cycle. initiateConnection() and receiveConnection() will have to be very > carefully synchronized otherwise, we could run into deadlocks. This code is > going to be difficult to maintain/modify. > Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zookeeper issue #638: This is only a test, do not merge! trial 3
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/638 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2226/ ---
Failed: ZOOKEEPER- PreCommit Build #2226
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2226/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 80.17 MB...] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2226//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2226//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2226//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=9BBDF13E485A8F5A7926706447575B1D. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: ‘/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess’ and ‘/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess’ are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1842: exec returned: 1 Total time: 15 minutes 34 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Could not determine description. Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of 2ece3ed1604c8e4ad0595dc6130d970e8f747a17 to FAILURE with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2226/ and message: 'FAILURE 1731 tests run, 1 skipped, 1 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2226/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentServersAreObserversInNextConfig Error Message: waiting for server 3 being up Stack Trace: junit.framework.AssertionFailedError: waiting for server 3 being up at org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentServersAreObserversInNextConfig(ReconfigRecoveryTest.java:224) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Re: About Jenkins pre-commit job
Too early... It seems that the first line with 'Jenkins' never goes away. Patrick, do you have some hint ? Otherwise I can ask to infra or bui...@apache.org Enrico Il giorno ven 21 set 2018 alle ore 14:45 Enrico Olivelli ha scritto: > > Okay > on github you see two lines, but there is only one job on Jenkins > > see > https://github.com/apache/zookeeper/pull/638 > > @Michael Han please double check > > If all it is okay I will send a new clean summary in another email thread > The short version is: > - an user sends a PR (against master, branch-3.5 and branch-3.4)> CI kicks in > an validates, on GitHub you see the status and the results > - an "Apache Committer" (from any project I guess, as I am not a ZK committer > but it works, it is important that he is registered in Apache Gihub > 'organization') can add a comment with only this phrase 'retest this please' > and the PR will be re-validated from CI > - if the description of the PR contains 'skip ci' validation won't be run > from jenkins > > configuration is here: > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ > > > Please tell me if there is any doc/wiki to update and if there is a common > place for storing CI Jobs configurations > > > Enrico > > Il giorno ven 21 set 2018 alle ore 11:56 Enrico Olivelli > ha scritto: >> >> >> >> Il giorno ven 21 set 2018 alle ore 11:27 Enrico Olivelli >> ha scritto: >>> >>> I have set up the 'retest this please' trigger. >>> But now we have two jobs running for each pull request. >>> >>> I can't find the flag which makes the validation named 'Jenkins' start >>> https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/configure >>> >>> the name of the new one is 'default', and it maps to the same Job >> >> >> Actually the result for the new 'default' validation looks better >> see it here >> https://github.com/apache/zookeeper/pull/637 >> >> The old one just told 'Looks like there's a problem with this pull request' >> the new one reports ' FAILURE 1731 tests run, 1 skipped, 2 failed' >> >> Enrico >> >> >>> >>> is there any other trigger ? >>> I will continue to search >>> >>> Enrico >>> >>> Il giorno ven 21 set 2018 alle ore 09:33 Enrico Olivelli >>> ha scritto: I apologize if there will be some spam on the list When you enable this trigger asfgit starts to add comments like this: Can one of the admins verify this patch? I am looking for a solution. We had the same on BookKeeper jobs Enrico Il giorno ven 21 set 2018 alle ore 09:31 Enrico Olivelli ha scritto: > > > > Il giorno ven 21 set 2018 alle ore 03:44 Michael Han ha > scritto: >> >> Thanks Enrico for volunteering, it'll be cool if we can fix this. > > > Ok, working now on > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ > > Stay tuned > Enrico > >> >> >> On Wed, Sep 19, 2018 at 10:42 PM, Enrico Olivelli >> wrote: >>> >>> Il gio 20 set 2018, 01:17 Michael Han ha scritto: >>> >>> > We've seen lots of confusions recently around Jenkins pre-commit check >>> > status on github pull request as well as questions like how to trigger >>> > Jenkins job to get a green build. I added a section "Jenkins >>> > Pre-commit >>> > Check" to our wiki page about this matter: >>> > >>> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute >>> > >>> > I suspect the permission issue is about the fact that apache zookeeper >>> > github is read only for most of us including committers. Only asfbot >>> > has >>> > write access. When a Jenkins job is triggered through "Rebuild" >>> > button the >>> > credential it's using is not asfbot thus the final job result will >>> > fail to >>> > write back to github. I don't know how to solve this, but hopefully >>> > we have >>> > workarounds (which though require PR creator to be more diligent on >>> > obtaining a green build). >>> > >>> >>> In Bookkeeper we have enabled the magic 'retest this please' and it is >>> very >>> useful. >>> >>> I can take a look in order to back port to ZK, I have access to ASF >>> Jenkins >>> so I can check current configuration for ZK. >>> >>> In Bookkeeper we also have the configuration of Jenkins jobs on git, >>> using >>> Jenkins DSL language. I can also take a look and see how to port to ZK >>> >>> >>> >>> Enrico >>> >>> >>> > Michael. >>> > >>> -- >>> >>> >>> -- Enrico Olivelli >> >>
[GitHub] zookeeper issue #638: This is only a test, do not merge! trial 3
Github user eolivelli commented on the issue: https://github.com/apache/zookeeper/pull/638 retest this please ---
Success: ZOOKEEPER- PreCommit Build #2225
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2225/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 82.88 MB...] [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2225//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2225//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2225//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=7D44B2C1803FB30FBF48FEB0CDEB515E. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 24 minutes 7 seconds Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Could not determine description. Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Adding one-line test results to commit status... Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting status of 2ece3ed1604c8e4ad0595dc6130d970e8f747a17 to SUCCESS with url https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2225/ and message: 'SUCCESS 1731 tests run, 1 skipped, 0 failed.' Using context: default Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2225/ Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
[GitHub] zookeeper issue #638: This is only a test, do not merge! trial 3
Github user asfgit commented on the issue: https://github.com/apache/zookeeper/pull/638 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/2225/ ---
Re: About Jenkins pre-commit job
Okay on github you see two lines, but there is only one job on Jenkins see https://github.com/apache/zookeeper/pull/638 @Michael Han please double check If all it is okay I will send a new clean summary in another email thread The short version is: - an user sends a PR (against master, branch-3.5 and branch-3.4)> CI kicks in an validates, on GitHub you see the status and the results - an "Apache Committer" (from any project I guess, as I am not a ZK committer but it works, it is important that he is registered in Apache Gihub 'organization') can add a comment with only this phrase 'retest this please' and the PR will be re-validated from CI - if the description of the PR contains 'skip ci' validation won't be run from jenkins configuration is here: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ Please tell me if there is any doc/wiki to update and if there is a common place for storing CI Jobs configurations Enrico Il giorno ven 21 set 2018 alle ore 11:56 Enrico Olivelli < eolive...@gmail.com> ha scritto: > > > Il giorno ven 21 set 2018 alle ore 11:27 Enrico Olivelli < > eolive...@gmail.com> ha scritto: > >> I have set up the 'retest this please' trigger. >> But now we have two jobs running for each pull request. >> >> I can't find the flag which makes the validation named 'Jenkins' start >> >> https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/configure >> >> the name of the new one is 'default', and it maps to the same Job >> > > Actually the result for the new 'default' validation looks better > see it here > https://github.com/apache/zookeeper/pull/637 > > The old one just told 'Looks like there's a problem with this pull request' > the new one reports ' FAILURE 1731 tests run, 1 skipped, 2 failed' > > Enrico > > > >> is there any other trigger ? >> I will continue to search >> >> Enrico >> >> Il giorno ven 21 set 2018 alle ore 09:33 Enrico Olivelli < >> eolive...@gmail.com> ha scritto: >> >>> I apologize if there will be some spam on the list >>> >>> When you enable this trigger asfgit starts to add comments like this: >>> Can one of the admins verify this patch? >>> >>> I am looking for a solution. We had the same on BookKeeper jobs >>> >>> Enrico >>> >>> Il giorno ven 21 set 2018 alle ore 09:31 Enrico Olivelli < >>> eolive...@gmail.com> ha scritto: >>> Il giorno ven 21 set 2018 alle ore 03:44 Michael Han ha scritto: > Thanks Enrico for volunteering, it'll be cool if we can fix this. > Ok, working now on https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/ Stay tuned Enrico > > On Wed, Sep 19, 2018 at 10:42 PM, Enrico Olivelli > wrote: > >> Il gio 20 set 2018, 01:17 Michael Han ha scritto: >> >> > We've seen lots of confusions recently around Jenkins pre-commit >> check >> > status on github pull request as well as questions like how to >> trigger >> > Jenkins job to get a green build. I added a section "Jenkins >> Pre-commit >> > Check" to our wiki page about this matter: >> > >> > >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute >> > >> > I suspect the permission issue is about the fact that apache >> zookeeper >> > github is read only for most of us including committers. Only >> asfbot has >> > write access. When a Jenkins job is triggered through "Rebuild" >> button the >> > credential it's using is not asfbot thus the final job result will >> fail to >> > write back to github. I don't know how to solve this, but hopefully >> we have >> > workarounds (which though require PR creator to be more diligent on >> > obtaining a green build). >> > >> >> In Bookkeeper we have enabled the magic 'retest this please' and it >> is very >> useful. >> >> I can take a look in order to back port to ZK, I have access to ASF >> Jenkins >> so I can check current configuration for ZK. >> >> In Bookkeeper we also have the configuration of Jenkins jobs on git, >> using >> Jenkins DSL language. I can also take a look and see how to port to ZK >> >> >> >> Enrico >> >> >> > Michael. >> > >> -- >> >> >> -- Enrico Olivelli >> > >
[GitHub] zookeeper pull request #638: This is only a test, do not merge! trial 3
GitHub user eolivelli opened a pull request: https://github.com/apache/zookeeper/pull/638 This is only a test, do not merge! trial 3 You can merge this pull request into a Git repository by running: $ git pull https://github.com/eolivelli/zookeeper fix/test-ci2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/638.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #638 commit 2ece3ed1604c8e4ad0595dc6130d970e8f747a17 Author: Enrico Olivelli - Diennea Date: 2018-09-21T09:17:41Z This is only a test, do not merge! ---
