Re: Interesting FB article on CM at scale; LAD
>> Are the patches/improvements in it going to be open sources and possibly backported to ZooKeeper I'll let folks from FB speak but recent activities of incoming patches is a positive sign. Also I hope ZK would still be used in prod environment at FB so it can got hammered and bugs got exposed. For infrastructure software, there is no better way of testing than testing in prod. The LAD paper does not mention ZK's role in the new system - I suspect it's still there but not on critical path (maybe just for config change notification?). >> by introducing a plugin-based data store replacing current ZkDatabase. I don't think this will make any substantial difference w.r.t performance, if we need keep same client side semantics and API compatibility. Regardless of which state machine implementation it's using, the serving layer (the processor pipelines) is under same ordering constraints. >> So maybe it was originally a use case problem which eventually escalated to a new system design. I feel the same way. Those who use ZK only for leader election and small metadata storage / change notification are very happy. Those who use ZK on their performance critical path all ends up with moving ZK out of the critical path, and questioning the "high availability" part of ZK . On Fri, Jul 20, 2018 at 1:49 AM, Andor Molnar wrote: > Great article, thanks Pat. > > There're 2 things which I found particularly interesting: > > - Is Zeus going to be decod as part of the migration? Are the > patches/improvements in it going to be open sources and possibly backported > to ZooKeeper? > > - "Independent data store: ZooKeeper couples a data store with its > distribution framework." > > I believe that decoupling the data store from the coordination framework > wouldn't be too much of a hassle in ZooKeeper by introducing a plugin-based > data store replacing current ZkDatabase. In addition to that I wonder if we > could make client libraries "datastore-aware" letting them serve read > requests directly from the database. e.g. from HDFS data nodes. It might > successfully decouple the load generated by large number of read requests > from writes. > > In which case will ZooKeeper be in any aspects different from an HDFS > cluster? It adds strong consistency and ordering guarantees. > Which parts of ZooKeeper do we still need after such change, because the > distributed data store guarantees a consistent view of the data tree across > participants so what would be the gain of dealing with consensus? > > Going back to Fb's original problem in terms of data/metadata decoupling: > they wanted to distribute large files like 100MB whereas they already 5MB > files present in Zk. I think 5MB is already too much data in a single znode > and ZooKeeper is not designed to deal with such load of data distribution. > So maybe it was originally a use case problem which eventually escalated to > a new system design. > > Propagating metadata updates to millions of subscribers sounds to me more > of an IoT / pubsub problem. Which Facebook has presented a very promising > answer to. Using a tree of subscribers combined with sharding has very > strong scalability potential. > > Dealing with the distribution of large files triggered by metadata change: > bittorrent? :) > > Regards, > Andor > > > > > On Fri, Jul 20, 2018 at 8:07 AM, Patrick Hunt wrote: > > > "Location-Aware Distribution (LAD), a new peer-to-peer system that > handles > > the distribution of configuration changes to millions of servers." > > https://code.fb.com/data-infrastructure/location-aware- > > distribution-configuring-servers-at-scale/ > > > > Previously Facebook’s configuration management system (called > Configerator) > > used ZooKeeper. SOSP 2015 paper: > > https://research.fb.com/publications/holistic- > configuration-management-at- > > facebook/ > > > > Patrick > > >
[GitHub] zookeeper issue #447: [ZOOKEEPER-2926] Fix potential data consistency issue ...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/447 I'll give it a review; I missed the original JIRA. ---
[jira] [Assigned] (ZOOKEEPER-3098) Add additional server metrics
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Han reassigned ZOOKEEPER-3098:
--
Assignee: Joseph Blomstedt
> Add additional server metrics
> -
>
> Key: ZOOKEEPER-3098
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3098
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
>Affects Versions: 3.6.0
>Reporter: Joseph Blomstedt
>Assignee: Joseph Blomstedt
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This patch adds several new server-side metrics as well as makes it easier to
> add new metrics in the future. This patch also includes a handful of other
> minor metrics-related changes.
> Here's a high-level summary of the changes.
> # This patch extends the request latency tracked in {{ServerStats}} to track
> {{read}} and {{update}} latency separately. Updates are any request that must
> be voted on and can change data, reads are all requests that can be handled
> locally and don't change data.
> # This patch adds the {{ServerMetrics}} logic and the related
> {{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
> make it incredibly easy to add new metrics. To add a new metric you just add
> one line to {{ServerMetrics}} and then directly reference that new metric
> anywhere in the code base. The {{ServerMetrics}} logic handles creating the
> metric, properly adding the metric to the JSON output of the {{/monitor}}
> admin command, and properly resetting the metric when necessary. The
> motivation behind {{ServerMetrics}} is to make things easy enough that it
> encourages new metrics to be added liberally. Lack of in-depth
> metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most
> of our internal changes build on {{ServerMetrics}} and we have nearly 100
> internal metrics at this time – all of which we'll be upstreaming in the
> coming months as we publish more internal patches.
> # This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
> # This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
> atomic operations.
> Here's a list of new metrics added in this patch:
> - {{uptime}}: time that a peer has been in a stable
> leading/following/observing state
> - {{leader_uptime}}: uptime for peer in leading state
> - {{global_sessions}}: count of global sessions
> - {{local_sessions}}: count of local sessions
> - {{quorum_size}}: configured ensemble size
> - {{synced_observers}}: similar to existing `synced_followers` but for
> observers
> - {{fsynctime}}: time to fsync transaction log (avg/min/max)
> - {{snapshottime}}: time to write a snapshot (avg/min/max)
> - {{dbinittime}}: time to reload database – read snapshot + apply
> transactions (avg/min/max)
> - {{readlatency}}: read request latency (avg/min/max)
> - {{updatelatency}}: update request latency (avg/min/max)
> - {{propagation_latency}}: end-to-end latency for updates, from proposal on
> leader to committed-to-datatree on a given host (avg/min/max)
> - {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
> - {{election_time}}: time between entering and leaving election (avg/min/max)
> - {{looking_count}}: number of transitions into looking state
> - {{diff_count}}: number of diff syncs performed
> - {{snap_count}}: number of snap syncs performed
> - {{commit_count}}: number of commits performed on leader
> - {{connection_request_count}}: number of incoming client connection requests
> - {{bytes_received_count}}: similar to existing `packets_received` but
> tracks bytes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3098) Add additional server metrics
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551364#comment-16551364
]
Michael Han commented on ZOOKEEPER-3098:
[~eolivelli] My thought is we still need a metric interface to hook with
external reporters - and also we need metrics type definition more than counter
which is the only type presented in the patch. ZOOKEEPER-3092 is more about the
general metrics framework infrastructure and the work in this Jira is more
about actual instrumentation and metrics collection.
> Add additional server metrics
> -
>
> Key: ZOOKEEPER-3098
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3098
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
>Affects Versions: 3.6.0
>Reporter: Joseph Blomstedt
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This patch adds several new server-side metrics as well as makes it easier to
> add new metrics in the future. This patch also includes a handful of other
> minor metrics-related changes.
> Here's a high-level summary of the changes.
> # This patch extends the request latency tracked in {{ServerStats}} to track
> {{read}} and {{update}} latency separately. Updates are any request that must
> be voted on and can change data, reads are all requests that can be handled
> locally and don't change data.
> # This patch adds the {{ServerMetrics}} logic and the related
> {{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
> make it incredibly easy to add new metrics. To add a new metric you just add
> one line to {{ServerMetrics}} and then directly reference that new metric
> anywhere in the code base. The {{ServerMetrics}} logic handles creating the
> metric, properly adding the metric to the JSON output of the {{/monitor}}
> admin command, and properly resetting the metric when necessary. The
> motivation behind {{ServerMetrics}} is to make things easy enough that it
> encourages new metrics to be added liberally. Lack of in-depth
> metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most
> of our internal changes build on {{ServerMetrics}} and we have nearly 100
> internal metrics at this time – all of which we'll be upstreaming in the
> coming months as we publish more internal patches.
> # This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
> # This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
> atomic operations.
> Here's a list of new metrics added in this patch:
> - {{uptime}}: time that a peer has been in a stable
> leading/following/observing state
> - {{leader_uptime}}: uptime for peer in leading state
> - {{global_sessions}}: count of global sessions
> - {{local_sessions}}: count of local sessions
> - {{quorum_size}}: configured ensemble size
> - {{synced_observers}}: similar to existing `synced_followers` but for
> observers
> - {{fsynctime}}: time to fsync transaction log (avg/min/max)
> - {{snapshottime}}: time to write a snapshot (avg/min/max)
> - {{dbinittime}}: time to reload database – read snapshot + apply
> transactions (avg/min/max)
> - {{readlatency}}: read request latency (avg/min/max)
> - {{updatelatency}}: update request latency (avg/min/max)
> - {{propagation_latency}}: end-to-end latency for updates, from proposal on
> leader to committed-to-datatree on a given host (avg/min/max)
> - {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
> - {{election_time}}: time between entering and leaving election (avg/min/max)
> - {{looking_count}}: number of transitions into looking state
> - {{diff_count}}: number of diff syncs performed
> - {{snap_count}}: number of snap syncs performed
> - {{commit_count}}: number of commits performed on leader
> - {{connection_request_count}}: number of incoming client connection requests
> - {{bytes_received_count}}: similar to existing `packets_received` but
> tracks bytes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zookeeper issue #580: ZOOKEEPER-3098: Add additional server metrics
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/580 Finally this happens, thanks for contributing the change back @jtuple . My quick comments inline. In addition, there are two things: * There is find bug warning. I believe it's the txn variable [here](https://github.com/jtuple/zookeeper/blob/e6935f8d99eace05d29c2d6659e68e8b90b9a633/src/java/main/org/apache/zookeeper/server/FinalRequestProcessor.java#L110) that should be removed. I know that's not part of this patch, and i am not sure why it gets triggered here, but please remove it so we can get a clean find bug check. * The test `org.apache.zookeeper.server.admin.CommandsTest` is failing. Please investigate. ---
[GitHub] zookeeper pull request #580: ZOOKEEPER-3098: Add additional server metrics
Github user hanm commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/580#discussion_r204179634 --- Diff: src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java --- @@ -271,7 +271,7 @@ private boolean checkFourLetterWord(final Channel channel, String cmd = FourLetterCommands.getCommandString(len); channel.setInterestOps(0).awaitUninterruptibly(); -packetReceived(); +packetReceived(4); --- End diff -- I am not sure what's the intention of this change.. ---
[GitHub] zookeeper pull request #580: ZOOKEEPER-3098: Add additional server metrics
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/580#discussion_r204180108
--- Diff: src/java/main/org/apache/zookeeper/server/ServerMetrics.java ---
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server;
+
+import org.apache.zookeeper.server.metric.AvgMinMaxCounter;
+import org.apache.zookeeper.server.metric.Metric;
+import org.apache.zookeeper.server.metric.SimpleCounter;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+public enum ServerMetrics {
+/**
+ * Txnlog fsync time
+ */
+FSYNC_TIME(new AvgMinMaxCounter("fsynctime")),
+
+/**
+ * Snapshot writing time
+ */
+SNAPSHOT_TIME(new AvgMinMaxCounter("snapshottime")),
+
+/**
+ * Db init time (snapshot loading + txnlog replay)
+ */
+DB_INIT_TIME(new AvgMinMaxCounter("dbinittime")),
+
+/**
+ * Stats for read request. The timing start from when the server see
the
+ * request until it leave final request processor.
+ */
+READ_LATENCY(new AvgMinMaxCounter("readlatency")),
+
+/**
+ * Stats for request that need quorum voting. Timing is the same as
read
+ * request. We only keep track of stats for request that originated
from
+ * this machine only.
+ */
+UPDATE_LATENCY(new AvgMinMaxCounter("updatelatency")),
+
+/**
+ * Stats for all quorum request. The timing start from when the leader
+ * see the request until it reach the learner.
+ */
+PROPAGATION_LATENCY(new AvgMinMaxCounter("propagation_latency")),
+
+FOLLOWER_SYNC_TIME(new AvgMinMaxCounter("follower_sync_time")),
+ELECTION_TIME(new AvgMinMaxCounter("election_time")),
--- End diff --
Should election_time be recorded as a gauge instead of a counter? Similar
for other times (fsync, init, etc)
---
[jira] [Commented] (ZOOKEEPER-3098) Add additional server metrics
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551274#comment-16551274
]
Enrico Olivelli commented on ZOOKEEPER-3098:
I have taken a look to the patch.
We can take this as a base but I would prefer not to have all the counters
retained by the values of an enum.
We should have some central non-static registry of all metrics.
Think will be the center of the work on ZOOKEEPER-3092.
The patch does not add tests, it would be good to test at least that new
metrics are reported by the monitor command.
> Add additional server metrics
> -
>
> Key: ZOOKEEPER-3098
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3098
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
>Affects Versions: 3.6.0
>Reporter: Joseph Blomstedt
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This patch adds several new server-side metrics as well as makes it easier to
> add new metrics in the future. This patch also includes a handful of other
> minor metrics-related changes.
> Here's a high-level summary of the changes.
> # This patch extends the request latency tracked in {{ServerStats}} to track
> {{read}} and {{update}} latency separately. Updates are any request that must
> be voted on and can change data, reads are all requests that can be handled
> locally and don't change data.
> # This patch adds the {{ServerMetrics}} logic and the related
> {{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
> make it incredibly easy to add new metrics. To add a new metric you just add
> one line to {{ServerMetrics}} and then directly reference that new metric
> anywhere in the code base. The {{ServerMetrics}} logic handles creating the
> metric, properly adding the metric to the JSON output of the {{/monitor}}
> admin command, and properly resetting the metric when necessary. The
> motivation behind {{ServerMetrics}} is to make things easy enough that it
> encourages new metrics to be added liberally. Lack of in-depth
> metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most
> of our internal changes build on {{ServerMetrics}} and we have nearly 100
> internal metrics at this time – all of which we'll be upstreaming in the
> coming months as we publish more internal patches.
> # This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
> # This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
> atomic operations.
> Here's a list of new metrics added in this patch:
> - {{uptime}}: time that a peer has been in a stable
> leading/following/observing state
> - {{leader_uptime}}: uptime for peer in leading state
> - {{global_sessions}}: count of global sessions
> - {{local_sessions}}: count of local sessions
> - {{quorum_size}}: configured ensemble size
> - {{synced_observers}}: similar to existing `synced_followers` but for
> observers
> - {{fsynctime}}: time to fsync transaction log (avg/min/max)
> - {{snapshottime}}: time to write a snapshot (avg/min/max)
> - {{dbinittime}}: time to reload database – read snapshot + apply
> transactions (avg/min/max)
> - {{readlatency}}: read request latency (avg/min/max)
> - {{updatelatency}}: update request latency (avg/min/max)
> - {{propagation_latency}}: end-to-end latency for updates, from proposal on
> leader to committed-to-datatree on a given host (avg/min/max)
> - {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
> - {{election_time}}: time between entering and leaving election (avg/min/max)
> - {{looking_count}}: number of transitions into looking state
> - {{diff_count}}: number of diff syncs performed
> - {{snap_count}}: number of snap syncs performed
> - {{commit_count}}: number of commits performed on leader
> - {{connection_request_count}}: number of incoming client connection requests
> - {{bytes_received_count}}: similar to existing `packets_received` but
> tracks bytes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3098) Add additional server metrics
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551244#comment-16551244
]
Enrico Olivelli commented on ZOOKEEPER-3098:
This sounds very interesting as we are discussing about the introduction of new
metrics system.
We could start from this work and maybe abstract the way metrics are published,
in order to support various metrics providers like Prometheus
> Add additional server metrics
> -
>
> Key: ZOOKEEPER-3098
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3098
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
>Affects Versions: 3.6.0
>Reporter: Joseph Blomstedt
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This patch adds several new server-side metrics as well as makes it easier to
> add new metrics in the future. This patch also includes a handful of other
> minor metrics-related changes.
> Here's a high-level summary of the changes.
> # This patch extends the request latency tracked in {{ServerStats}} to track
> {{read}} and {{update}} latency separately. Updates are any request that must
> be voted on and can change data, reads are all requests that can be handled
> locally and don't change data.
> # This patch adds the {{ServerMetrics}} logic and the related
> {{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
> make it incredibly easy to add new metrics. To add a new metric you just add
> one line to {{ServerMetrics}} and then directly reference that new metric
> anywhere in the code base. The {{ServerMetrics}} logic handles creating the
> metric, properly adding the metric to the JSON output of the {{/monitor}}
> admin command, and properly resetting the metric when necessary. The
> motivation behind {{ServerMetrics}} is to make things easy enough that it
> encourages new metrics to be added liberally. Lack of in-depth
> metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most
> of our internal changes build on {{ServerMetrics}} and we have nearly 100
> internal metrics at this time – all of which we'll be upstreaming in the
> coming months as we publish more internal patches.
> # This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
> # This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
> atomic operations.
> Here's a list of new metrics added in this patch:
> - {{uptime}}: time that a peer has been in a stable
> leading/following/observing state
> - {{leader_uptime}}: uptime for peer in leading state
> - {{global_sessions}}: count of global sessions
> - {{local_sessions}}: count of local sessions
> - {{quorum_size}}: configured ensemble size
> - {{synced_observers}}: similar to existing `synced_followers` but for
> observers
> - {{fsynctime}}: time to fsync transaction log (avg/min/max)
> - {{snapshottime}}: time to write a snapshot (avg/min/max)
> - {{dbinittime}}: time to reload database – read snapshot + apply
> transactions (avg/min/max)
> - {{readlatency}}: read request latency (avg/min/max)
> - {{updatelatency}}: update request latency (avg/min/max)
> - {{propagation_latency}}: end-to-end latency for updates, from proposal on
> leader to committed-to-datatree on a given host (avg/min/max)
> - {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
> - {{election_time}}: time between entering and leaving election (avg/min/max)
> - {{looking_count}}: number of transitions into looking state
> - {{diff_count}}: number of diff syncs performed
> - {{snap_count}}: number of snap syncs performed
> - {{commit_count}}: number of commits performed on leader
> - {{connection_request_count}}: number of incoming client connection requests
> - {{bytes_received_count}}: similar to existing `packets_received` but
> tracks bytes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: ZOOKEEPER- PreCommit Build #1984
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1984/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 78.95 MB...]
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] -1 findbugs. The patch appears to introduce 1 new Findbugs
(version 3.0.1) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] -1 core tests. The patch failed core unit tests.
[exec]
[exec] +1 contrib tests. The patch passed contrib unit tests.
[exec]
[exec] Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1984//testReport/
[exec] Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1984//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
[exec] Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1984//console
[exec]
[exec] This message is automatically generated.
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Adding comment to Jira.
[exec]
==
[exec]
==
[exec]
[exec]
[exec] Comment with id 16551223 added to ZOOKEEPER-3098.
[exec] Session logged out. Session was
JSESSIONID=1166632461E34C425AAB8BFA35B6B518.
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
[exec]
[exec]
[exec] mv:
'/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess'
and
'/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess'
are the same file
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1806:
exec returned: 2
Total time: 12 minutes 55 seconds
Build step 'Execute shell' marked build as failure
Archiving artifacts
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
Recording test results
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
[description-setter] Description set: ZOOKEEPER-3098
Putting comment on the pull request
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8
###
## FAILED TESTS (if any)
##
2 tests failed.
FAILED: org.apache.zookeeper.SaslAuthTest.testThreadsShutdownOnAuthFailed
Error Message:
Failed to connect to ZooKeeper server.
Stack Trace:
java.util.concurrent.TimeoutException: Failed to connect to ZooKeeper server.
at
org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForConnected(ClientBase.java:150)
at
org.apache.zookeeper.SaslAuthTest.testThreadsShutdownOnAuthFailed(SaslAuthTest.java:221)
at
org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
FAILED: org.apache.zookeeper.server.admin.CommandsTest.testMonitor
Error Message:
Result from command monitor contains extra fields: {uptime=9,
global_sessions=0, local_sessions=0, avg_fsynctime=0, min_fsynctime=0,
max_fsynctime=0, cnt_fsynctime=0, avg_snapshottime=0, min_snapshottime=0,
max_snapshottime=2, cnt_snapshottime=9, avg_dbinittime=1, min_dbinittime=0,
max_dbinittime=2, cnt_dbinittime=9, avg_readlatency=0, min_readlatency=0,
max_readlatency=0, cnt_readlatency=0, avg_updatelatency=0, min_updatelatency=0,
max_updatelatency=0, cnt_updatelatency=0, avg_propagation_latency=0,
min_propagation_latency=0, max_propagation_latency=0,
cnt_propagation_latency=0, avg_follower_sync_time=0, min_follower_sync_time=0,
[jira] [Commented] (ZOOKEEPER-3098) Add additional server metrics
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551223#comment-16551223
]
Hadoop QA commented on ZOOKEEPER-3098:
--
-1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 3.0.1)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1984//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1984//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1984//console
This message is automatically generated.
> Add additional server metrics
> -
>
> Key: ZOOKEEPER-3098
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3098
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
>Affects Versions: 3.6.0
>Reporter: Joseph Blomstedt
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This patch adds several new server-side metrics as well as makes it easier to
> add new metrics in the future. This patch also includes a handful of other
> minor metrics-related changes.
> Here's a high-level summary of the changes.
> # This patch extends the request latency tracked in {{ServerStats}} to track
> {{read}} and {{update}} latency separately. Updates are any request that must
> be voted on and can change data, reads are all requests that can be handled
> locally and don't change data.
> # This patch adds the {{ServerMetrics}} logic and the related
> {{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
> make it incredibly easy to add new metrics. To add a new metric you just add
> one line to {{ServerMetrics}} and then directly reference that new metric
> anywhere in the code base. The {{ServerMetrics}} logic handles creating the
> metric, properly adding the metric to the JSON output of the {{/monitor}}
> admin command, and properly resetting the metric when necessary. The
> motivation behind {{ServerMetrics}} is to make things easy enough that it
> encourages new metrics to be added liberally. Lack of in-depth
> metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most
> of our internal changes build on {{ServerMetrics}} and we have nearly 100
> internal metrics at this time – all of which we'll be upstreaming in the
> coming months as we publish more internal patches.
> # This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
> # This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
> atomic operations.
> Here's a list of new metrics added in this patch:
> - {{uptime}}: time that a peer has been in a stable
> leading/following/observing state
> - {{leader_uptime}}: uptime for peer in leading state
> - {{global_sessions}}: count of global sessions
> - {{local_sessions}}: count of local sessions
> - {{quorum_size}}: configured ensemble size
> - {{synced_observers}}: similar to existing `synced_followers` but for
> observers
> - {{fsynctime}}: time to fsync transaction log (avg/min/max)
> - {{snapshottime}}: time to write a snapshot (avg/min/max)
> - {{dbinittime}}: time to reload database – read snapshot + apply
> transactions (avg/min/max)
> - {{readlatency}}: read request latency (avg/min/max)
> - {{updatelatency}}: update request latency (avg/min/max)
> - {{propagation_latency}}: end-to-end latency for updates, from proposal on
> leader to committed-to-datatree on a given host (avg/min/max)
> - {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
> - {{election_time}}: time between entering and leaving election (avg/min/max)
> - {{looking_count}}: number of transitions into looking state
> - {{diff_count}}: number of diff syncs performed
> - {{snap_count}}: number of snap syncs performed
> - {{commit_count}}: number of commits performed on leader
> - {{connection_request_count}}: number of incoming client connection requests
> - {{bytes_received_count}}: similar to existing `packets_received` but
> tracks bytes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ZOOKEEPER-3098) Add additional server metrics
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Blomstedt updated ZOOKEEPER-3098:
Description:
This patch adds several new server-side metrics as well as makes it easier to
add new metrics in the future. This patch also includes a handful of other
minor metrics-related changes.
Here's a high-level summary of the changes.
# This patch extends the request latency tracked in {{ServerStats}} to track
{{read}} and {{update}} latency separately. Updates are any request that must
be voted on and can change data, reads are all requests that can be handled
locally and don't change data.
# This patch adds the {{ServerMetrics}} logic and the related
{{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
make it incredibly easy to add new metrics. To add a new metric you just add
one line to {{ServerMetrics}} and then directly reference that new metric
anywhere in the code base. The {{ServerMetrics}} logic handles creating the
metric, properly adding the metric to the JSON output of the {{/monitor}} admin
command, and properly resetting the metric when necessary. The motivation
behind {{ServerMetrics}} is to make things easy enough that it encourages new
metrics to be added liberally. Lack of in-depth metrics/visibility is a
long-standing ZooKeeper weakness. At Facebook, most of our internal changes
build on {{ServerMetrics}} and we have nearly 100 internal metrics at this time
– all of which we'll be upstreaming in the coming months as we publish more
internal patches.
# This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
# This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
atomic operations.
Here's a list of new metrics added in this patch:
- {{uptime}}: time that a peer has been in a stable
leading/following/observing state
- {{leader_uptime}}: uptime for peer in leading state
- {{global_sessions}}: count of global sessions
- {{local_sessions}}: count of local sessions
- {{quorum_size}}: configured ensemble size
- {{synced_observers}}: similar to existing `synced_followers` but for
observers
- {{fsynctime}}: time to fsync transaction log (avg/min/max)
- {{snapshottime}}: time to write a snapshot (avg/min/max)
- {{dbinittime}}: time to reload database – read snapshot + apply transactions
(avg/min/max)
- {{readlatency}}: read request latency (avg/min/max)
- {{updatelatency}}: update request latency (avg/min/max)
- {{propagation_latency}}: end-to-end latency for updates, from proposal on
leader to committed-to-datatree on a given host (avg/min/max)
- {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
- {{election_time}}: time between entering and leaving election (avg/min/max)
- {{looking_count}}: number of transitions into looking state
- {{diff_count}}: number of diff syncs performed
- {{snap_count}}: number of snap syncs performed
- {{commit_count}}: number of commits performed on leader
- {{connection_request_count}}: number of incoming client connection requests
- {{bytes_received_count}}: similar to existing `packets_received` but tracks
bytes
was:
This patch adds several new server-side metrics as well as makes it easier to
add new metrics in the future. This patch also includes a handful of other
minor metrics-related changes.
Here's a high-level summary of the changes.
# This patch extends the request latency tracked in {{ServerStats}} to track
{{read}} and {{update}} latency separately. Updates are any request that must
be voted on and can change data, reads are all requests that can be handled
locally and don't change data.
# This patch adds the {{ServerMetrics}} logic and the related
{{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
make it incredibly easy to add new metrics. To add a new metric you just add
one line to {{ServerMetrics}} and then directly reference that new metric
anywhere in the code base. The {{ServerMetrics}} logic handles creating the
metric, properly adding the metric to the JSON output of the {{/monitor}} admin
command, and properly resetting the metric when necessary.
The motivation behind {{ServerMetrics}} is to make things easy enough that it
encourages new metrics to be added liberally. Lack of in-depth
metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most of
our internal changes build on {{ServerMetrics}} and we have nearly 100 internal
metrics at this time – all of which we'll be upstreaming in the coming months
as we publish more internal patches.
# This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
# This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
atomic operations.
Here's a list of new metrics added in this patch:
- {{uptime}}: time that a peer has been in a stable
[jira] [Updated] (ZOOKEEPER-3098) Add additional server metrics
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ZOOKEEPER-3098:
--
Labels: pull-request-available (was: )
> Add additional server metrics
> -
>
> Key: ZOOKEEPER-3098
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3098
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
>Affects Versions: 3.6.0
>Reporter: Joseph Blomstedt
>Priority: Major
> Labels: pull-request-available
>
> This patch adds several new server-side metrics as well as makes it easier to
> add new metrics in the future. This patch also includes a handful of other
> minor metrics-related changes.
> Here's a high-level summary of the changes.
> # This patch extends the request latency tracked in {{ServerStats}} to track
> {{read}} and {{update}} latency separately. Updates are any request that must
> be voted on and can change data, reads are all requests that can be handled
> locally and don't change data.
> # This patch adds the {{ServerMetrics}} logic and the related
> {{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
> make it incredibly easy to add new metrics. To add a new metric you just add
> one line to {{ServerMetrics}} and then directly reference that new metric
> anywhere in the code base. The {{ServerMetrics}} logic handles creating the
> metric, properly adding the metric to the JSON output of the {{/monitor}}
> admin command, and properly resetting the metric when necessary.
> The motivation behind {{ServerMetrics}} is to make things easy enough that it
> encourages new metrics to be added liberally. Lack of in-depth
> metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most
> of our internal changes build on {{ServerMetrics}} and we have nearly 100
> internal metrics at this time – all of which we'll be upstreaming in the
> coming months as we publish more internal patches.
> # This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
> # This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
> atomic operations.
> Here's a list of new metrics added in this patch:
> - {{uptime}}: time that a peer has been in a stable
> leading/following/observing state
> - {{leader_uptime}}: uptime for peer in leading state
> - {{global_sessions}}: count of global sessions
> - {{local_sessions}}: count of local sessions
> - {{quorum_size}}: configured ensemble size
> - {{synced_observers}}: similar to existing `synced_followers` but for
> observers
> - {{fsynctime}}: time to fsync transaction log (avg/min/max)
> - {{snapshottime}}: time to write a snapshot (avg/min/max)
> - {{dbinittime}}: time to reload database – read snapshot + apply
> transactions (avg/min/max)
> - {{readlatency}}: read request latency (avg/min/max)
> - {{updatelatency}}: update request latency (avg/min/max)
> - {{propagation_latency}}: end-to-end latency for updates, from proposal on
> leader to committed-to-datatree on a given host (avg/min/max)
> - {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
> - {{election_time}}: time between entering and leaving election (avg/min/max)
> - {{looking_count}}: number of transitions into looking state
> - {{diff_count}}: number of diff syncs performed
> - {{snap_count}}: number of snap syncs performed
> - {{commit_count}}: number of commits performed on leader
> - {{connection_request_count}}: number of incoming client connection requests
> - {{bytes_received_count}}: similar to existing `packets_received` but
> tracks bytes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zookeeper pull request #580: ZOOKEEPER-3098: Add additional server metrics
GitHub user jtuple opened a pull request: https://github.com/apache/zookeeper/pull/580 ZOOKEEPER-3098: Add additional server metrics This patch adds several new server-side metrics as well as makes it easier to add new metrics in the future. This patch also includes a handful of other minor metrics-related changes. Here's a high-level summary of the changes. 1. This patch extends the request latency tracked in `ServerStats` to track `read` and `update` latency separately. Updates are any request that must be voted on and can change data, reads are all requests that can be handled locally and don't change data. 2. This patch adds the `ServerMetrics` logic and the related `AvgMinMaxCounter` and `SimpleCounter` classes. This code is designed to make it incredibly easy to add new metrics. To add a new metric you just add one line to `ServerMetrics` and then directly reference that new metric anywhere in the code base. The `ServerMetrics` logic handles creating the metric, properly adding the metric to the JSON output of the `/monitor` admin command, and properly resetting the metric when necessary. The motivation behind `ServerMetrics` is to make things easy enough that it encourages new metrics to be added liberally. Lack of in-depth metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most of our internal changes build on `ServerMetrics` and we have nearly 100 internal metrics at this time -- all of which we'll be upstreaming in the coming months as we publish more internal patches. 3. This patch adds 20 new metrics, 14 which are handled by `ServerMetrics`. 4. This patch replaces some uses of `synchronized` in `ServerStats` with atomic operations. Here's a list of new metrics added in this patch: - `uptime`: time that a peer has been in a stable leading/following/observing state - `leader_uptime`: uptime for peer in leading state - `global_sessions`: count of global sessions - `local_sessions`: count of local sessions - `quorum_size`: configured ensemble size - `synced_observers`: similar to existing `synced_followers` but for observers - `fsynctime`: time to fsync transaction log (avg/min/max) - `snapshottime`: time to write a snapshot (avg/min/max) - `dbinittime`: time to reload database -- read snapshot + apply transactions (avg/min/max) - `readlatency`: read request latency (avg/min/max) - `updatelatency`: update request latency (avg/min/max) - `propagation_latency`: end-to-end latency for updates, from proposal on leader to committed-to-datatree on a given host (avg/min/max) - `follower_sync_time`: time for follower to sync with leader (avg/min/max) - `election_time`: time between entering and leaving election (avg/min/max) - `looking_count`: number of transitions into looking state - `diff_count`: number of diff syncs performed - `snap_count`: number of snap syncs performed - `commit_count`: number of commits performed on leader - `connection_request_count`: number of incoming client connection requests - `bytes_received_count`: similar to existing `packets_received` but tracks bytes You can merge this pull request into a Git repository by running: $ git pull https://github.com/jtuple/zookeeper ZOOKEEPER-3098 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/580.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #580 commit e6935f8d99eace05d29c2d6659e68e8b90b9a633 Author: Joseph Blomstedt Date: 2018-07-19T19:47:15Z ZOOKEEPER-3098: Add additional server metrics This patch adds several new server-side metrics as well as makes it easier to add new metrics in the future. This patch also includes a handful of other minor metrics-related changes. Here's a high-level summary of the changes. 1. This patch extends the request latency tracked in `ServerStats` to track `read` and `update` latency separately. Updates are any request that must be voted on and can change data, reads are all requests that can be handled locally and don't change data. 2. This patch adds the `ServerMetrics` logic and the related `AvgMinMaxCounter` and `SimpleCounter` classes. This code is designed to make it incredibly easy to add new metrics. To add a new metric you just add one line to `ServerMetrics` and then directly reference that new metric anywhere in the code base. The `ServerMetrics` logic handles creating the metric, properly adding the metric to the JSON output of the `/monitor` admin command, and properly resetting the metric when necessary.
[jira] [Updated] (ZOOKEEPER-3095) Connect string fix for non-existent hosts
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamed Jeelani updated ZOOKEEPER-3095: --- Labels: pull-request-available (was: ) > Connect string fix for non-existent hosts > - > > Key: ZOOKEEPER-3095 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3095 > Project: ZooKeeper > Issue Type: Improvement > Components: other >Affects Versions: 3.4.0 >Reporter: Mohamed Jeelani >Assignee: Mohamed Jeelani >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > > Connect string fix for non-existent hosts -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Success: ZOOKEEPER- PreCommit Build #1983
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1983/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 78.34 MB...] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1983//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1983//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1983//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=6F3A838DC5572BD9030A55E88AC04378. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 18 minutes 5 seconds Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Could not determine description. Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Created] (ZOOKEEPER-3098) Add additional server metrics
Joseph Blomstedt created ZOOKEEPER-3098:
---
Summary: Add additional server metrics
Key: ZOOKEEPER-3098
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3098
Project: ZooKeeper
Issue Type: Improvement
Components: server
Affects Versions: 3.6.0
Reporter: Joseph Blomstedt
This patch adds several new server-side metrics as well as makes it easier to
add new metrics in the future. This patch also includes a handful of other
minor metrics-related changes.
Here's a high-level summary of the changes.
# This patch extends the request latency tracked in {{ServerStats}} to track
{{read}} and {{update}} latency separately. Updates are any request that must
be voted on and can change data, reads are all requests that can be handled
locally and don't change data.
# This patch adds the {{ServerMetrics}} logic and the related
{{AvgMinMaxCounter}} and {{SimpleCounter}} classes. This code is designed to
make it incredibly easy to add new metrics. To add a new metric you just add
one line to {{ServerMetrics}} and then directly reference that new metric
anywhere in the code base. The {{ServerMetrics}} logic handles creating the
metric, properly adding the metric to the JSON output of the {{/monitor}} admin
command, and properly resetting the metric when necessary.
The motivation behind {{ServerMetrics}} is to make things easy enough that it
encourages new metrics to be added liberally. Lack of in-depth
metrics/visibility is a long-standing ZooKeeper weakness. At Facebook, most of
our internal changes build on {{ServerMetrics}} and we have nearly 100 internal
metrics at this time – all of which we'll be upstreaming in the coming months
as we publish more internal patches.
# This patch adds 20 new metrics, 14 which are handled by {{ServerMetrics}}.
# This patch replaces some uses of {{synchronized}} in {{ServerStats}} with
atomic operations.
Here's a list of new metrics added in this patch:
- {{uptime}}: time that a peer has been in a stable
leading/following/observing state
- {{leader_uptime}}: uptime for peer in leading state
- {{global_sessions}}: count of global sessions
- {{local_sessions}}: count of local sessions
- {{quorum_size}}: configured ensemble size
- {{synced_observers}}: similar to existing `synced_followers` but for
observers
- {{fsynctime}}: time to fsync transaction log (avg/min/max)
- {{snapshottime}}: time to write a snapshot (avg/min/max)
- {{dbinittime}}: time to reload database – read snapshot + apply transactions
(avg/min/max)
- {{readlatency}}: read request latency (avg/min/max)
- {{updatelatency}}: update request latency (avg/min/max)
- {{propagation_latency}}: end-to-end latency for updates, from proposal on
leader to committed-to-datatree on a given host (avg/min/max)
- {{follower_sync_time}}: time for follower to sync with leader (avg/min/max)
- {{election_time}}: time between entering and leaving election (avg/min/max)
- {{looking_count}}: number of transitions into looking state
- {{diff_count}}: number of diff syncs performed
- {{snap_count}}: number of snap syncs performed
- {{commit_count}}: number of commits performed on leader
- {{connection_request_count}}: number of incoming client connection requests
- {{bytes_received_count}}: similar to existing `packets_received` but tracks
bytes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zookeeper pull request #579: Connect string fix for non-existent hosts
GitHub user mjeelanimsft opened a pull request: https://github.com/apache/zookeeper/pull/579 Connect string fix for non-existent hosts ZKPatch: eda58d9970c76831046ddc45251c9b110856836e (extract) You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjeelanimsft/zookeeper connect-string-fix-for-non-existent-hosts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/579.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #579 commit b3e0cdcef95a1ad043c9a09d8847d0ea0c69361f Author: Jeelani Mohamed Abdul Khader Date: 2018-07-20T18:02:04Z Connect string fix for non-existent hosts ZKPatch: eda58d9970c76831046ddc45251c9b110856836e (extract) ---
[jira] [Commented] (ZOOKEEPER-3097) Use Runnable instead of Thread for working items in WorkerService to improve the throughput of CommitProcessor
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551175#comment-16551175 ] Hadoop QA commented on ZOOKEEPER-3097: -- +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1982//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1982//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1982//console This message is automatically generated. > Use Runnable instead of Thread for working items in WorkerService to improve > the throughput of CommitProcessor > -- > > Key: ZOOKEEPER-3097 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3097 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Affects Versions: 3.6.0 >Reporter: Fangmin Lv >Assignee: Fangmin Lv >Priority: Minor > Labels: performance, pull-request-available > Fix For: 3.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > CommitProcessor is using this to submit read/write tasks, each task is > initialized as a thread, which is heavy, change it to a lighter Runnable > object to avoid the overhead of initializing the thread, it shows promised > improvement in the CommitProcessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Success: ZOOKEEPER- PreCommit Build #1982
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1982/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 78.13 MB...] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +0 tests included. The patch appears to be a documentation patch that doesn't require tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1982//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1982//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1982//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment with id 16551175 added to ZOOKEEPER-3097. [exec] Session logged out. Session was JSESSIONID=450230AEA8B7D912AC7CE3811E180C36. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 18 minutes 6 seconds Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3097 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Updated] (ZOOKEEPER-3097) Use Runnable instead of Thread for working items in WorkerService to improve the throughput of CommitProcessor
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ZOOKEEPER-3097: -- Labels: performance pull-request-available (was: performance) > Use Runnable instead of Thread for working items in WorkerService to improve > the throughput of CommitProcessor > -- > > Key: ZOOKEEPER-3097 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3097 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Affects Versions: 3.6.0 >Reporter: Fangmin Lv >Assignee: Fangmin Lv >Priority: Minor > Labels: performance, pull-request-available > Fix For: 3.6.0 > > > CommitProcessor is using this to submit read/write tasks, each task is > initialized as a thread, which is heavy, change it to a lighter Runnable > object to avoid the overhead of initializing the thread, it shows promised > improvement in the CommitProcessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zookeeper pull request #578: [ZOOKEEPER-3097] Using Runnable instead of thre...
GitHub user lvfangmin opened a pull request: https://github.com/apache/zookeeper/pull/578 [ZOOKEEPER-3097] Using Runnable instead of thread work items to improve the throughput of CommitProcessor You can merge this pull request into a Git repository by running: $ git pull https://github.com/lvfangmin/zookeeper ZOOKEEPER-3097 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/578.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #578 commit bc22a34f30656a14e3d7bb2383156bc662df568f Author: Fangmin Lyu Date: 2018-07-20T19:16:05Z using runnable instead of thread for work items ---
[jira] [Created] (ZOOKEEPER-3097) Use Runnable instead of Thread for working items in WorkerService to improve the throughput of CommitProcessor
Fangmin Lv created ZOOKEEPER-3097: - Summary: Use Runnable instead of Thread for working items in WorkerService to improve the throughput of CommitProcessor Key: ZOOKEEPER-3097 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3097 Project: ZooKeeper Issue Type: Improvement Components: server Affects Versions: 3.6.0 Reporter: Fangmin Lv Assignee: Fangmin Lv Fix For: 3.6.0 CommitProcessor is using this to submit read/write tasks, each task is initialized as a thread, which is heavy, change it to a lighter Runnable object to avoid the overhead of initializing the thread, it shows promised improvement in the CommitProcessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Failed: ZOOKEEPER-2504 PreCommit Build #3699
Jira: https://issues.apache.org/jira/browse/ZOOKEEPER-2504 Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3699/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7.11 KB...] [exec] [exec] [exec] [exec] -1 overall. Here are the results of testing the latest attachment [exec] http://issues.apache.org/jira/secure/attachment/12823503/ZOOKEEPER-2504.patch [exec] against trunk revision ed4689fbf81c56f09be26cf32ede66ad191f4aed. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] -1 patch. The patch command could not apply the patch. [exec] [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3699//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment with id 16551140 added to ZOOKEEPER-2504. [exec] Session logged out. Session was JSESSIONID=5E8FEB543A4B51FC7DA615C3203E6D89. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/build.xml:1785: exec returned: 1 Total time: 47 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [Fast Archiver] No prior successful build to compare, so performing full copy of artifacts Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3068 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Commented] (ZOOKEEPER-2504) Enforce that server ids are unique in a cluster
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551140#comment-16551140 ] Hadoop QA commented on ZOOKEEPER-2504: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12823503/ZOOKEEPER-2504.patch against trunk revision ed4689fbf81c56f09be26cf32ede66ad191f4aed. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3699//console This message is automatically generated. > Enforce that server ids are unique in a cluster > --- > > Key: ZOOKEEPER-2504 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2504 > Project: ZooKeeper > Issue Type: Bug >Reporter: Dan Benediktson >Assignee: Michael Han >Priority: Major > Attachments: ZOOKEEPER-2504.patch > > > The leader will happily accept connections from learners that have the same > server id (i.e., due to misconfiguration). This can lead to various issues > including non-unique session_ids being generated by these servers. > The leader can enforce that all learners come in with unique server IDs; if a > learner attempts to connect with an id that is already in use, it should be > denied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ZOOKEEPER-2504) Enforce that server ids are unique in a cluster
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Han reassigned ZOOKEEPER-2504: -- Assignee: Michael Han (was: Dan Benediktson) > Enforce that server ids are unique in a cluster > --- > > Key: ZOOKEEPER-2504 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2504 > Project: ZooKeeper > Issue Type: Bug >Reporter: Dan Benediktson >Assignee: Michael Han >Priority: Major > Attachments: ZOOKEEPER-2504.patch > > > The leader will happily accept connections from learners that have the same > server id (i.e., due to misconfiguration). This can lead to various issues > including non-unique session_ids being generated by these servers. > The leader can enforce that all learners come in with unique server IDs; if a > learner attempts to connect with an id that is already in use, it should be > denied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-2504) Enforce that server ids are unique in a cluster
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551122#comment-16551122 ] Michael Han commented on ZOOKEEPER-2504: This patch still has its value, I am taking it over here. > Enforce that server ids are unique in a cluster > --- > > Key: ZOOKEEPER-2504 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2504 > Project: ZooKeeper > Issue Type: Bug >Reporter: Dan Benediktson >Assignee: Dan Benediktson >Priority: Major > Attachments: ZOOKEEPER-2504.patch > > > The leader will happily accept connections from learners that have the same > server id (i.e., due to misconfiguration). This can lead to various issues > including non-unique session_ids being generated by these servers. > The leader can enforce that all learners come in with unique server IDs; if a > learner attempts to connect with an id that is already in use, it should be > denied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZOOKEEPER-3096) Leader should not leak LearnerHandler threads
Michael Han created ZOOKEEPER-3096: -- Summary: Leader should not leak LearnerHandler threads Key: ZOOKEEPER-3096 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3096 Project: ZooKeeper Issue Type: Bug Components: quorum, server Affects Versions: 3.4.13, 3.5.4, 3.6.0 Reporter: Michael Han Assignee: Michael Han Currently we don't track LearnerHandler threads in leader; we rely on the socket timeout to raise an exception and use that exception as a signal to let the LearnerHandler thread kills itself. In cases where the learners restarts, if the time between restart beginning to finishing is less than the socket timeout value (currently hardcoded as initLimit * tickTime), then there will be no exception raised and the previous LearnerHandler thread corresponding to this learner will leak. I have a test case and a proposed fix which I will submit later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Failed: ZOOKEEPER- PreCommit Build #1981
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1981/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 77.38 MB...] [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1981//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1981//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1981//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Error: No value specified for option "issue" [exec] Session logged out. Session was JSESSIONID=477D281492A33AFF07C09442FE59761C. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1806: exec returned: 1 Total time: 12 minutes 38 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Could not determine description. Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testLeaderOutOfView Error Message: null Stack Trace: junit.framework.AssertionFailedError at org.apache.zookeeper.server.quorum.QuorumPeerMainTest.testLeaderOutOfView(QuorumPeerMainTest.java:1089) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
[jira] [Created] (ZOOKEEPER-3095) Connect string fix for non-existent hosts
Mohamed Jeelani created ZOOKEEPER-3095: -- Summary: Connect string fix for non-existent hosts Key: ZOOKEEPER-3095 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3095 Project: ZooKeeper Issue Type: Improvement Components: other Affects Versions: 3.4.0 Reporter: Mohamed Jeelani Assignee: Mohamed Jeelani Fix For: 3.6.0 Connect string fix for non-existent hosts -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ZOOKEEPER-3094) Make BufferSizeTest reliable
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamed Jeelani updated ZOOKEEPER-3094: --- Labels: pull-request-available (was: ) > Make BufferSizeTest reliable > > > Key: ZOOKEEPER-3094 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3094 > Project: ZooKeeper > Issue Type: Improvement > Components: tests >Affects Versions: 3.4.0 >Reporter: Mohamed Jeelani >Assignee: Mohamed Jeelani >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > > Improve reliability of BufferSizeTest. > Changes made to the testStartupFailure test to remember the old directory and > switch back to it after the test has completed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3086) [server] Lack of write timeouts causes quorum to stuck
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551051#comment-16551051
]
Ruslan Nigmatullin commented on ZOOKEEPER-3086:
---
ZK codebase uses `Socket.setSoTimeout` to setup a timeout, however based on the
[documentation|https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html#setSoTimeout-int-]
it's only used for read operations.
{quote}With this option set to a non-zero timeout, a read() call on the
InputStream associated with this Socket will block for only this amount of time.
{quote}
> [server] Lack of write timeouts causes quorum to stuck
> --
>
> Key: ZOOKEEPER-3086
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3086
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
>Affects Versions: 3.5.4, 3.4.12
> Environment: Linux 4.13.0-32-generic, Java HotSpot(TM) 64-Bit Server
> VM (build 25.121-b13, mixed mode)
>Reporter: Ruslan Nigmatullin
>Priority: Major
> Attachments: zookeeper-threads.txt
>
>
> Network outage on leader host can cause `QuorumPeer` thread to stuck for
> prolonged period of time (2+ hours, depends on tcp keep alive settings). It
> effectively stalls the whole zookeeper server making it inoperable. We've
> found it during one of our internal DRTs (Disaster Recovery Test).
> The scenario which triggers the behavior (requires relatively high ping-load
> to the follower):
> # `Follower.processPacket` processes `Leader.PING` message
> # Leader is network partitioned
> # `Learner.ping` makes attempt to write to the leader socket
> # If write socket buffer is full (due to other ping/sync calls)
> `Learner.ping` blocks
> # As leader is partitioned - `Learner.ping` blocks forever due to lack of
> write timeout
> # `QuorumPeer` is the only thread reading from the leader socket,
> effectively meaning that the whole server is stuck and can't recover without
> manual process restart.
>
> Thread dump from the affected server is in attachments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ZOOKEEPER-3094) Make BufferSizeTest reliable
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamed Jeelani updated ZOOKEEPER-3094: --- Changes made to the testStartupFailure test to remember the old directory and switch back to it after the test has completed. > Make BufferSizeTest reliable > > > Key: ZOOKEEPER-3094 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3094 > Project: ZooKeeper > Issue Type: Improvement > Components: tests >Affects Versions: 3.4.0 >Reporter: Mohamed Jeelani >Assignee: Mohamed Jeelani >Priority: Minor > Fix For: 3.6.0 > > > Improve reliability of BufferSizeTest. > Make changes to improve BufferSizeTest more reliable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (ZOOKEEPER-3094) Make BufferSizeTest reliable
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamed Jeelani updated ZOOKEEPER-3094: --- Comment: was deleted (was: Changes made to the testStartupFailure test to remember the old directory and switch back to it after the test has completed.) > Make BufferSizeTest reliable > > > Key: ZOOKEEPER-3094 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3094 > Project: ZooKeeper > Issue Type: Improvement > Components: tests >Affects Versions: 3.4.0 >Reporter: Mohamed Jeelani >Assignee: Mohamed Jeelani >Priority: Minor > Fix For: 3.6.0 > > > Improve reliability of BufferSizeTest. > Changes made to the testStartupFailure test to remember the old directory and > switch back to it after the test has completed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ZOOKEEPER-3094) Make BufferSizeTest reliable
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamed Jeelani updated ZOOKEEPER-3094: --- Description: Improve reliability of BufferSizeTest. Changes made to the testStartupFailure test to remember the old directory and switch back to it after the test has completed. was: Improve reliability of BufferSizeTest. Make changes to improve BufferSizeTest more reliable. > Make BufferSizeTest reliable > > > Key: ZOOKEEPER-3094 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3094 > Project: ZooKeeper > Issue Type: Improvement > Components: tests >Affects Versions: 3.4.0 >Reporter: Mohamed Jeelani >Assignee: Mohamed Jeelani >Priority: Minor > Fix For: 3.6.0 > > > Improve reliability of BufferSizeTest. > Changes made to the testStartupFailure test to remember the old directory and > switch back to it after the test has completed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zookeeper pull request #577: [3094] Make BufferSizeTest reliable
GitHub user mjeelanimsft opened a pull request: https://github.com/apache/zookeeper/pull/577 [3094] Make BufferSizeTest reliable ZKPatch: d8b825d6533e9848565beff09e9a018136ca32bd (extract) Changes made to the testStartupFailure test to remember the old directory and switch back to it after the test has completed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjeelanimsft/zookeeper make-buffersizetest-reliable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #577 commit c519b5116791ce9cfd04e8afb5a3639e22b6ee21 Author: Jeelani Mohamed Abdul Khader Date: 2018-07-20T17:12:13Z Make BufferSizeTest reliable ZKPatch: d8b825d6533e9848565beff09e9a018136ca32bd (extract) ---
[jira] [Commented] (ZOOKEEPER-3086) [server] Lack of write timeouts causes quorum to stuck
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551044#comment-16551044 ] Fangmin Lv commented on ZOOKEEPER-3086: --- [~elessar] thanks for reporting the issue, you mentioned at step 5, the follower blocks on Learner.ping forever due to the lack of write timeout, but in the follower/leader socket, we do set the timeout to be tickTime * syncLimit, it should timed out if we cannot send the ping within that time. > [server] Lack of write timeouts causes quorum to stuck > -- > > Key: ZOOKEEPER-3086 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3086 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.5.4, 3.4.12 > Environment: Linux 4.13.0-32-generic, Java HotSpot(TM) 64-Bit Server > VM (build 25.121-b13, mixed mode) >Reporter: Ruslan Nigmatullin >Priority: Major > Attachments: zookeeper-threads.txt > > > Network outage on leader host can cause `QuorumPeer` thread to stuck for > prolonged period of time (2+ hours, depends on tcp keep alive settings). It > effectively stalls the whole zookeeper server making it inoperable. We've > found it during one of our internal DRTs (Disaster Recovery Test). > The scenario which triggers the behavior (requires relatively high ping-load > to the follower): > # `Follower.processPacket` processes `Leader.PING` message > # Leader is network partitioned > # `Learner.ping` makes attempt to write to the leader socket > # If write socket buffer is full (due to other ping/sync calls) > `Learner.ping` blocks > # As leader is partitioned - `Learner.ping` blocks forever due to lack of > write timeout > # `QuorumPeer` is the only thread reading from the leader socket, > effectively meaning that the whole server is stuck and can't recover without > manual process restart. > > Thread dump from the affected server is in attachments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZOOKEEPER-3094) Make BufferSizeTest reliable
Mohamed Jeelani created ZOOKEEPER-3094: -- Summary: Make BufferSizeTest reliable Key: ZOOKEEPER-3094 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3094 Project: ZooKeeper Issue Type: Improvement Components: tests Affects Versions: 3.4.0 Reporter: Mohamed Jeelani Assignee: Mohamed Jeelani Fix For: 3.6.0 Improve reliability of BufferSizeTest. Make changes to improve BufferSizeTest more reliable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ZOOKEEPER-3057) Fix IPv6 literal usage
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamed Jeelani reassigned ZOOKEEPER-3057: -- Assignee: Mohamed Jeelani > Fix IPv6 literal usage > -- > > Key: ZOOKEEPER-3057 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3057 > Project: ZooKeeper > Issue Type: Bug > Components: other >Affects Versions: 3.4.12 >Reporter: Mohamed Jeelani >Assignee: Mohamed Jeelani >Priority: Minor > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > IPv6 literals are not parsed correctly and can lead to potential errors if > not be an eye sore. Need to parse and display them correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3068) Improve C client logging of IPv6 hosts
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550988#comment-16550988
]
Hudson commented on ZOOKEEPER-3068:
---
SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #113 (See
[https://builds.apache.org/job/ZooKeeper-trunk/113/])
ZOOKEEPER-3068: Improve C client logging of IPv6 hosts (andor: rev
ed4689fbf81c56f09be26cf32ede66ad191f4aed)
* (edit) src/c/src/zookeeper.c
* (edit) src/c/tests/TestReconfig.cc
> Improve C client logging of IPv6 hosts
> --
>
> Key: ZOOKEEPER-3068
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3068
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.6.0
>Reporter: Brian Nixon
>Assignee: Brian Nixon
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.6.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The C client formats host-port pairings as [host:port] when logging. This is
> visually confusing when the host is an IPv6 address (see the below). In that
> case, it would be preferable to cleanly separate the IPv6 from the port.
> {code:java}
> ZOO_INFO@check_events@2736: initiated connection to server
> [2401:db00:1020:40bf:face:0:5:0:2181]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Success: ZOOKEEPER- PreCommit Build #1980
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1980/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 77.63 MB...] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 89 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1980//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1980//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1980//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment with id 16550979 added to ZOOKEEPER-3030. [exec] Session logged out. Session was JSESSIONID=6B238E101B2612C3C21BD52CE8887C8A. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 17 minutes 40 seconds Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3030 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (ZOOKEEPER-3030) Step 1.3 - Create zk-contrib maven structure
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550979#comment-16550979
]
Hadoop QA commented on ZOOKEEPER-3030:
--
+1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 89 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1980//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1980//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1980//console
This message is automatically generated.
> Step 1.3 - Create zk-contrib maven structure
>
>
> Key: ZOOKEEPER-3030
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3030
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: build, scripts
>Affects Versions: 3.6.0
>Reporter: Norbert Kalmar
>Assignee: Norbert Kalmar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Create a project structure that separates the different parts of ZooKeeper
> into a more meaningful packages for the future maven build.
> This should be done in iterations to limit the impact.
> * First iteration - safe changes including moving src/docs to zk-docs,
> creating zk-it empty directory. Build and conf directory remains unchanged.
> These changes also have minimum impact on PR’s.
> * Second iteration - move src/recipes to zk-recipes.
> * *Third iteration* - move src/contrib to zk-contrib.
> * Fourth iteration - move src/c to zk-client (java will be moved in Phase 2)
> * Fifth iteration - move jute under src directory
> * Sixth iteration - move src/java/main to zk-server, which will be further
> separated in Step 2.
> {noformat}
> zookeeper
> | -bin
> | -conf
> | -zookeeper-contrib
> || -zookeeper-contrib-fatjar
> || -zookeeper-contrib-huebrowser
> || -zookeeper-contrib-loggraph
> || -zookeeper-contrib-monitoring
> || -zookeeper-contrib-rest
> || -zookeeper-contrib-zkfuse
> || -zookeeper-contrib-zkperl
> || -zookeeper-contrib-zkpython
> || -zookeeper-contrib-zktreeutil
> |\ -zookeeper-contrib-zooinspector
> | -zookeeper-docs
> | -zookeeper-it (integration tests)
> | -zookeeper-recipes
> || -zookeeper-recipes-election
> || -zookeeper-recipes-lock
> \\ -zookeeper-recipes-queue
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: ZOOKEEPER- PreCommit Build #1979
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1979/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 76.93 MB...] [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1979//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1979//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1979//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment with id 16550944 added to ZOOKEEPER-3030. [exec] Session logged out. Session was JSESSIONID=784D128933C7B71257157FEC9D9FD2DC. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1813: exec returned: 1 Total time: 18 minutes 59 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3030 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun Error Message: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. Stack Trace: junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. at java.lang.Thread.run(Thread.java:748)
[jira] [Commented] (ZOOKEEPER-3030) Step 1.3 - Create zk-contrib maven structure
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550944#comment-16550944
]
Hadoop QA commented on ZOOKEEPER-3030:
--
-1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 89 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1979//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1979//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1979//console
This message is automatically generated.
> Step 1.3 - Create zk-contrib maven structure
>
>
> Key: ZOOKEEPER-3030
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3030
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: build, scripts
>Affects Versions: 3.6.0
>Reporter: Norbert Kalmar
>Assignee: Norbert Kalmar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Create a project structure that separates the different parts of ZooKeeper
> into a more meaningful packages for the future maven build.
> This should be done in iterations to limit the impact.
> * First iteration - safe changes including moving src/docs to zk-docs,
> creating zk-it empty directory. Build and conf directory remains unchanged.
> These changes also have minimum impact on PR’s.
> * Second iteration - move src/recipes to zk-recipes.
> * *Third iteration* - move src/contrib to zk-contrib.
> * Fourth iteration - move src/c to zk-client (java will be moved in Phase 2)
> * Fifth iteration - move jute under src directory
> * Sixth iteration - move src/java/main to zk-server, which will be further
> separated in Step 2.
> {noformat}
> zookeeper
> | -bin
> | -conf
> | -zookeeper-contrib
> || -zookeeper-contrib-fatjar
> || -zookeeper-contrib-huebrowser
> || -zookeeper-contrib-loggraph
> || -zookeeper-contrib-monitoring
> || -zookeeper-contrib-rest
> || -zookeeper-contrib-zkfuse
> || -zookeeper-contrib-zkperl
> || -zookeeper-contrib-zkpython
> || -zookeeper-contrib-zktreeutil
> |\ -zookeeper-contrib-zooinspector
> | -zookeeper-docs
> | -zookeeper-it (integration tests)
> | -zookeeper-recipes
> || -zookeeper-recipes-election
> || -zookeeper-recipes-lock
> \\ -zookeeper-recipes-queue
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: ZOOKEEPER- PreCommit Build #1978
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 77.38 MB...] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 0 warnings). [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//testReport/ [exec] Release audit warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment with id 16550914 added to ZOOKEEPER-3030. [exec] Session logged out. Session was JSESSIONID=754122361F8632540FCD1706DDB1BF5A. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1811: exec returned: 2 Total time: 18 minutes 57 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3030 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun Error Message: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. Stack Trace: junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. at java.lang.Thread.run(Thread.java:748)
[jira] [Commented] (ZOOKEEPER-3030) Step 1.3 - Create zk-contrib maven structure
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550914#comment-16550914
]
Hadoop QA commented on ZOOKEEPER-3030:
--
-1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 89 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1)
warnings.
-1 release audit. The applied patch generated 3 release audit warnings
(more than the trunk's current 0 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1978//console
This message is automatically generated.
> Step 1.3 - Create zk-contrib maven structure
>
>
> Key: ZOOKEEPER-3030
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3030
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: build, scripts
>Affects Versions: 3.6.0
>Reporter: Norbert Kalmar
>Assignee: Norbert Kalmar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Create a project structure that separates the different parts of ZooKeeper
> into a more meaningful packages for the future maven build.
> This should be done in iterations to limit the impact.
> * First iteration - safe changes including moving src/docs to zk-docs,
> creating zk-it empty directory. Build and conf directory remains unchanged.
> These changes also have minimum impact on PR’s.
> * Second iteration - move src/recipes to zk-recipes.
> * *Third iteration* - move src/contrib to zk-contrib.
> * Fourth iteration - move src/c to zk-client (java will be moved in Phase 2)
> * Fifth iteration - move jute under src directory
> * Sixth iteration - move src/java/main to zk-server, which will be further
> separated in Step 2.
> {noformat}
> zookeeper
> | -bin
> | -conf
> | -zookeeper-contrib
> || -zookeeper-contrib-fatjar
> || -zookeeper-contrib-huebrowser
> || -zookeeper-contrib-loggraph
> || -zookeeper-contrib-monitoring
> || -zookeeper-contrib-rest
> || -zookeeper-contrib-zkfuse
> || -zookeeper-contrib-zkperl
> || -zookeeper-contrib-zkpython
> || -zookeeper-contrib-zktreeutil
> |\ -zookeeper-contrib-zooinspector
> | -zookeeper-docs
> | -zookeeper-it (integration tests)
> | -zookeeper-recipes
> || -zookeeper-recipes-election
> || -zookeeper-recipes-lock
> \\ -zookeeper-recipes-queue
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ZOOKEEPER-3068) Improve C client logging of IPv6 hosts
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andor Molnar resolved ZOOKEEPER-3068.
-
Resolution: Fixed
Fix Version/s: 3.6.0
Issue resolved by pull request 547
[https://github.com/apache/zookeeper/pull/547]
> Improve C client logging of IPv6 hosts
> --
>
> Key: ZOOKEEPER-3068
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3068
> Project: ZooKeeper
> Issue Type: Improvement
> Components: c client
>Affects Versions: 3.6.0
>Reporter: Brian Nixon
>Assignee: Brian Nixon
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.6.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The C client formats host-port pairings as [host:port] when logging. This is
> visually confusing when the host is an IPv6 address (see the below). In that
> case, it would be preferable to cleanly separate the IPv6 from the port.
> {code:java}
> ZOO_INFO@check_events@2736: initiated connection to server
> [2401:db00:1020:40bf:face:0:5:0:2181]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zookeeper issue #547: ZOOKEEPER-3068: Improve C client logging of IPv6 hosts
Github user anmolnar commented on the issue: https://github.com/apache/zookeeper/pull/547 Committed to master branch. Thanks @enixon ! ---
[GitHub] zookeeper pull request #547: ZOOKEEPER-3068: Improve C client logging of IPv...
Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/547 ---
[jira] [Commented] (ZOOKEEPER-3030) Step 1.3 - Create zk-contrib maven structure
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550850#comment-16550850
]
Hadoop QA commented on ZOOKEEPER-3030:
--
-1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 85 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1)
warnings.
-1 release audit. The applied patch generated 13 release audit warnings
(more than the trunk's current 0 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//console
This message is automatically generated.
> Step 1.3 - Create zk-contrib maven structure
>
>
> Key: ZOOKEEPER-3030
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3030
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: build, scripts
>Affects Versions: 3.6.0
>Reporter: Norbert Kalmar
>Assignee: Norbert Kalmar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Create a project structure that separates the different parts of ZooKeeper
> into a more meaningful packages for the future maven build.
> This should be done in iterations to limit the impact.
> * First iteration - safe changes including moving src/docs to zk-docs,
> creating zk-it empty directory. Build and conf directory remains unchanged.
> These changes also have minimum impact on PR’s.
> * Second iteration - move src/recipes to zk-recipes.
> * *Third iteration* - move src/contrib to zk-contrib.
> * Fourth iteration - move src/c to zk-client (java will be moved in Phase 2)
> * Fifth iteration - move jute under src directory
> * Sixth iteration - move src/java/main to zk-server, which will be further
> separated in Step 2.
> {noformat}
> zookeeper
> | -bin
> | -conf
> | -zookeeper-contrib
> || -zookeeper-contrib-fatjar
> || -zookeeper-contrib-huebrowser
> || -zookeeper-contrib-loggraph
> || -zookeeper-contrib-monitoring
> || -zookeeper-contrib-rest
> || -zookeeper-contrib-zkfuse
> || -zookeeper-contrib-zkperl
> || -zookeeper-contrib-zkpython
> || -zookeeper-contrib-zktreeutil
> |\ -zookeeper-contrib-zooinspector
> | -zookeeper-docs
> | -zookeeper-it (integration tests)
> | -zookeeper-recipes
> || -zookeeper-recipes-election
> || -zookeeper-recipes-lock
> \\ -zookeeper-recipes-queue
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: ZOOKEEPER- PreCommit Build #1977
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 76.91 MB...] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] -1 release audit. The applied patch generated 13 release audit warnings (more than the trunk's current 0 warnings). [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//testReport/ [exec] Release audit warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1977//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment with id 16550850 added to ZOOKEEPER-3030. [exec] Session logged out. Session was JSESSIONID=C24027F61CDD335451971FEA491C7F1E. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1811: exec returned: 2 Total time: 12 minutes 19 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3030 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentServersAreObserversInNextConfig Error Message: waiting for server 4 being up Stack Trace: junit.framework.AssertionFailedError: waiting for server 4 being up at org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentServersAreObserversInNextConfig(ReconfigRecoveryTest.java:224) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
[GitHub] zookeeper issue #574: ZOOKEEPER-3030 - MAVEN MIGRATION - Step 1.3 - move con...
Github user nkalmar commented on the issue: https://github.com/apache/zookeeper/pull/574 So... The problem was that by moving directories from src, ant did not copy them to build/zookeeper-[version]/src/ - now I don't know why it is required here, as it won't be in the jars. I made a script to dif all the filenames in build directory and to diff all the jars (with jar tf output), and the jars do not differ, but the files do. For example, used to be: build/zookeeper-3.6.0-SNAPSHOT/src/recipes/lock/build.xml now: build/zookeeper-3.6.0-SNAPSHOT/src/zookeeper-recipes/zookeeper-recipes-lock/build.xml Again the jar's did not change as packages remained the same! ---
ZooKeeper_branch35_jdk8 - Build # 1049 - Failure
See https://builds.apache.org/job/ZooKeeper_branch35_jdk8/1049/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 10.78 KB...] at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) at hudson.model.Run.execute(Run.java:1819) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8 does not exist. at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:484) at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:461) at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:288) at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:268) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Build step 'Archive the artifacts' marked build as failure Recording test results ERROR: Build step failed with exception Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to H24 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:955) at hudson.FilePath.act(FilePath.java:1036) at hudson.FilePath.act(FilePath.java:1025) at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114) at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:136) at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:166) at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:153) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) at hudson.model.Build$BuildExecution.post2(Build.java:186) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) at hudson.model.Run.execute(Run.java:1819) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8 does not exist. at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:484) at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:461) at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:141) at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Build step 'Publish JUnit test result report' marked build as failure Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if
Re: PreCommit-ZOOKEEPER-github-pr-build failure
Looks like I found the problem, it was a build problem I overwatched. Looking for a fix now. Sorry! On Fri, Jul 20, 2018 at 11:44 AM Norbert Kalmar wrote: > Hi, > > My PR (https://github.com/apache/zookeeper/pull/574) keeps failing on the > pre-commit job. > I see the log: > https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1975/console > > > https://builds.apache.org/blue/organizations/jenkins/PreCommit-ZOOKEEPER-github-pr-build/detail/PreCommit-ZOOKEEPER-github-pr-build/1975/pipeline > > I might be missing something, but I don't see any information pointing me > to why it keeps failing (tried to run it multiple times). > > On my local machine, the build works fine after the change. > > Thanks in advance for the help! > > Regards, > Norbert >
Failed: ZOOKEEPER- PreCommit Build #1976
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1976/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 6.52 KB...] [exec] Pull request title: ZOOKEEPER-3030 - MAVEN MIGRATION - Step 1.3 - move contrib directories % Total% Received % Xferd Average Speed Time Time Time Current [exec] Defect number: ZOOKEEPER-3030 [exec] - Parsed args, going to checkout - [exec] [exec] [exec] == [exec] == [exec] [exec] Dload Upload Total Spent Left SpeedTesting patch for pull request 574. [exec] [exec] [exec] == [exec] == [exec] [exec] [exec] [exec] 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100 1410 1410 0540 0 --:--:-- --:--:-- --:--:-- 542 [exec] [exec] [exec] == [exec] == [exec] Pre-build trunk to verify trunk stability and javac warnings [exec] == [exec] == [exec] [exec] [exec] /home/jenkins/tools/ant/apache-ant-1.9.9/bin/ant -Djavac.args=-Xlint -Xmaxwarns 1000 -Djava5.home=/home/jenkins/tools/java5/latest/ -Dforrest.home=/home/jenkins/tools/forrest/latest/ -DZookeeperPatchProcess= clean tar > /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess/trunkJavacWarnings.txt 2>&1 [exec] Trunk compilation is broken? [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 00 115k0 0 154k 0 --:--:-- --:--:-- --:--:-- 237kmv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1806: exec returned: 1 Total time: 34 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3030 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## No tests ran.
[GitHub] zookeeper issue #574: ZOOKEEPER-3030 - MAVEN MIGRATION - Step 1.3 - move con...
Github user nkalmar commented on the issue: https://github.com/apache/zookeeper/pull/574 The problem with that is that it requires a lot of ant change and the final artifact could change. I agree @tamaashu , that was the plan. I will try to do this with the contrib directory (this PR), and carefully check the end artifacts. ---
PreCommit-ZOOKEEPER-github-pr-build failure
Hi, My PR (https://github.com/apache/zookeeper/pull/574) keeps failing on the pre-commit job. I see the log: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1975/console https://builds.apache.org/blue/organizations/jenkins/PreCommit-ZOOKEEPER-github-pr-build/detail/PreCommit-ZOOKEEPER-github-pr-build/1975/pipeline I might be missing something, but I don't see any information pointing me to why it keeps failing (tried to run it multiple times). On my local machine, the build works fine after the change. Thanks in advance for the help! Regards, Norbert
Failed: ZOOKEEPER- PreCommit Build #1975
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1975/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 6.52 KB...] [exec] [exec] Pull request title: ZOOKEEPER-3030 - MAVEN MIGRATION - Step 1.3 - move contrib directories Dload Upload Total SpentLeft Speed [exec] [exec] [exec] Defect number: ZOOKEEPER-3030 [exec] - Parsed args, going to checkout - [exec] [exec] [exec] == [exec] == [exec] Testing patch for pull request 574. [exec] == [exec] == [exec] [exec] [exec] [exec] 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100 1410 1410 0652 0 --:--:-- --:--:-- --:--:-- 655 [exec] [exec] [exec] == [exec] == [exec] Pre-build trunk to verify trunk stability and javac warnings [exec] == [exec] == [exec] [exec] [exec] /home/jenkins/tools/ant/apache-ant-1.9.9/bin/ant -Djavac.args=-Xlint -Xmaxwarns 1000 -Djava5.home=/home/jenkins/tools/java5/latest/ -Dforrest.home=/home/jenkins/tools/forrest/latest/ -DZookeeperPatchProcess= clean tar > /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess/trunkJavacWarnings.txt 2>&1 [exec] Trunk compilation is broken? [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] 0 00 114k0 0 186k 0 --:--:-- --:--:-- --:--:-- 186kmv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1806: exec returned: 1 Total time: 33 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-3030 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Updated] (ZOOKEEPER-3091) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated ZOOKEEPER-3091: --- Component/s: metric system jmx > Prometheus /metrics http endpoint for monitoring integration > > > Key: ZOOKEEPER-3091 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3091 > Project: ZooKeeper > Issue Type: New Feature > Components: jmx, metric system >Affects Versions: 3.4.6 >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Interesting FB article on CM at scale; LAD
Great article, thanks Pat. There're 2 things which I found particularly interesting: - Is Zeus going to be decod as part of the migration? Are the patches/improvements in it going to be open sources and possibly backported to ZooKeeper? - "Independent data store: ZooKeeper couples a data store with its distribution framework." I believe that decoupling the data store from the coordination framework wouldn't be too much of a hassle in ZooKeeper by introducing a plugin-based data store replacing current ZkDatabase. In addition to that I wonder if we could make client libraries "datastore-aware" letting them serve read requests directly from the database. e.g. from HDFS data nodes. It might successfully decouple the load generated by large number of read requests from writes. In which case will ZooKeeper be in any aspects different from an HDFS cluster? It adds strong consistency and ordering guarantees. Which parts of ZooKeeper do we still need after such change, because the distributed data store guarantees a consistent view of the data tree across participants so what would be the gain of dealing with consensus? Going back to Fb's original problem in terms of data/metadata decoupling: they wanted to distribute large files like 100MB whereas they already 5MB files present in Zk. I think 5MB is already too much data in a single znode and ZooKeeper is not designed to deal with such load of data distribution. So maybe it was originally a use case problem which eventually escalated to a new system design. Propagating metadata updates to millions of subscribers sounds to me more of an IoT / pubsub problem. Which Facebook has presented a very promising answer to. Using a tree of subscribers combined with sharding has very strong scalability potential. Dealing with the distribution of large files triggered by metadata change: bittorrent? :) Regards, Andor On Fri, Jul 20, 2018 at 8:07 AM, Patrick Hunt wrote: > "Location-Aware Distribution (LAD), a new peer-to-peer system that handles > the distribution of configuration changes to millions of servers." > https://code.fb.com/data-infrastructure/location-aware- > distribution-configuring-servers-at-scale/ > > Previously Facebook’s configuration management system (called Configerator) > used ZooKeeper. SOSP 2015 paper: > https://research.fb.com/publications/holistic-configuration-management-at- > facebook/ > > Patrick >
Re: Trying to find pattern in Flaky Tests
Something that's significantly different about the 3.4 and 3.5/master Jenkins jobs is that 3.5/master has test.junit.threads=8 set while this is not supported in 3.4 (see build.xml). It's very likely that the paralyzation of the tests is causing the discrepancy. setting threads > 1 significantly improves the speed of the jobs, that's why it was originally added to 3.5+. See a358280fb2b3cc7852cded3fe67769765a519beb Perhaps we should try one/more of the 3.5/master jobs with threads=1 and see? Patrick On Thu, Jul 19, 2018 at 1:26 PM Molnár Andor wrote: > Sorry guys for this aweful email. Looks like Apache converted my nicely > illustrated email into plain text. :( > > Maybe I could attach the test reports as images, but I think you already > got the idea. > > > Andor > > > > On 07/18/2018 05:42 PM, Andor Molnar wrote: > > Hi, > > > > *branch-3.4* > > > > I've taken a quick look at our Jenkins builds and in terms of flaky > tests, > > it looks like branch-3.4 is in a pretty good shape. The build hasn't > failed > > for 5-6 days on all JDKs which I think is pretty awesome. > > > > *branch-3.5* > > > > This branch is in very bad condition. Which is quite unfortunate given > > we're in the middle of stabilising it. :) > > Especially on JDK8, last successful build was 11 days ago. JDK9 (50% > > failing) and JDK10 (30% failing) are looking better in the last 10 > builds. > > > > Interestingly (apart from a few quite rare ones) it looks there's only 1 > > test which is quite nasty on this branch: testManyChildWatchersAutoReset > > > > There's a Jira about fixing it and a fix has been merged by increasing > the > > timeout of the test, but having a bug on the branch is also possible > > causing the test to fail even with 10 min timeout. > > > > I wasn't able to repro the failing test on my machine (Mac and CentOS7), > it > > always finished in 30-40 seconds maximum. On jenkins slaves it shows the > > following: > > > > *JDK 8:* > > > > Report creation timed out. > > > > > > *JDK 9:* > > > > New Failures > > Chart > > See children > > Build Number ⇒ > > Package-Class-Testmethod names ⇓ > > 351 > > 350 > > 349 > > 348 > > 347 > > 346 > > 345 > > 344 > > 343 > > 342 > > 341 > > 340 > > 339 > > 338 > > 337 > > 336 > > 335 > > 334 > > testManyChildWatchersAutoReset > > 45.604 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/351/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 600.337 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/350/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 21.904 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/349/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 583.063 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/348/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 600.325 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/347/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 600.383 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/346/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 600.362 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/345/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 21.139 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/344/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 24.031 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/343/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 584.200 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/342/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 600.327 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/341/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 600.323 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/340/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 23.737 > > < > https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/339/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset > > > > 600.406 > > < >
Success: ZOOKEEPER- PreCommit Build #1974
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1974/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 79.35 MB...] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 20 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1974//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1974//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1974//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment with id 16550299 added to ZOOKEEPER-2926. [exec] Session logged out. Session was JSESSIONID=A11278B94F8D917CBF45D3FAE72A9DE5. [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD SUCCESSFUL Total time: 17 minutes 49 seconds Archiving artifacts Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Recording test results Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 [description-setter] Description set: ZOOKEEPER-2926 Putting comment on the pull request Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Email was triggered for: Success Sending email for trigger: Success Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 Setting JDK_1_8_LATEST__HOME=/home/jenkins/tools/java/latest1.8 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (ZOOKEEPER-2926) Data inconsistency issue due to the flaw in the session management
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550299#comment-16550299 ] Hadoop QA commented on ZOOKEEPER-2926: -- +1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1974//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1974//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1974//console This message is automatically generated. > Data inconsistency issue due to the flaw in the session management > -- > > Key: ZOOKEEPER-2926 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2926 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Fangmin Lv >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The local session upgrading feature will upgrade the session locally before > receving a quorum commit of creating global session. It's possible that the > server shutdown before the creating session request being sent to leader, if > we retained the ZKDatabase or there is Snapshot happened just before > shutdown, then only this server will have the global session. > If that server didn't become leader, then it will have more global sessions > than others, and those global sessions won't expire as the leader doesn't > know it's existence. If the server became leader, it will accept the client > renew session request and the client is allowed to create ephemeral nodes, > which means other servers only have ephemeral nodes but not that global > session. If there is follower going to have SNAP sync with it, then it will > also have the global session. If the server without that global session > becomes new leader, it will check and delete those dangling ephemeral node > before serving traffic. These could introduce the issues that the ephemeral > node being exist on some servers but not others. > There is dangling global session issue even without local session feature, > because on leader it will update the ZKDatabase when processing > ConnectionRequest and in the PrepRequestProcessor before it's quorum > committed, which also has this risk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zookeeper issue #447: [ZOOKEEPER-2926] Fix potential data consistency issue ...
Github user lvfangmin commented on the issue: https://github.com/apache/zookeeper/pull/447 @anmolnar do you have time to revisit the code here? ---
[jira] [Updated] (ZOOKEEPER-2926) Data inconsistency issue due to the flaw in the session management
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ZOOKEEPER-2926: -- Labels: pull-request-available (was: ) > Data inconsistency issue due to the flaw in the session management > -- > > Key: ZOOKEEPER-2926 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2926 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Fangmin Lv >Priority: Critical > Labels: pull-request-available > > The local session upgrading feature will upgrade the session locally before > receving a quorum commit of creating global session. It's possible that the > server shutdown before the creating session request being sent to leader, if > we retained the ZKDatabase or there is Snapshot happened just before > shutdown, then only this server will have the global session. > If that server didn't become leader, then it will have more global sessions > than others, and those global sessions won't expire as the leader doesn't > know it's existence. If the server became leader, it will accept the client > renew session request and the client is allowed to create ephemeral nodes, > which means other servers only have ephemeral nodes but not that global > session. If there is follower going to have SNAP sync with it, then it will > also have the global session. If the server without that global session > becomes new leader, it will check and delete those dangling ephemeral node > before serving traffic. These could introduce the issues that the ephemeral > node being exist on some servers but not others. > There is dangling global session issue even without local session feature, > because on leader it will update the ZKDatabase when processing > ConnectionRequest and in the PrepRequestProcessor before it's quorum > committed, which also has this risk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Interesting FB article on CM at scale; LAD
"Location-Aware Distribution (LAD), a new peer-to-peer system that handles the distribution of configuration changes to millions of servers." https://code.fb.com/data-infrastructure/location-aware-distribution-configuring-servers-at-scale/ Previously Facebook’s configuration management system (called Configerator) used ZooKeeper. SOSP 2015 paper: https://research.fb.com/publications/holistic-configuration-management-at-facebook/ Patrick
