[jira] [Commented] (ZOOKEEPER-2619) Client library reconnecting breaks FIFO client order
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596910#comment-15596910
]
Diego Ongaro commented on ZOOKEEPER-2619:
-
bq. Callback should be called with ConnectionLossException before createSync()
is send to the server, because internally all requests — sync or async — are
serialized through the same queue. --[~rgs]
I've observed that subsequent requests do not block on the return of the
callback from prior requests. In other words, I can't count on my /data-23857
callback to see and process the ConnectionLoss before my /pointer request is
sent out on the new connection. (And even if I could, that programming model
would be error-prone and burdensome.)
BTW, whether the second create is sync or async isn't important in the example.
The problematic situation could arise even if both were createAsync.
bq. It sounds like the assumption here is that a createSync() should fail if a
previous createAsync() call failed? That should be left to the application, no?
Internally, all replies/events are delivered in order so ordering shouldn't be
broken — no? --[~rgs]
Can you clarify?
> Client library reconnecting breaks FIFO client order
>
>
> Key: ZOOKEEPER-2619
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2619
> Project: ZooKeeper
> Issue Type: Bug
>Reporter: Diego Ongaro
>
> According to the USENIX ATC 2010
> [paper|https://www.usenix.org/conference/usenix-atc-10/zookeeper-wait-free-coordination-internet-scale-systems],
> ZooKeeper provides "FIFO client order: all requests from a given client are
> executed in the order that they were sent by the client." I believe
> applications written using the Java client library are unable to rely on this
> guarantee, and any current application that does so is broken. Other client
> libraries are also likely to be affected.
> Consider this application, which is simplified from the algorithm described
> on Page 4 (right column) of the paper:
> {code}
> zk = new ZooKeeper(...)
> zk.createAsync("/data-23857", "...", callback)
> zk.createSync("/pointer", "/data-23857")
> {code}
> Assume an empty ZooKeeper database to begin with and no other writers.
> Applying the above definition, if the ZooKeeper database contains /pointer,
> it must also contain /data-23857.
> Now consider this series of unfortunate events:
> {code}
> zk = new ZooKeeper(...)
> // The library establishes a TCP connection.
> zk.createAsync("/data-23857", "...", callback)
> // The library/kernel closes the TCP connection because it times out, and
> // the create of /data-23857 is doomed to fail with ConnectionLoss. Suppose
> // that it never reaches the server.
> // The library establishes a new TCP connection.
> zk.createSync("/pointer", "/data-23857")
> // The create of /pointer succeeds.
> {code}
> That's the problem: subsequent operations get assigned to the new connection
> and succeed, while earlier operations fail.
> In general, I believe it's impossible to have a system with the following
> three properties:
> # FIFO client order for asynchronous operations,
> # Failing operations when connections are lost, AND
> # Transparently reconnecting when connections are lost.
> To argue this, consider an application that issues a series of pipelined
> operations, then upon noticing a connection loss, issues a series of recovery
> operations, repeating the recovery procedure as necessary. If a pipelined
> operation fails, all subsequent operations in the pipeline must also fail.
> Yet the client must also carry on eventually: the recovery operations cannot
> be trivially failed forever. Unfortunately, the client library does not know
> where the pipelined operations end and the recovery operations begin. At the
> time of a connection loss, subsequent pipelined operations may or may not be
> queued in the library; others might be upcoming in the application thread. If
> the library re-establishes a connection too early, it will send pipelined
> operations out of FIFO client order.
> I considered a possible workaround of having the client diligently check its
> callbacks and watchers for connection loss events, and do its best to stop
> the subsequent pipelined operations at the first sign of a connection loss.
> In addition to being a large burden for the application, this does not solve
> the problem all the time. In particular, if the callback thread is delayed
> significantly (as can happen due to excessive computation or scheduling
> hiccups), the application may not learn about the connection loss event until
> after the connection has been re-established and after dependent pipelined
> operations have already been transmitted over the new connection.
> I suggest the follow
[jira] [Commented] (ZOOKEEPER-2619) Client library reconnecting breaks FIFO client order
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596822#comment-15596822
]
Raul Gutierrez Segales commented on ZOOKEEPER-2619:
---
[~ongardie]: thanks for reporting this. In the example given though:
{code}
zk = new ZooKeeper(...)
// The library establishes a TCP connection.
zk.createAsync("/data-23857", "...", callback)
// The library/kernel closes the TCP connection because it times out, and
// the create of /data-23857 is doomed to fail with ConnectionLoss. Suppose
// that it never reaches the server.
// The library establishes a new TCP connection.
zk.createSync("/pointer", "/data-23857")
// The create of /pointer succeeds.
{code}
Callback should be called with ConnectionLossException before createSync() is
send to the server, because internally all requests — sync or async — are
serialized through the same queue.
It sounds like the assumption here is that a createSync() should fail if a
previous createAsync() call failed? That should be left to the application, no?
Internally, all replies/events are delivered in order so ordering shouldn't be
broken — no?
> Client library reconnecting breaks FIFO client order
>
>
> Key: ZOOKEEPER-2619
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2619
> Project: ZooKeeper
> Issue Type: Bug
>Reporter: Diego Ongaro
>
> According to the USENIX ATC 2010
> [paper|https://www.usenix.org/conference/usenix-atc-10/zookeeper-wait-free-coordination-internet-scale-systems],
> ZooKeeper provides "FIFO client order: all requests from a given client are
> executed in the order that they were sent by the client." I believe
> applications written using the Java client library are unable to rely on this
> guarantee, and any current application that does so is broken. Other client
> libraries are also likely to be affected.
> Consider this application, which is simplified from the algorithm described
> on Page 4 (right column) of the paper:
> {code}
> zk = new ZooKeeper(...)
> zk.createAsync("/data-23857", "...", callback)
> zk.createSync("/pointer", "/data-23857")
> {code}
> Assume an empty ZooKeeper database to begin with and no other writers.
> Applying the above definition, if the ZooKeeper database contains /pointer,
> it must also contain /data-23857.
> Now consider this series of unfortunate events:
> {code}
> zk = new ZooKeeper(...)
> // The library establishes a TCP connection.
> zk.createAsync("/data-23857", "...", callback)
> // The library/kernel closes the TCP connection because it times out, and
> // the create of /data-23857 is doomed to fail with ConnectionLoss. Suppose
> // that it never reaches the server.
> // The library establishes a new TCP connection.
> zk.createSync("/pointer", "/data-23857")
> // The create of /pointer succeeds.
> {code}
> That's the problem: subsequent operations get assigned to the new connection
> and succeed, while earlier operations fail.
> In general, I believe it's impossible to have a system with the following
> three properties:
> # FIFO client order for asynchronous operations,
> # Failing operations when connections are lost, AND
> # Transparently reconnecting when connections are lost.
> To argue this, consider an application that issues a series of pipelined
> operations, then upon noticing a connection loss, issues a series of recovery
> operations, repeating the recovery procedure as necessary. If a pipelined
> operation fails, all subsequent operations in the pipeline must also fail.
> Yet the client must also carry on eventually: the recovery operations cannot
> be trivially failed forever. Unfortunately, the client library does not know
> where the pipelined operations end and the recovery operations begin. At the
> time of a connection loss, subsequent pipelined operations may or may not be
> queued in the library; others might be upcoming in the application thread. If
> the library re-establishes a connection too early, it will send pipelined
> operations out of FIFO client order.
> I considered a possible workaround of having the client diligently check its
> callbacks and watchers for connection loss events, and do its best to stop
> the subsequent pipelined operations at the first sign of a connection loss.
> In addition to being a large burden for the application, this does not solve
> the problem all the time. In particular, if the callback thread is delayed
> significantly (as can happen due to excessive computation or scheduling
> hiccups), the application may not learn about the connection loss event until
> after the connection has been re-established and after dependent pipelined
> operations have already been transmitted over the new connection.
> I suggest t
ZooKeeper_branch34 - Build # 1696 - Failure
See https://builds.apache.org/job/ZooKeeper_branch34/1696/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 225531 lines...] [junit] 2016-10-22 00:20:47,281 [myid:] - INFO [main:ZooKeeperServer@497] - shutting down [junit] 2016-10-22 00:20:47,282 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-22 00:20:47,282 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2016-10-22 00:20:47,282 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down [junit] 2016-10-22 00:20:47,282 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2016-10-22 00:20:47,282 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2016-10-22 00:20:47,283 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2016-10-22 00:20:47,283 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2016-10-22 00:20:47,284 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2016-10-22 00:20:47,285 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[] [junit] 2016-10-22 00:20:47,287 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2016-10-22 00:20:47,287 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2016-10-22 00:20:47,288 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2016-10-22 00:20:47,288 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2016-10-22 00:20:47,288 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34/build/test/tmp/test2668551843310069307.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34/build/test/tmp/test2668551843310069307.junit.dir/version-2 [junit] 2016-10-22 00:20:47,293 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-22 00:20:47,293 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2016-10-22 00:20:47,294 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:43900 [junit] 2016-10-22 00:20:47,294 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@827] - Processing stat command from /127.0.0.1:43900 [junit] 2016-10-22 00:20:47,295 [myid:] - INFO [Thread-5:NIOServerCnxn$StatCommand@663] - Stat command output [junit] 2016-10-22 00:20:47,295 [myid:] - INFO [Thread-5:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:43900 (no session established for client) [junit] 2016-10-22 00:20:47,295 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2016-10-22 00:20:47,297 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree [junit] 2016-10-22 00:20:47,298 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2016-10-22 00:20:47,298 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port [junit] 2016-10-22 00:20:47,298 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2016-10-22 00:20:47,298 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 31153 [junit] 2016-10-22 00:20:47,299 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2016-10-22 00:20:47,299 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2016-10-22 00:20:47,299 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2016-10-22 00:20:47,362 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x157e9c2e794 closed [junit] 2016-10-22 00:20:47,363 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2016-10-22 00:20:47,363 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x157e9c2e794 [junit] 2016-10-22 00:20:47,363 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method [junit] 2016-10-22 00:20:47,363 [myid:] - INFO
[jira] [Created] (ZOOKEEPER-2619) Client library reconnecting breaks FIFO client order
Diego Ongaro created ZOOKEEPER-2619:
---
Summary: Client library reconnecting breaks FIFO client order
Key: ZOOKEEPER-2619
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2619
Project: ZooKeeper
Issue Type: Bug
Reporter: Diego Ongaro
According to the USENIX ATC 2010
[paper|https://www.usenix.org/conference/usenix-atc-10/zookeeper-wait-free-coordination-internet-scale-systems],
ZooKeeper provides "FIFO client order: all requests from a given client are
executed in the order that they were sent by the client." I believe
applications written using the Java client library are unable to rely on this
guarantee, and any current application that does so is broken. Other client
libraries are also likely to be affected.
Consider this application, which is simplified from the algorithm described on
Page 4 (right column) of the paper:
{code}
zk = new ZooKeeper(...)
zk.createAsync("/data-23857", "...", callback)
zk.createSync("/pointer", "/data-23857")
{code}
Assume an empty ZooKeeper database to begin with and no other writers. Applying
the above definition, if the ZooKeeper database contains /pointer, it must also
contain /data-23857.
Now consider this series of unfortunate events:
{code}
zk = new ZooKeeper(...)
// The library establishes a TCP connection.
zk.createAsync("/data-23857", "...", callback)
// The library/kernel closes the TCP connection because it times out, and
// the create of /data-23857 is doomed to fail with ConnectionLoss. Suppose
// that it never reaches the server.
// The library establishes a new TCP connection.
zk.createSync("/pointer", "/data-23857")
// The create of /pointer succeeds.
{code}
That's the problem: subsequent operations get assigned to the new connection
and succeed, while earlier operations fail.
In general, I believe it's impossible to have a system with the following three
properties:
# FIFO client order for asynchronous operations,
# Failing operations when connections are lost, AND
# Transparently reconnecting when connections are lost.
To argue this, consider an application that issues a series of pipelined
operations, then upon noticing a connection loss, issues a series of recovery
operations, repeating the recovery procedure as necessary. If a pipelined
operation fails, all subsequent operations in the pipeline must also fail. Yet
the client must also carry on eventually: the recovery operations cannot be
trivially failed forever. Unfortunately, the client library does not know where
the pipelined operations end and the recovery operations begin. At the time of
a connection loss, subsequent pipelined operations may or may not be queued in
the library; others might be upcoming in the application thread. If the library
re-establishes a connection too early, it will send pipelined operations out of
FIFO client order.
I considered a possible workaround of having the client diligently check its
callbacks and watchers for connection loss events, and do its best to stop the
subsequent pipelined operations at the first sign of a connection loss. In
addition to being a large burden for the application, this does not solve the
problem all the time. In particular, if the callback thread is delayed
significantly (as can happen due to excessive computation or scheduling
hiccups), the application may not learn about the connection loss event until
after the connection has been re-established and after dependent pipelined
operations have already been transmitted over the new connection.
I suggest the following API changes to fix the problem:
- Add a method ZooKeeper.getConnection() returning a ZKConnection object.
ZKConnection would wrap a TCP connection. It would include all synchronous and
asynchronous operations currently defined on the ZooKeeper class. Upon a
connection loss on a ZKConnection, all subsequent operations on the same
ZKConnection would return a Connection Loss error. Upon noticing, the client
would need to call ZooKeeper.getConnection() again to get a working
ZKConnection object, and it would execute its recovery procedure on this new
connection.
- Deprecate all asynchronous methods on the ZooKeeper object. These are unsafe
to use if the caller assumes they're getting FIFO client order.
- No changes to the protocols or servers are required.
I recognize this could cause a lot of code churn for both ZooKeeper and
projects that use it. On the other hand, the existing asynchronous calls in
applications should now be audited anyhow.
The code affected by this issue may be difficult to contain:
- It likely affects all ZooKeeper client libraries that provide both
asynchronous operations and transparent reconnection. That's probably all
versions of the official Java client library, as well as most other client
libraries.
- It affects all applications using those librari
ZooKeeper-trunk-openjdk7 - Build # 1211 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-openjdk7/1211/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 42 lines...] clean: [echo] contrib: zktreeutil clean: [echo] contrib: ZooInspector clean-recipes: clean: clean: [echo] recipes: election clean: [echo] recipes: lock clean: [echo] recipes: queue clean: init: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/classes [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/lib [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/package/lib [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build/test/lib ivy-download: [get] Getting: https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar [get] To: /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/src/java/lib/ivy-2.4.0.jar [get] Error getting https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar to /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/src/java/lib/ivy-2.4.0.jar BUILD FAILED /home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-openjdk7/build.xml:352: java.net.UnknownHostException: repo1.maven.org at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:635) at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:160) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:934) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) at org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:728) at org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:641) at org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:631) Total time: 6 seconds Build step 'Invoke Ant' marked build as failure Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
ZooKeeper_branch35_solaris - Build # 291 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_solaris/291/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 440371 lines...] [junit] 2016-10-21 17:15:19,906 [myid:] - INFO [main:ClientBase@386] - CREATING server instance 127.0.0.1:11222 [junit] 2016-10-21 17:15:19,907 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. [junit] 2016-10-21 17:15:19,907 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222 [junit] 2016-10-21 17:15:19,908 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:11222 [junit] 2016-10-21 17:15:19,908 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000 [junit] 2016-10-21 17:15:19,908 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 6 [junit] 2016-10-21 17:15:19,909 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test3661996715173879381.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test3661996715173879381.junit.dir/version-2 [junit] 2016-10-21 17:15:19,909 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test3661996715173879381.junit.dir/version-2/snapshot.b [junit] 2016-10-21 17:15:19,911 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0xb to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test3661996715173879381.junit.dir/version-2/snapshot.b [junit] 2016-10-21 17:15:19,912 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-21 17:15:19,913 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 [junit] 2016-10-21 17:15:19,913 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:61138 [junit] 2016-10-21 17:15:19,914 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:61138 [junit] 2016-10-21 17:15:19,914 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output [junit] 2016-10-21 17:15:19,914 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:61138 (no session established for client) [junit] 2016-10-21 17:15:19,914 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2016-10-21 17:15:19,916 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree [junit] 2016-10-21 17:15:19,916 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree [junit] 2016-10-21 17:15:19,916 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port [junit] 2016-10-21 17:15:19,916 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222 [junit] 2016-10-21 17:15:19,916 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 17854 [junit] 2016-10-21 17:15:19,917 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 24 [junit] 2016-10-21 17:15:19,917 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota [junit] 2016-10-21 17:15:19,917 [myid:] - INFO [main:ClientBase@543] - tearDown starting [junit] 2016-10-21 17:15:19,992 [myid:] - INFO [main:ZooKeeper@1313] - Session: 0x12465382b8b closed [junit] 2016-10-21 17:15:19,992 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x12465382b8b [junit] 2016-10-21 17:15:19,992 [myid:] - INFO [main:ClientBase@513] - STOPPING server [junit] 2016-10-21 17:15:19,993 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted [junit] 2016-10-21 17:15:19,993 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2016-10-21 17:15:19,993 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactor
[jira] [Updated] (ZOOKEEPER-1045) Support Quorum Peer mutual authentication via SASL
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated ZOOKEEPER-1045:
Attachment: ZOOKEEPER-1045-br-3-4.patch
I've attached another patch with few changes to the authorization part by
adding {{SecurityUtils#getServerPrincipal()}} "_HOST" replacing logic.
> Support Quorum Peer mutual authentication via SASL
> --
>
> Key: ZOOKEEPER-1045
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> Project: ZooKeeper
> Issue Type: New Feature
> Components: quorum, security
>Reporter: Eugene Koontz
>Assignee: Rakesh R
>Priority: Critical
> Fix For: 3.4.10, 3.5.3
>
> Attachments: 0001-ZOOKEEPER-1045-br-3-4.patch,
> 1045_failing_phunt.tar.gz, HOST_RESOLVER-ZK-1045.patch,
> TEST-org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.txt,
> ZK-1045-test-case-failure-logs.zip, ZOOKEEPER-1045 Test Plan.pdf,
> ZOOKEEPER-1045-00.patch, ZOOKEEPER-1045-Rolling Upgrade Design Proposal.pdf,
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch,
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch,
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch,
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch,
> ZOOKEEPER-1045-br-3-4.patch, ZOOKEEPER-1045-br-3-4.patch,
> ZOOKEEPER-1045TestValidationDesign.pdf,
> org.apache.zookeeper.server.quorum.auth.QuorumAuthUpgradeTest.testRollingUpgrade.log
>
>
> ZOOKEEPER-938 addresses mutual authentication between clients and servers.
> This bug, on the other hand, is for authentication among quorum peers.
> Hopefully much of the work done on SASL integration with Zookeeper for
> ZOOKEEPER-938 can be used as a foundation for this enhancement.
> Review board: https://reviews.apache.org/r/47354/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ZOOKEEPER-2618) fix ClassNotFoundException on shutdown of client
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wu wen updated ZOOKEEPER-2618: -- Description: see ZOOKEEPER-1394 2016-10-21 13:17:21.618 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from localhost-startStop-1-SendThread(172.21.134.7:2005) java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128) Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139) ... 1 more Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325) at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313) ... 3 more was: 2016-10-21 13:17:21.618 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from localhost-startStop-1-SendThread(172.21.134.7:2005) java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128) Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139) ... 1 more Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325) at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313) ... 3 more > fix ClassNotFoundException on shutdown of client > > > Key: ZOOKEEPER-2618 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2618 > Project: ZooKeeper > Issue Type: Sub-task > Components: java client >Affects Versions: 3.4.9 >Reporter: wu wen >Priority: Minor > Attachments: ZOOKEEPER-1394.patch > > > see ZOOKEEPER-1394 > 2016-10-21 13:17:21.618 ERROR > localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from > localhost-startStop-1-SendThread(172.21.134.7:2005) > java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128) > Caused by: java.lang.ClassNotFoundException: Illegal access: this web > application instance has been stopped already. Could not load > [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown > for debugging purposes as well as to attempt to terminate the thread which > caused the illegal access. > at > org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315) > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178) > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139) > ... 1 more > Caused by: java.lang.IllegalStateException: Illegal access: this web > application instance has been stopped already. Could not load > [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown > for debugging purposes as well as to attempt to terminate th
[jira] [Updated] (ZOOKEEPER-2618) fix ClassNotFoundException on shutdown of client
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wu wen updated ZOOKEEPER-2618: -- Description: 2016-10-21 13:17:21.618 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from localhost-startStop-1-SendThread(172.21.134.7:2005) java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128) Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139) ... 1 more Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325) at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313) ... 3 more Summary: fix ClassNotFoundException on shutdown of client (was: fix this) > fix ClassNotFoundException on shutdown of client > > > Key: ZOOKEEPER-2618 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2618 > Project: ZooKeeper > Issue Type: Sub-task > Components: java client >Affects Versions: 3.4.9 >Reporter: wu wen >Priority: Minor > Attachments: ZOOKEEPER-1394.patch > > > 2016-10-21 13:17:21.618 ERROR > localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from > localhost-startStop-1-SendThread(172.21.134.7:2005) > java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128) > Caused by: java.lang.ClassNotFoundException: Illegal access: this web > application instance has been stopped already. Could not load > [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown > for debugging purposes as well as to attempt to terminate the thread which > caused the illegal access. > at > org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315) > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178) > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139) > ... 1 more > Caused by: java.lang.IllegalStateException: Illegal access: this web > application instance has been stopped already. Could not load > [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown > for debugging purposes as well as to attempt to terminate the thread which > caused the illegal access. > at > org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325) > at > org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313) > ... 3 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ZOOKEEPER-2618) fix this
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wu wen updated ZOOKEEPER-2618: -- Attachment: ZOOKEEPER-1394.patch > fix this > > > Key: ZOOKEEPER-2618 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2618 > Project: ZooKeeper > Issue Type: Sub-task > Components: java client >Affects Versions: 3.4.9 >Reporter: wu wen >Priority: Minor > Attachments: ZOOKEEPER-1394.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ZOOKEEPER-2618) fix this
wu wen created ZOOKEEPER-2618: - Summary: fix this Key: ZOOKEEPER-2618 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2618 Project: ZooKeeper Issue Type: Sub-task Components: java client Affects Versions: 3.4.9 Reporter: wu wen Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
ZooKeeper_branch34_openjdk7 - Build # 1251 - Failure
See https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/1251/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 231919 lines...] [junit] 2016-10-21 15:11:27,899 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method [junit] 2016-10-21 15:11:27,899 [myid:] - INFO [main:ZooKeeperServer@497] - shutting down [junit] 2016-10-21 15:11:27,899 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-21 15:11:27,899 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2016-10-21 15:11:27,899 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down [junit] 2016-10-21 15:11:27,899 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2016-10-21 15:11:27,899 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2016-10-21 15:11:27,901 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2016-10-21 15:11:27,901 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2016-10-21 15:11:27,901 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2016-10-21 15:11:27,902 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[] [junit] 2016-10-21 15:11:27,903 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2016-10-21 15:11:27,903 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2016-10-21 15:11:27,903 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2016-10-21 15:11:27,903 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2016-10-21 15:11:27,904 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build/test/tmp/test1998957213012383208.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build/test/tmp/test1998957213012383208.junit.dir/version-2 [junit] 2016-10-21 15:11:27,908 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-21 15:11:27,908 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2016-10-21 15:11:27,908 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:46994 [junit] 2016-10-21 15:11:27,909 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@827] - Processing stat command from /127.0.0.1:46994 [junit] 2016-10-21 15:11:27,909 [myid:] - INFO [Thread-4:NIOServerCnxn$StatCommand@663] - Stat command output [junit] 2016-10-21 15:11:27,909 [myid:] - INFO [Thread-4:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:46994 (no session established for client) [junit] 2016-10-21 15:11:27,909 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2016-10-21 15:11:27,911 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree [junit] 2016-10-21 15:11:27,911 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2016-10-21 15:11:27,911 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port [junit] 2016-10-21 15:11:27,911 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2016-10-21 15:11:27,911 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 19682 [junit] 2016-10-21 15:11:27,911 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2016-10-21 15:11:27,912 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2016-10-21 15:11:27,912 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2016-10-21 15:11:27,986 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x157e7cbfc04 closed [junit] 2016-10-21 15:11:27,986 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2016-10-21 15:11:27,987 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method [junit]
Test plan for ZK-1045 - Call for volunteers
Michael Han posted an update to the test plan to the jira and I want to call the attention of the community to it. It is a big change that we need to be extra careful about because it is supposed to go to the 3.4 branch. It'd be great to have more folks in the community involved with the testing. If you have cycles and interest, please help with testing. -Flavio
ZooKeeper_branch34_solaris - Build # 1333 - Failure
See https://builds.apache.org/job/ZooKeeper_branch34_solaris/1333/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 183903 lines...] [junit] 2016-10-21 13:51:07,897 [myid:] - INFO [main:ZooKeeperServer@497] - shutting down [junit] 2016-10-21 13:51:07,897 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-21 13:51:07,897 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2016-10-21 13:51:07,897 [myid:] - INFO [main:PrepRequestProcessor@765] - Shutting down [junit] 2016-10-21 13:51:07,897 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2016-10-21 13:51:07,897 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2016-10-21 13:51:07,898 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2016-10-21 13:51:07,898 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2016-10-21 13:51:07,898 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2016-10-21 13:51:07,899 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[] [junit] 2016-10-21 13:51:07,899 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2016-10-21 13:51:07,900 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2016-10-21 13:51:07,900 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2016-10-21 13:51:07,900 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2016-10-21 13:51:07,901 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/build/test/tmp/test2196846410270935273.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/build/test/tmp/test2196846410270935273.junit.dir/version-2 [junit] 2016-10-21 13:51:07,903 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-21 13:51:07,903 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2016-10-21 13:51:07,903 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:43865 [junit] 2016-10-21 13:51:07,904 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@827] - Processing stat command from /127.0.0.1:43865 [junit] 2016-10-21 13:51:07,904 [myid:] - INFO [Thread-5:NIOServerCnxn$StatCommand@663] - Stat command output [junit] 2016-10-21 13:51:07,904 [myid:] - INFO [Thread-5:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:43865 (no session established for client) [junit] 2016-10-21 13:51:07,904 [myid:] - INFO [main:JMXEnv@229] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2016-10-21 13:51:07,905 [myid:] - INFO [main:JMXEnv@246] - expect:InMemoryDataTree [junit] 2016-10-21 13:51:07,906 [myid:] - INFO [main:JMXEnv@250] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2016-10-21 13:51:07,906 [myid:] - INFO [main:JMXEnv@246] - expect:StandaloneServer_port [junit] 2016-10-21 13:51:07,906 [myid:] - INFO [main:JMXEnv@250] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2016-10-21 13:51:07,906 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 8868 [junit] 2016-10-21 13:51:07,906 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2016-10-21 13:51:07,906 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2016-10-21 13:51:07,907 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2016-10-21 13:51:07,992 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x157e7826fd2 closed [junit] 2016-10-21 13:51:07,992 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x157e7826fd2 [junit] 2016-10-21 13:51:07,992 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2016-10-21 13:51:07,993 [myid:] - INFO [main:ZooKeeperServer@497] - shutting down [junit] 2016-10-21 13:51:07,994 [myid:] - E
ZooKeeper-trunk-jdk8 - Build # 791 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/791/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 491777 lines...]
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:27380
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[junit] at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit] 2016-10-21 12:04:45,731 [myid:] - INFO [New I/O boss
#66:ClientCnxnSocketNetty@208] - channel is told closing
[junit] 2016-10-21 12:04:45,731 [myid:127.0.0.1:27380] - INFO
[main-SendThread(127.0.0.1:27380):ClientCnxn$SendThread@1231] - channel for
sessionid 0x10006c2ca29 is lost, closing socket connection and attempting
reconnect
[junit] 2016-10-21 12:04:46,248 [myid:127.0.0.1:27501] - INFO
[main-SendThread(127.0.0.1:27501):ClientCnxn$SendThread@1113] - Opening socket
connection to server 127.0.0.1/127.0.0.1:27501. Will not attempt to
authenticate using SASL (unknown error)
[junit] 2016-10-21 12:04:46,249 [myid:] - INFO [New I/O boss
#9438:ClientCnxnSocketNetty$1@127] - future isn't success, cause: {}
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:27501
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[junit] at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit] 2016-10-21 12:04:46,249 [myid:] - WARN [New I/O boss
#9438:ClientCnxnSocketNetty$ZKClientHandler@439] - Exception caught: [id:
0x90870979] EXCEPTION: java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:27501
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:27501
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[junit] at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit] 2016-10-21 12:04:46,249 [myid:]
ZooKeeper_branch35_jdk8 - Build # 274 - Failure
See https://builds.apache.org/job/ZooKeeper_branch35_jdk8/274/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 47 lines...] clean: [echo] contrib: zktreeutil clean: [echo] contrib: ZooInspector clean-recipes: clean: clean: [echo] recipes: election clean: [echo] recipes: lock clean: [echo] recipes: queue clean: init: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/classes [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/lib [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/package/lib [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build/test/lib ivy-download: [get] Getting: https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar [get] To: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/src/java/lib/ivy-2.4.0.jar [get] Error getting https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar to /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/src/java/lib/ivy-2.4.0.jar BUILD FAILED /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk8/build.xml:352: java.net.UnknownHostException: repo1.maven.org at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) at org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:728) at org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:641) at org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:631) Total time: 31 seconds Build step 'Invoke Ant' marked build as failure Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
ZooKeeper-trunk-solaris - Build # 1356 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-solaris/1356/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 477924 lines...] [junit] 2016-10-21 08:59:02,533 [myid:] - INFO [main:ClientBase@386] - CREATING server instance 127.0.0.1:11222 [junit] 2016-10-21 08:59:02,533 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. [junit] 2016-10-21 08:59:02,534 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222 [junit] 2016-10-21 08:59:02,535 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:11222 [junit] 2016-10-21 08:59:02,535 [myid:] - INFO [main:ZooKeeperServer@889] - minSessionTimeout set to 6000 [junit] 2016-10-21 08:59:02,535 [myid:] - INFO [main:ZooKeeperServer@898] - maxSessionTimeout set to 6 [junit] 2016-10-21 08:59:02,535 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test5893088625624667747.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test5893088625624667747.junit.dir/version-2 [junit] 2016-10-21 08:59:02,536 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test5893088625624667747.junit.dir/version-2/snapshot.b [junit] 2016-10-21 08:59:02,538 [myid:] - INFO [main:FileTxnSnapLog@306] - Snapshotting: 0xb to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test5893088625624667747.junit.dir/version-2/snapshot.b [junit] 2016-10-21 08:59:02,540 [myid:] - ERROR [main:ZooKeeperServer@501] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2016-10-21 08:59:02,540 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 [junit] 2016-10-21 08:59:02,540 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:36370 [junit] 2016-10-21 08:59:02,541 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:36370 [junit] 2016-10-21 08:59:02,541 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output [junit] 2016-10-21 08:59:02,542 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:36370 (no session established for client) [junit] 2016-10-21 08:59:02,542 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2016-10-21 08:59:02,543 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree [junit] 2016-10-21 08:59:02,543 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree [junit] 2016-10-21 08:59:02,544 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port [junit] 2016-10-21 08:59:02,544 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222 [junit] 2016-10-21 08:59:02,544 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 17665 [junit] 2016-10-21 08:59:02,544 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 24 [junit] 2016-10-21 08:59:02,544 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota [junit] 2016-10-21 08:59:02,544 [myid:] - INFO [main:ClientBase@543] - tearDown starting [junit] 2016-10-21 08:59:02,623 [myid:] - INFO [main:ZooKeeper@1315] - Session: 0x1246371ca94 closed [junit] 2016-10-21 08:59:02,623 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x1246371ca94 [junit] 2016-10-21 08:59:02,624 [myid:] - INFO [main:ClientBase@513] - STOPPING server [junit] 2016-10-21 08:59:02,625 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted [junit] 2016-10-21 08:59:02,625 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2016-10-21 08:59:02,625 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selecto
