[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877512#comment-15877512
]
Rakesh R commented on ZOOKEEPER-2693:
-
Thanks [~hanm] for making good progress.
{quote}
I will once I get this landed in 3.5. PR to 3.4 will not be much different, but
I'd like to finalize this PR first to avoid potential duplicated efforts.
Meanwhile, I'll create a set of follow JIRAs to address concerns of abusing 4lw
in general:
{quote}
Makes sense to me. If I understand your comment correctly, we could push
{{whitelist config}} changes first which will unblock 3.4.10 releasing. But for
3.5.3 releasing, we need both {{whitelist config}} and {{deprecating 4lw
tasks}} together. Probably, {{4lw rate limiting task}} is nice to have
feature(keep it low priority compare to other tasks), right?
> DOS attack on wchp/wchc four letter words (4lw)
> ---
>
> Key: ZOOKEEPER-2693
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693
> Project: ZooKeeper
> Issue Type: Bug
> Components: security, server
>Affects Versions: 3.4.0, 3.5.1, 3.5.2
>Reporter: Patrick Hunt
>Assignee: Michael Han
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2693-01.patch
>
>
> The wchp/wchc four letter words can be exploited in a DOS attack on the ZK
> client port - typically 2181. The following POC attack was recently published
> on the web:
> https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us
> The most straightforward way to block this attack is to not allow access to
> the client port to non-trusted clients - i.e. firewall the ZooKeeper service
> and only allow access to trusted applications using it for coordination.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2696) Eclipse ant task no longer determines correct classpath for tests after ZOOKEEPER-2689
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877462#comment-15877462 ] ASF GitHub Bot commented on ZOOKEEPER-2696: --- Github user rakeshadr commented on the issue: https://github.com/apache/zookeeper/pull/178 > I did not add the transitive="false" because we are not using it for the default configuration on the previous line. If you feel strongly about it I will add it. As we know, earlier `transitive=false` in build.xml avoids pulling these dependencies. IMHO, we can add `transitive=false` in checkstyle because the related libraries are not needed now. > Eclipse ant task no longer determines correct classpath for tests after > ZOOKEEPER-2689 > -- > > Key: ZOOKEEPER-2696 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2696 > Project: ZooKeeper > Issue Type: Bug >Affects Versions: 3.4.10 >Reporter: Abraham Fine >Assignee: Abraham Fine > > Following the changes made in ZOOKEEPER-2689 IDE's using the .classpath file > generated by the eclipse ant task (I tested both idea and eclipse) cannot > compile the tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] zookeeper issue #178: ZOOKEEPER-2696: Eclipse ant ask no longer determines c...
Github user rakeshadr commented on the issue: https://github.com/apache/zookeeper/pull/178 > I did not add the transitive="false" because we are not using it for the default configuration on the previous line. If you feel strongly about it I will add it. As we know, earlier `transitive=false` in build.xml avoids pulling these dependencies. IMHO, we can add `transitive=false` in checkstyle because the related libraries are not needed now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877343#comment-15877343
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102372975
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@hanm Thanks for your advice :)
We are using a similar structure as `2.2 Alternative 2 – Learners` in the
paper, it is good enough for most of online scenes. I don't think it is worthy
to introduce another layer because we give up the write operation to all
observers, just use it as a read only view.
For the remote replication, I doubt it also blocked by packet loss rate
like Observer. On the other hand, we have an internal project named `zkpipe`,
it read Zookeeper snapshot/binlog and send it to a Kafka topic, our client
could choose to rebuild the transaction or subscribe the changes. I believe it
will better than hack Zookeeper itself. If you have interested, I could push it
to github later.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102372975
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@hanm Thanks for your advice :)
We are using a similar structure as `2.2 Alternative 2 â Learners` in the
paper, it is good enough for most of online scenes. I don't think it is worthy
to introduce another layer because we give up the write operation to all
observers, just use it as a read only view.
For the remote replication, I doubt it also blocked by packet loss rate
like Observer. On the other hand, we have an internal project named `zkpipe`,
it read Zookeeper snapshot/binlog and send it to a Kafka topic, our client
could choose to rebuild the transaction or subscribe the changes. I believe it
will better than hack Zookeeper itself. If you have interested, I could push it
to github later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
ZooKeeper_branch34_jdk7 - Build # 1415 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1415/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 31.34 MB...] [junit] 2017-02-22 02:42:03,073 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2017-02-22 02:42:03,074 [myid:] - INFO [main:PrepRequestProcessor@764] - Shutting down [junit] 2017-02-22 02:42:03,074 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2017-02-22 02:42:03,074 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2017-02-22 02:42:03,075 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2017-02-22 02:42:03,075 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2017-02-22 02:42:03,076 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-22 02:42:03,076 [myid:] - INFO [main:JMXEnv@147] - ensureOnly:[] [junit] 2017-02-22 02:42:03,078 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2017-02-22 02:42:03,078 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2017-02-22 02:42:03,079 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2017-02-22 02:42:03,079 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2017-02-22 02:42:03,079 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/build/test/tmp/test2691973495394559611.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/build/test/tmp/test2691973495394559611.junit.dir/version-2 [junit] 2017-02-22 02:42:03,084 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-22 02:42:03,084 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-22 02:42:03,084 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:36344 [junit] 2017-02-22 02:42:03,085 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@838] - Processing stat command from /127.0.0.1:36344 [junit] 2017-02-22 02:42:03,086 [myid:] - INFO [Thread-4:NIOServerCnxn$StatCommand@674] - Stat command output [junit] 2017-02-22 02:42:03,086 [myid:] - INFO [Thread-4:NIOServerCnxn@1019] - Closed socket connection for client /127.0.0.1:36344 (no session established for client) [junit] 2017-02-22 02:42:03,086 [myid:] - INFO [main:JMXEnv@230] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-22 02:42:03,089 [myid:] - INFO [main:JMXEnv@247] - expect:InMemoryDataTree [junit] 2017-02-22 02:42:03,089 [myid:] - INFO [main:JMXEnv@251] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2017-02-22 02:42:03,089 [myid:] - INFO [main:JMXEnv@247] - expect:StandaloneServer_port [junit] 2017-02-22 02:42:03,089 [myid:] - INFO [main:JMXEnv@251] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2017-02-22 02:42:03,090 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 35675 [junit] 2017-02-22 02:42:03,090 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2017-02-22 02:42:03,090 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2017-02-22 02:42:03,090 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2017-02-22 02:42:03,154 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x15a63b27028 closed [junit] 2017-02-22 02:42:03,154 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2017-02-22 02:42:03,155 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x15a63b27028 [junit] 2017-02-22 02:42:03,155 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method [junit] 2017-02-22 02:42:03,155 [myid:] - INFO [main:ZooKeeperServer@505] - shutting down [junit] 2017-02-22 02:42:03,155 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-22 02:42:03,156
ZooKeeper_branch34 - Build # 1844 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34/1844/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 3.61 KB...] collisions:0 txqueuelen:0 RX bytes:747102 (747.1 KB) TX bytes:27965662 (27.9 MB) core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 386178 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 6 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 10240 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 6 /home/jenkins/tools/ant/latest/bin/ant clean Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) mv: cannot stat 'build/*.tar.gz': No such file or directory mv: cannot stat 'build/*.jar': No such file or directory mv: cannot stat 'build/test/findbugs': No such file or directory mv: cannot stat 'build/docs/api': No such file or directory Build Failed Build step 'Execute shell' marked build as failure Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
ZooKeeper_branch34_jdk8 - Build # 892 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_jdk8/892/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 32.41 MB...] [junit] 2017-02-22 00:49:45,074 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2017-02-22 00:49:45,074 [myid:] - INFO [main:PrepRequestProcessor@764] - Shutting down [junit] 2017-02-22 00:49:45,075 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2017-02-22 00:49:45,075 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2017-02-22 00:49:45,075 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2017-02-22 00:49:45,075 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2017-02-22 00:49:45,076 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-22 00:49:45,076 [myid:] - INFO [main:JMXEnv@147] - ensureOnly:[] [junit] 2017-02-22 00:49:45,078 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2017-02-22 00:49:45,078 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2017-02-22 00:49:45,078 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2017-02-22 00:49:45,079 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2017-02-22 00:49:45,079 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/build/test/tmp/test482003377656285.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk8/build/test/tmp/test482003377656285.junit.dir/version-2 [junit] 2017-02-22 00:49:45,082 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-22 00:49:45,082 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-22 00:49:45,083 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:33819 [junit] 2017-02-22 00:49:45,083 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@838] - Processing stat command from /127.0.0.1:33819 [junit] 2017-02-22 00:49:45,083 [myid:] - INFO [Thread-4:NIOServerCnxn$StatCommand@674] - Stat command output [junit] 2017-02-22 00:49:45,084 [myid:] - INFO [Thread-4:NIOServerCnxn@1019] - Closed socket connection for client /127.0.0.1:33819 (no session established for client) [junit] 2017-02-22 00:49:45,084 [myid:] - INFO [main:JMXEnv@230] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-22 00:49:45,086 [myid:] - INFO [main:JMXEnv@247] - expect:InMemoryDataTree [junit] 2017-02-22 00:49:45,086 [myid:] - INFO [main:JMXEnv@251] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2017-02-22 00:49:45,086 [myid:] - INFO [main:JMXEnv@247] - expect:StandaloneServer_port [junit] 2017-02-22 00:49:45,086 [myid:] - INFO [main:JMXEnv@251] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2017-02-22 00:49:45,087 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 35517 [junit] 2017-02-22 00:49:45,087 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2017-02-22 00:49:45,087 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2017-02-22 00:49:45,087 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2017-02-22 00:49:45,158 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x15a634b9fd8 closed [junit] 2017-02-22 00:49:45,158 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2017-02-22 00:49:45,158 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x15a634b9fd8 [junit] 2017-02-22 00:49:45,159 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method [junit] 2017-02-22 00:49:45,159 [myid:] - INFO [main:ZooKeeperServer@505] - shutting down [junit] 2017-02-22 00:49:45,159 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-22 00:49:45,159
ZooKeeper-trunk - Build # 3288 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk/3288/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 9.27 KB...] POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 10240 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 6 /home/jenkins/tools/ant/latest/bin/ant clean Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) mv: cannot stat 'build/*.tar.gz': No such file or directory mv: cannot stat 'build/*.jar': No such file or directory mv: cannot stat 'build/test/findbugs': No such file or directory mv: cannot stat 'build/docs/api': No such file or directory Build Failed Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE [WARNINGS] Skipping publisher since build result is FAILURE Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording fingerprints Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Publishing Javadoc Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876999#comment-15876999 ] Hadoop QA commented on ZOOKEEPER-2693: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/366//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/366//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/366//console This message is automatically generated. > DOS attack on wchp/wchc four letter words (4lw) > --- > > Key: ZOOKEEPER-2693 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693 > Project: ZooKeeper > Issue Type: Bug > Components: security, server >Affects Versions: 3.4.0, 3.5.1, 3.5.2 >Reporter: Patrick Hunt >Assignee: Michael Han >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2693-01.patch > > > The wchp/wchc four letter words can be exploited in a DOS attack on the ZK > client port - typically 2181. The following POC attack was recently published > on the web: > https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us > The most straightforward way to block this attack is to not allow access to > the client port to non-trusted clients - i.e. firewall the ZooKeeper service > and only allow access to trusted applications using it for coordination. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Failed: ZOOKEEPER- PreCommit Build #366
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/366/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 68.30 MB...] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/366//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/366//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/366//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] aa8629874527b4910e2e8877e2ab47d9e3f772ba logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1635: exec returned: 1 Total time: 16 minutes 4 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-2693 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876971#comment-15876971 ] ASF GitHub Bot commented on ZOOKEEPER-2693: --- Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/179 Update patch to address review comments from @arshadmohammad and @rakeshadr * Added a nop command that does nothing but print message back to client for better user experience. * Test case update. I think the remaining issue is to decide how to deal with isro and srvr which is being discussed on jira. > DOS attack on wchp/wchc four letter words (4lw) > --- > > Key: ZOOKEEPER-2693 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693 > Project: ZooKeeper > Issue Type: Bug > Components: security, server >Affects Versions: 3.4.0, 3.5.1, 3.5.2 >Reporter: Patrick Hunt >Assignee: Michael Han >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2693-01.patch > > > The wchp/wchc four letter words can be exploited in a DOS attack on the ZK > client port - typically 2181. The following POC attack was recently published > on the web: > https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us > The most straightforward way to block this attack is to not allow access to > the client port to non-trusted clients - i.e. firewall the ZooKeeper service > and only allow access to trusted applications using it for coordination. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] zookeeper issue #179: ZOOKEEPER-2693: DOS attack on wchp/wchc four letter wo...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/179 Update patch to address review comments from @arshadmohammad and @rakeshadr * Added a nop command that does nothing but print message back to client for better user experience. * Test case update. I think the remaining issue is to decide how to deal with isro and srvr which is being discussed on jira. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876881#comment-15876881 ] Michael Han commented on ZOOKEEPER-2693: bq. srvr is used in zookeeper/bin/zkServer.sh status bq. isro is used in org.apache.zookeeper.ClientCnxn.SendThread.pingRwServer() Good catch [~arshadmohammad] - I hope this is an exhaustive list of 4lw used by ZK :) are there other commands used by ZK itself if you may know? Read only server is disabled by default, so we can leave isro out of white list by default and document in admin manual that if read only server is enabled, this command must be put back in white list. We can use a separate JIRA to get ride of isro from ZooKeeper client library later. For srvr, it is only used in zkServer.sh's stat option - not sure if anyone actually use this feature but we could just remove the Stat option from zkServer.sh so we don't have to include srvr in whitelist. Another option is to include srvr in white list by default for 3.4/3.5. I think include it by default in whitelist sounds the way to go from a compatibility point of view. > DOS attack on wchp/wchc four letter words (4lw) > --- > > Key: ZOOKEEPER-2693 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693 > Project: ZooKeeper > Issue Type: Bug > Components: security, server >Affects Versions: 3.4.0, 3.5.1, 3.5.2 >Reporter: Patrick Hunt >Assignee: Michael Han >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2693-01.patch > > > The wchp/wchc four letter words can be exploited in a DOS attack on the ZK > client port - typically 2181. The following POC attack was recently published > on the web: > https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us > The most straightforward way to block this attack is to not allow access to > the client port to non-trusted clients - i.e. firewall the ZooKeeper service > and only allow access to trusted applications using it for coordination. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876870#comment-15876870 ] Michael Han commented on ZOOKEEPER-2693: bq. IIUC, these are two problems -> case-1) restrict 4lw cmd execution as few cmds taking too much time for execution. case-2) protection against overuse because it creates many connections. Yes, this is a good summary. Two problems - one is to fix the obvious exploits related to watcher 4lw and the other is to prevent abuse of 4lw in general. This JIRA's scope is targeting the first one, which fixes immediate issue and unblocks two important ongoing releases. We can easily get out of scope if we want to completely fix the security of the 4lw which was not designed with security in mind while balancing compatibility and minimize disrupt to existing users, so I'd recommend we stick to the current scope (unless as I mentioned earlier folks feel strongly against the white list approach.). bq. could you create a PR for branch-3.4 I will once I get this landed in 3.5. PR to 3.4 will not be much different, but I'd like to finalize this PR first to avoid potential duplicated efforts. Meanwhile, I'll create a set of follow JIRAs to address concerns of abusing 4lw in general: * A new config option to turn on / off 4lw w/o a middle ground (sure we can use empty white list for this purpose but a separate option is better IMO from the point of view of deprecating a feature.). * 4lw rate limiting including concurrent command runs configuration. * Fix client / script to avoid using 4lw - it is unfortunate ZK itself depends on 4lw. > DOS attack on wchp/wchc four letter words (4lw) > --- > > Key: ZOOKEEPER-2693 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693 > Project: ZooKeeper > Issue Type: Bug > Components: security, server >Affects Versions: 3.4.0, 3.5.1, 3.5.2 >Reporter: Patrick Hunt >Assignee: Michael Han >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2693-01.patch > > > The wchp/wchc four letter words can be exploited in a DOS attack on the ZK > client port - typically 2181. The following POC attack was recently published > on the web: > https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us > The most straightforward way to block this attack is to not allow access to > the client port to non-trusted clients - i.e. firewall the ZooKeeper service > and only allow access to trusted applications using it for coordination. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (ZOOKEEPER-2692) Fix race condition in testWatchAutoResetWithPending
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Han resolved ZOOKEEPER-2692. Resolution: Fixed > Fix race condition in testWatchAutoResetWithPending > --- > > Key: ZOOKEEPER-2692 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2692 > Project: ZooKeeper > Issue Type: Sub-task > Components: tests >Affects Versions: 3.4.9, 3.5.3, 3.6.0 >Reporter: Abraham Fine >Assignee: Abraham Fine > Fix For: 3.4.10, 3.5.3, 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2692) Fix race condition in testWatchAutoResetWithPending
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876772#comment-15876772 ] Michael Han commented on ZOOKEEPER-2692: [~abrahamfine] I haven't looked into details on these tests but looks like your change was not impacting the failures here. Though I am pretty sure this flaky test is new, it sounds good to close this jira and open another one to track it. > Fix race condition in testWatchAutoResetWithPending > --- > > Key: ZOOKEEPER-2692 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2692 > Project: ZooKeeper > Issue Type: Sub-task > Components: tests >Affects Versions: 3.4.9, 3.5.3, 3.6.0 >Reporter: Abraham Fine >Assignee: Abraham Fine > Fix For: 3.4.10, 3.5.3, 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
ZooKeeper-trunk-openjdk7 - Build # 1374 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-openjdk7/1374/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 61.54 MB...] [junit] at java.lang.Thread.run(Thread.java:745) [junit] 2017-02-21 20:02:10,459 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port22240,name1=Connections,name2=127.0.0.1,name3=0x1028d37f8e8] [junit] 2017-02-21 20:02:10,560 [myid:] - INFO [main:ZooKeeper@1324] - Session: 0x1028d37f8e8 closed [junit] 2017-02-21 20:02:10,560 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x1028d37f8e8 [junit] 2017-02-21 20:02:10,560 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 217326 [junit] 2017-02-21 20:02:10,560 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 2426 [junit] 2017-02-21 20:02:10,560 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testWatcherAutoResetWithLocal [junit] 2017-02-21 20:02:10,560 [myid:] - INFO [main:ClientBase@558] - tearDown starting [junit] 2017-02-21 20:02:10,561 [myid:] - INFO [main:ClientBase@528] - STOPPING server [junit] 2017-02-21 20:02:10,561 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:22240 [junit] 2017-02-21 20:02:10,570 [myid:] - INFO [main:ZooKeeperServer@542] - shutting down [junit] 2017-02-21 20:02:10,570 [myid:] - ERROR [main:ZooKeeperServer@506] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-21 20:02:10,570 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down [junit] 2017-02-21 20:02:10,570 [myid:] - INFO [main:PrepRequestProcessor@1008] - Shutting down [junit] 2017-02-21 20:02:10,570 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down [junit] 2017-02-21 20:02:10,570 [myid:] - INFO [ProcessThread(sid:0 cport:22240)::PrepRequestProcessor@157] - PrepRequestProcessor exited loop! [junit] 2017-02-21 20:02:10,571 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited! [junit] 2017-02-21 20:02:10,571 [myid:] - INFO [main:FinalRequestProcessor@481] - shutdown of request processor complete [junit] 2017-02-21 20:02:10,571 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port22240,name1=InMemoryDataTree] [junit] 2017-02-21 20:02:10,572 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port22240] [junit] 2017-02-21 20:02:10,572 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 22240 [junit] 2017-02-21 20:02:10,572 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[] [junit] 2017-02-21 20:02:10,581 [myid:] - INFO [main:ClientBase@583] - fdcount after test is: 7130 at start it was 7130 [junit] 2017-02-21 20:02:10,582 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testWatcherAutoResetWithLocal [junit] 2017-02-21 20:02:10,582 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testWatcherAutoResetWithLocal [junit] Tests run: 103, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 482.079 sec, Thread: 5, Class: org.apache.zookeeper.test.NioNettySuiteTest [junit] 2017-02-21 20:02:10,814 [myid:127.0.0.1:22117] - INFO [main-SendThread(127.0.0.1:22117):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:22117. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-02-21 20:02:10,815 [myid:127.0.0.1:22117] - WARN [main-SendThread(127.0.0.1:22117):ClientCnxn$SendThread@1235] - Session 0x1028d348963 for server 127.0.0.1/127.0.0.1:22117, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-02-21 20:02:10,927 [myid:127.0.0.1:22123] - INFO [main-SendThread(127.0.0.1:22123):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:22123. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-02-21 20:02:10,927 [myid:127.0.0.1:22123] - WARN [main-SendThread(127.0.0.1:22123):ClientCnxn$SendThread@1235] - Session 0x3028d348968 for server 127.0.0.1/127.0.0.1:22123,
[GitHub] zookeeper pull request #179: ZOOKEEPER-2693: DOS attack on wchp/wchc four le...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/179#discussion_r102301090
--- Diff: src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java ---
@@ -267,10 +267,17 @@ private boolean checkFourLetterWord(final Channel
channel,
{
// We take advantage of the limited size of the length to look
// for cmds. They are all 4-bytes which fits inside of an int
-String cmd = FourLetterCommands.getCmdMapView().get(len);
-if (cmd == null) {
+if (!FourLetterCommands.isKnown(len)) {
return false;
}
+
+// ZOOKEEPER-2693: don't execute 4lw if it's not enabled.
+String cmd = FourLetterCommands.getCommandString(len);
+if (!FourLetterCommands.isEnabled(cmd)) {
+LOG.debug("Command {} is not executed because it is not white
listed.", cmd);
+return true;
--- End diff --
@rakeshadr Turns out server socket not closed is a by design. Had a chat
with @phunt offline and the idea is we always prefer client to close socket
first (which led to server socket close) because a socket close at server might
be premature and led to client not getting all the data due to how TCP works.
I'll leave these code as they are.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876587#comment-15876587
]
ASF GitHub Bot commented on ZOOKEEPER-2693:
---
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/179#discussion_r102301090
--- Diff: src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java ---
@@ -267,10 +267,17 @@ private boolean checkFourLetterWord(final Channel
channel,
{
// We take advantage of the limited size of the length to look
// for cmds. They are all 4-bytes which fits inside of an int
-String cmd = FourLetterCommands.getCmdMapView().get(len);
-if (cmd == null) {
+if (!FourLetterCommands.isKnown(len)) {
return false;
}
+
+// ZOOKEEPER-2693: don't execute 4lw if it's not enabled.
+String cmd = FourLetterCommands.getCommandString(len);
+if (!FourLetterCommands.isEnabled(cmd)) {
+LOG.debug("Command {} is not executed because it is not white
listed.", cmd);
+return true;
--- End diff --
@rakeshadr Turns out server socket not closed is a by design. Had a chat
with @phunt offline and the idea is we always prefer client to close socket
first (which led to server socket close) because a socket close at server might
be premature and led to client not getting all the data due to how TCP works.
I'll leave these code as they are.
> DOS attack on wchp/wchc four letter words (4lw)
> ---
>
> Key: ZOOKEEPER-2693
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693
> Project: ZooKeeper
> Issue Type: Bug
> Components: security, server
>Affects Versions: 3.4.0, 3.5.1, 3.5.2
>Reporter: Patrick Hunt
>Assignee: Michael Han
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2693-01.patch
>
>
> The wchp/wchc four letter words can be exploited in a DOS attack on the ZK
> client port - typically 2181. The following POC attack was recently published
> on the web:
> https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us
> The most straightforward way to block this attack is to not allow access to
> the client port to non-trusted clients - i.e. firewall the ZooKeeper service
> and only allow access to trusted applications using it for coordination.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (ZOOKEEPER-2692) Fix race condition in testWatchAutoResetWithPending
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876550#comment-15876550 ] Abraham Fine edited comment on ZOOKEEPER-2692 at 2/21/17 7:32 PM: -- [~hanm] I am not convinced that this is a regression. Although the timing of the failure seems to indicate a regression, the WatcherTest class modified in ZOOKEEPER-2692 is unrelated to WatchEventWhenAutoResetTest. In addition we see this failure happening before this patch was committed here: https://builds.apache.org/job/ZooKeeper-trunk-solaris/1505/ Perhaps this deserves its own JIRA? was (Author: abrahamfine): [~hanm] I am not convinced that this is a regression. Although the timing of the failure seems to indicate a regression, the WatcherTest class modified in ZOOKEEPER-2686 is unrelated to WatchEventWhenAutoResetTest. In addition we see this failure happening before this patch was committed here: https://builds.apache.org/job/ZooKeeper-trunk-solaris/1505/ Perhaps this deserves its own JIRA? > Fix race condition in testWatchAutoResetWithPending > --- > > Key: ZOOKEEPER-2692 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2692 > Project: ZooKeeper > Issue Type: Sub-task > Components: tests >Affects Versions: 3.4.9, 3.5.3, 3.6.0 >Reporter: Abraham Fine >Assignee: Abraham Fine > Fix For: 3.4.10, 3.5.3, 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2692) Fix race condition in testWatchAutoResetWithPending
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876550#comment-15876550 ] Abraham Fine commented on ZOOKEEPER-2692: - [~hanm] I am not convinced that this is a regression. Although the timing of the failure seems to indicate a regression, the WatcherTest class modified in ZOOKEEPER-2686 is unrelated to WatchEventWhenAutoResetTest. In addition we see this failure happening before this patch was committed here: https://builds.apache.org/job/ZooKeeper-trunk-solaris/1505/ Perhaps this deserves its own JIRA? > Fix race condition in testWatchAutoResetWithPending > --- > > Key: ZOOKEEPER-2692 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2692 > Project: ZooKeeper > Issue Type: Sub-task > Components: tests >Affects Versions: 3.4.9, 3.5.3, 3.6.0 >Reporter: Abraham Fine >Assignee: Abraham Fine > Fix For: 3.4.10, 3.5.3, 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2696) Eclipse ant task no longer determines correct classpath for tests after ZOOKEEPER-2689
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876464#comment-15876464 ] ASF GitHub Bot commented on ZOOKEEPER-2696: --- Github user afine commented on the issue: https://github.com/apache/zookeeper/pull/178 I did not add the `transitive="false"` because we are not using it for the `default` configuration on the previous line. If you feel strongly about it I will add it. In addition, since this is only used for setting up dev environments I think it is probably ok to pull in a few dependencies that we do not need. > Eclipse ant task no longer determines correct classpath for tests after > ZOOKEEPER-2689 > -- > > Key: ZOOKEEPER-2696 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2696 > Project: ZooKeeper > Issue Type: Bug >Affects Versions: 3.4.10 >Reporter: Abraham Fine >Assignee: Abraham Fine > > Following the changes made in ZOOKEEPER-2689 IDE's using the .classpath file > generated by the eclipse ant task (I tested both idea and eclipse) cannot > compile the tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] zookeeper issue #178: ZOOKEEPER-2696: Eclipse ant ask no longer determines c...
Github user afine commented on the issue: https://github.com/apache/zookeeper/pull/178 I did not add the `transitive="false"` because we are not using it for the `default` configuration on the previous line. If you feel strongly about it I will add it. In addition, since this is only used for setting up dev environments I think it is probably ok to pull in a few dependencies that we do not need. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102283801
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@flier Having SyncRequestProcessor look at the number of snapshot files
feels like a bit of a hack, but I am not expert here. I would be fine with
having SyncRequestProcessor delegate all throttling of snapshots to
ZooKeeperServer, but then we need some sort of synchronization to prevent a
snapshot from being taken when it is not in a proper state (during the ZABP
sync phase).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876461#comment-15876461
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102283801
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@flier Having SyncRequestProcessor look at the number of snapshot files
feels like a bit of a hack, but I am not expert here. I would be fine with
having SyncRequestProcessor delegate all throttling of snapshots to
ZooKeeperServer, but then we need some sort of synchronization to prevent a
snapshot from being taken when it is not in a proper state (during the ZABP
sync phase).
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102282461
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
Cross DC deployment is an interesting topic and ZooKeeper does not
intrinsically support it very well. Not saying that you don't need your snap
command (I understand it is a quick and dirty way to get things working for
your case), but here is some design that you might find useful for your
deployment:
https://www.usenix.org/system/files/conference/atc16/atc16_paper-lev-ari.pdf
The basic idea of this is to partition your data to have multiple ZK
ensembles (this loses global strong consistency) and then patch global
consistency at client side. The client library is open sourced somewhere.
https://issues.apache.org/jira/browse/ZOOKEEPER-892
This is an old issue that no one driving at moment but sounds a good fit
for your use case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876447#comment-15876447
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102282461
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
Cross DC deployment is an interesting topic and ZooKeeper does not
intrinsically support it very well. Not saying that you don't need your snap
command (I understand it is a quick and dirty way to get things working for
your case), but here is some design that you might find useful for your
deployment:
https://www.usenix.org/system/files/conference/atc16/atc16_paper-lev-ari.pdf
The basic idea of this is to partition your data to have multiple ZK
ensembles (this loses global strong consistency) and then patch global
consistency at client side. The client library is open sourced somewhere.
https://issues.apache.org/jira/browse/ZOOKEEPER-892
This is an old issue that no one driving at moment but sounds a good fit
for your use case.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876406#comment-15876406
]
ASF GitHub Bot commented on ZOOKEEPER-2693:
---
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/179#discussion_r102275734
--- Diff: src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java ---
@@ -267,10 +267,17 @@ private boolean checkFourLetterWord(final Channel
channel,
{
// We take advantage of the limited size of the length to look
// for cmds. They are all 4-bytes which fits inside of an int
-String cmd = FourLetterCommands.getCmdMapView().get(len);
-if (cmd == null) {
+if (!FourLetterCommands.isKnown(len)) {
return false;
}
+
+// ZOOKEEPER-2693: don't execute 4lw if it's not enabled.
+String cmd = FourLetterCommands.getCommandString(len);
+if (!FourLetterCommands.isEnabled(cmd)) {
+LOG.debug("Command {} is not executed because it is not white
listed.", cmd);
+return true;
--- End diff --
@rakeshadr h I think we did not even close server sockets today for
four letter words in general, see
https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java#L343
and
https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java#L388
This is not a problem in general for the command line interface because in
that case client socket will close first and then server socket will close as a
result of client socket close... however if someone writes a client that opens
and holds a socket then server will not close the socket even after 4lw finish
execution. This probably is a by design as it allows clients to pipe 4lw
commands w/o re-opening sockets but I see this is another potential point of
vulnerability where a server could run out of sockets..
I think we should probably close the sockets in the two links I posted in
the beginning. Let me know what you think @rakeshadr.
> DOS attack on wchp/wchc four letter words (4lw)
> ---
>
> Key: ZOOKEEPER-2693
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693
> Project: ZooKeeper
> Issue Type: Bug
> Components: security, server
>Affects Versions: 3.4.0, 3.5.1, 3.5.2
>Reporter: Patrick Hunt
>Assignee: Michael Han
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2693-01.patch
>
>
> The wchp/wchc four letter words can be exploited in a DOS attack on the ZK
> client port - typically 2181. The following POC attack was recently published
> on the web:
> https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us
> The most straightforward way to block this attack is to not allow access to
> the client port to non-trusted clients - i.e. firewall the ZooKeeper service
> and only allow access to trusted applications using it for coordination.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #179: ZOOKEEPER-2693: DOS attack on wchp/wchc four le...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/179#discussion_r102275734
--- Diff: src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java ---
@@ -267,10 +267,17 @@ private boolean checkFourLetterWord(final Channel
channel,
{
// We take advantage of the limited size of the length to look
// for cmds. They are all 4-bytes which fits inside of an int
-String cmd = FourLetterCommands.getCmdMapView().get(len);
-if (cmd == null) {
+if (!FourLetterCommands.isKnown(len)) {
return false;
}
+
+// ZOOKEEPER-2693: don't execute 4lw if it's not enabled.
+String cmd = FourLetterCommands.getCommandString(len);
+if (!FourLetterCommands.isEnabled(cmd)) {
+LOG.debug("Command {} is not executed because it is not white
listed.", cmd);
+return true;
--- End diff --
@rakeshadr h I think we did not even close server sockets today for
four letter words in general, see
https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java#L343
and
https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java#L388
This is not a problem in general for the command line interface because in
that case client socket will close first and then server socket will close as a
result of client socket close... however if someone writes a client that opens
and holds a socket then server will not close the socket even after 4lw finish
execution. This probably is a by design as it allows clients to pipe 4lw
commands w/o re-opening sockets but I see this is another potential point of
vulnerability where a server could run out of sockets..
I think we should probably close the sockets in the two links I posted in
the beginning. Let me know what you think @rakeshadr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876403#comment-15876403 ] Hadoop QA commented on ZOOKEEPER-2700: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/365//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/365//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/365//console This message is automatically generated. > Force ZooKeeper to generate snapshot > > > Key: ZOOKEEPER-2700 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Flier Lu >Priority: Minor > > When cold backup or remote offline sync Zookeeper instances, we need the > latest snapshot. > Add a four letter `snap` command to force Zookeeper to generate snapshot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Failed: ZOOKEEPER- PreCommit Build #365
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/365/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 70.72 MB...] [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/365//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/365//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/365//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] 7c7f75308ba7402cb131e2bc3b700ad13835bb62 logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1635: exec returned: 2 Total time: 14 minutes 50 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Compressed 579.54 KB of artifacts by 49.7% relative to #362 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-2700 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## 5 tests failed. FAILED: org.apache.zookeeper.server.quorum.StandaloneDisabledTest.startSingleServerTest Error Message: client could not connect to reestablished quorum: giving up after 30+ seconds. Stack Trace: junit.framework.AssertionFailedError: client could not connect to reestablished quorum: giving up after 30+ seconds. at org.apache.zookeeper.test.ReconfigTest.reconfig(ReconfigTest.java:93) at org.apache.zookeeper.server.quorum.StandaloneDisabledTest.testReconfig(StandaloneDisabledTest.java:233) at org.apache.zookeeper.server.quorum.StandaloneDisabledTest.startSingleServerTest(StandaloneDisabledTest.java:109) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) FAILED: org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged Error Message: expected: but was: Stack Trace: junit.framework.AssertionFailedError: expected: but was: at org.apache.zookeeper.test.WatchEventWhenAutoResetTest$EventsWatcher.assertEvent(WatchEventWhenAutoResetTest.java:67) at org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged(WatchEventWhenAutoResetTest.java:117) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) FAILED:
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876398#comment-15876398
]
ASF GitHub Bot commented on ZOOKEEPER-2693:
---
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/179#discussion_r102273810
--- Diff: src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java ---
@@ -267,10 +267,17 @@ private boolean checkFourLetterWord(final Channel
channel,
{
// We take advantage of the limited size of the length to look
// for cmds. They are all 4-bytes which fits inside of an int
-String cmd = FourLetterCommands.getCmdMapView().get(len);
-if (cmd == null) {
+if (!FourLetterCommands.isKnown(len)) {
return false;
}
+
+// ZOOKEEPER-2693: don't execute 4lw if it's not enabled.
+String cmd = FourLetterCommands.getCommandString(len);
+if (!FourLetterCommands.isEnabled(cmd)) {
+LOG.debug("Command {} is not executed because it is not white
listed.", cmd);
+return true;
--- End diff --
Thanks @arshadmohammad for your suggestion, I like this approach from a
user experience point of view as it provides nice error message on client side.
I am a little bit concerned that instead of doing a (nearly) NOP on server
side to block a command the way the patch is doing now (which just cost a look
up), any command including garbage now costs some string printing plus has to
go through network stack to send the bytes back. This may lead potential
vulnerable point, though it might also be OK as one could argue if we are in
such case (ZK server is wide open, every bets is off.). I tend to lean towards
the safer side though so I'll wait for a while for other comments regarding
this issue.
> DOS attack on wchp/wchc four letter words (4lw)
> ---
>
> Key: ZOOKEEPER-2693
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693
> Project: ZooKeeper
> Issue Type: Bug
> Components: security, server
>Affects Versions: 3.4.0, 3.5.1, 3.5.2
>Reporter: Patrick Hunt
>Assignee: Michael Han
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2693-01.patch
>
>
> The wchp/wchc four letter words can be exploited in a DOS attack on the ZK
> client port - typically 2181. The following POC attack was recently published
> on the web:
> https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us
> The most straightforward way to block this attack is to not allow access to
> the client port to non-trusted clients - i.e. firewall the ZooKeeper service
> and only allow access to trusted applications using it for coordination.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876396#comment-15876396 ] Hadoop QA commented on ZOOKEEPER-2700: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/364//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/364//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/364//console This message is automatically generated. > Force ZooKeeper to generate snapshot > > > Key: ZOOKEEPER-2700 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Flier Lu >Priority: Minor > > When cold backup or remote offline sync Zookeeper instances, we need the > latest snapshot. > Add a four letter `snap` command to force Zookeeper to generate snapshot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] zookeeper pull request #179: ZOOKEEPER-2693: DOS attack on wchp/wchc four le...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/179#discussion_r102273810
--- Diff: src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java ---
@@ -267,10 +267,17 @@ private boolean checkFourLetterWord(final Channel
channel,
{
// We take advantage of the limited size of the length to look
// for cmds. They are all 4-bytes which fits inside of an int
-String cmd = FourLetterCommands.getCmdMapView().get(len);
-if (cmd == null) {
+if (!FourLetterCommands.isKnown(len)) {
return false;
}
+
+// ZOOKEEPER-2693: don't execute 4lw if it's not enabled.
+String cmd = FourLetterCommands.getCommandString(len);
+if (!FourLetterCommands.isEnabled(cmd)) {
+LOG.debug("Command {} is not executed because it is not white
listed.", cmd);
+return true;
--- End diff --
Thanks @arshadmohammad for your suggestion, I like this approach from a
user experience point of view as it provides nice error message on client side.
I am a little bit concerned that instead of doing a (nearly) NOP on server
side to block a command the way the patch is doing now (which just cost a look
up), any command including garbage now costs some string printing plus has to
go through network stack to send the bytes back. This may lead potential
vulnerable point, though it might also be OK as one could argue if we are in
such case (ZK server is wide open, every bets is off.). I tend to lean towards
the safer side though so I'll wait for a while for other comments regarding
this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
Failed: ZOOKEEPER- PreCommit Build #364
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/364/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 68.42 MB...] [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/364//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/364//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/364//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] Comment added. [exec] 8ee079f3431887016af11c5bbaa71cc594e7 logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' and '/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess' are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1635: exec returned: 2 Total time: 13 minutes 15 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Compressed 579.49 KB of artifacts by 49.7% relative to #362 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-2700 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## 4 tests failed. FAILED: org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalFollowerRunWithDiff Error Message: followerZooKeeperServer.takeSnapshot(); Never wanted here: -> at org.apache.zookeeper.server.quorum.Zab1_0Test$5.converseWithFollower(Zab1_0Test.java:883) But invoked here: -> at org.apache.zookeeper.server.SyncRequestProcessor$1.run(SyncRequestProcessor.java:134) Stack Trace: junit.framework.AssertionFailedError: followerZooKeeperServer.takeSnapshot(); Never wanted here: -> at org.apache.zookeeper.server.quorum.Zab1_0Test$5.converseWithFollower(Zab1_0Test.java:883) But invoked here: -> at org.apache.zookeeper.server.SyncRequestProcessor$1.run(SyncRequestProcessor.java:134) at org.apache.zookeeper.server.quorum.Zab1_0Test$5.converseWithFollower(Zab1_0Test.java:883) at org.apache.zookeeper.server.quorum.Zab1_0Test.testFollowerConversation(Zab1_0Test.java:517) at org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalFollowerRunWithDiff(Zab1_0Test.java:784) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79) FAILED: org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncByTxnlogThenDiffAfterFollowerCrashes Error Message: Should have same number of ephemerals in both followers expected:<108> but was:<120> Stack Trace:
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876386#comment-15876386 ] Hadoop QA commented on ZOOKEEPER-2700: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/363//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/363//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/363//console This message is automatically generated. > Force ZooKeeper to generate snapshot > > > Key: ZOOKEEPER-2700 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Flier Lu >Priority: Minor > > When cold backup or remote offline sync Zookeeper instances, we need the > latest snapshot. > Add a four letter `snap` command to force Zookeeper to generate snapshot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Failed: ZOOKEEPER- PreCommit Build #363
Build: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/363/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 68.27 MB...] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests. [exec] [exec] Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/363//testReport/ [exec] Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/363//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/363//console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] == [exec] == [exec] Adding comment to Jira. [exec] == [exec] == [exec] [exec] [exec] [exec] Remote error: (0)null [exec] 00870d9cab7809aa60531436ffa336e290ee4024 logged out [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] [exec] mv: ‘/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess’ and ‘/home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/patchprocess’ are the same file BUILD FAILED /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-github-pr-build/build.xml:1635: exec returned: 2 Total time: 19 minutes 39 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Compressed 579.53 KB of artifacts by 49.7% relative to #362 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 [description-setter] Description set: ZOOKEEPER-2700 Putting comment on the pull request Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.StandaloneDisabledTest.startSingleServerTest Error Message: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. Stack Trace: junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. at java.lang.Thread.run(Thread.java:745)
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876385#comment-15876385
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102271596
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@revans2 It's not easy to find `SyncRequestProcessor `, but we may use
[reversed
workaroud](https://github.com/apache/zookeeper/pull/180/files#diff-a8615898f7be77e9cfdb686e21c97b1eL123),
let's `SyncRequestProcessor` use
`zks.getLastProcessedZxid()-zks.getLastSnapshotZxid()` to trace snapshot
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102271596
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@revans2 It's not easy to find `SyncRequestProcessor `, but we may use
[reversed
workaroud](https://github.com/apache/zookeeper/pull/180/files#diff-a8615898f7be77e9cfdb686e21c97b1eL123),
let's `SyncRequestProcessor` use
`zks.getLastProcessedZxid()-zks.getLastSnapshotZxid()` to trace snapshot
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876335#comment-15876335 ] ASF GitHub Bot commented on ZOOKEEPER-2700: --- Github user flier commented on the issue: https://github.com/apache/zookeeper/pull/180 @revans2 added a [generating](https://github.com/apache/zookeeper/pull/180/files#diff-8c1d5f61ac728fda8f4b706ae4277364R524) field to the response of `TakeSnapshotCommand ` command, and JMX `takeSnapshot` will return a boolean for same reason. > Force ZooKeeper to generate snapshot > > > Key: ZOOKEEPER-2700 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Flier Lu >Priority: Minor > > When cold backup or remote offline sync Zookeeper instances, we need the > latest snapshot. > Add a four letter `snap` command to force Zookeeper to generate snapshot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] zookeeper issue #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method and Jetty...
Github user flier commented on the issue: https://github.com/apache/zookeeper/pull/180 @revans2 added a [generating](https://github.com/apache/zookeeper/pull/180/files#diff-8c1d5f61ac728fda8f4b706ae4277364R524) field to the response of `TakeSnapshotCommand ` command, and JMX `takeSnapshot` will return a boolean for same reason. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876313#comment-15876313
]
Rakesh R commented on ZOOKEEPER-2693:
-
bq. There are some 4lw commands which ZooKeeper is using by itself
Good catch, [~arshad.mohammad].
bq. even though we are not doing much work in these commands but still the
connections will be created for every call.
IIUC, these are two problems -> {{case-1)}} restrict 4lw cmd execution as few
cmds taking too much time for execution. {{case-2)}} protection against overuse
because it creates many connections.
I think, {{case-1}} is matching with this jira reported issue and we could
provide {{whitelist}} config in both branches 3.4 and 3.5+ to solve this
problem.
In branch-3.4, we could give a simple fix by exposing {{whitelist}}
configuration and include {{srvr, isro}} cmds in default list. Since we have
plans to deprecate 4lws in branch-3.5, we could find alternative ways instead
of using {{srvr, isro}} cmd internally. Probably, we could raise separate jira
task and handle this case.
[~hanm], could you create a PR for branch-3.4, if no objection from anyone
about {{whitelist}} idea and that would help to unblock 3.4.10 releasing.
{{case-2}}, I like [~phunt]'s idea of introducing configuration to limit the
number of 4lw that would be allowed to run in parallel. For example, allows
only 1-n number of 4lw cmds to run in parallel. How about raising another jira
to implement this instead of clubbing with this issue?
> DOS attack on wchp/wchc four letter words (4lw)
> ---
>
> Key: ZOOKEEPER-2693
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693
> Project: ZooKeeper
> Issue Type: Bug
> Components: security, server
>Affects Versions: 3.4.0, 3.5.1, 3.5.2
>Reporter: Patrick Hunt
>Assignee: Michael Han
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2693-01.patch
>
>
> The wchp/wchc four letter words can be exploited in a DOS attack on the ZK
> client port - typically 2181. The following POC attack was recently published
> on the web:
> https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us
> The most straightforward way to block this attack is to not allow access to
> the client port to non-trusted clients - i.e. firewall the ZooKeeper service
> and only allow access to trusted applications using it for coordination.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
ZooKeeper_branch35_solaris - Build # 447 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_solaris/447/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 59.35 MB...] [junit] 2017-02-21 17:17:05,410 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:11222 [junit] 2017-02-21 17:17:05,410 [myid:] - INFO [main:ZooKeeperServer@907] - minSessionTimeout set to 6000 [junit] 2017-02-21 17:17:05,411 [myid:] - INFO [main:ZooKeeperServer@916] - maxSessionTimeout set to 6 [junit] 2017-02-21 17:17:05,411 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test756011653083360957.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test756011653083360957.junit.dir/version-2 [junit] 2017-02-21 17:17:05,411 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test756011653083360957.junit.dir/version-2/snapshot.b [junit] 2017-02-21 17:17:05,413 [myid:] - INFO [main:FileTxnSnapLog@320] - Snapshotting: 0xb to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test756011653083360957.junit.dir/version-2/snapshot.b [junit] 2017-02-21 17:17:05,415 [myid:] - ERROR [main:ZooKeeperServer@505] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-21 17:17:05,415 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 [junit] 2017-02-21 17:17:05,415 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44672 [junit] 2017-02-21 17:17:05,416 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:44672 [junit] 2017-02-21 17:17:05,416 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output [junit] 2017-02-21 17:17:05,416 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@614] - Closed socket connection for client /127.0.0.1:44672 (no session established for client) [junit] 2017-02-21 17:17:05,417 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-21 17:17:05,418 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree [junit] 2017-02-21 17:17:05,418 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree [junit] 2017-02-21 17:17:05,418 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port [junit] 2017-02-21 17:17:05,418 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222 [junit] 2017-02-21 17:17:05,419 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 17887 [junit] 2017-02-21 17:17:05,419 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 24 [junit] 2017-02-21 17:17:05,419 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota [junit] 2017-02-21 17:17:05,419 [myid:] - INFO [main:ClientBase@543] - tearDown starting [junit] 2017-02-21 17:17:05,502 [myid:] - INFO [main:ZooKeeper@1324] - Session: 0x126dea609d5 closed [junit] 2017-02-21 17:17:05,502 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x126dea609d5 [junit] 2017-02-21 17:17:05,502 [myid:] - INFO [main:ClientBase@513] - STOPPING server [junit] 2017-02-21 17:17:05,502 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted [junit] 2017-02-21 17:17:05,502 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method [junit] 2017-02-21 17:17:05,502 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-21 17:17:05,502 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-21 17:17:05,503 [myid:] - INFO [main:ZooKeeperServer@541] - shutting down [junit] 2017-02-21 17:17:05,503 [myid:] - ERROR [main:ZooKeeperServer@505] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876305#comment-15876305
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102263924
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
I have some scenes need to take snapshot, for example
First, our major Zookeeper cluster was deployed in an AWS zone, some
observers running at a dozen IDC. We use this topological structure because
Zookeeper cluster is not friendly to multi-IDC deployment. Besize, our
zookeeper snapshot and transaction logs are huge, because some wrong client
usage that hard to fix in short time :(
Sometimes, we plan to maintains the major cluster, we have to start another
mirror cluster in same DC, and switch from the major cluster to the mirror
cluster. If we do it faster enough, the observer and client will not concern
the changes. That's why we need take snapshot to speed up the migration. If
something got wrong, we could switch back to the old cluster, lost some
transaction better than the whole system down.
Second, our backup policy need a daily/hourly offline backup, to AWS S3 or
other DC. I would like to take and upload a latest and clean snapshot, instead
of tar an old snapshot with a number of transaction logs.
Third, sometimes we need to deploy a new observer or a testing cluster in
different DC, we have to copy the latest snapshot offline, because Zookeeper
observer sync progress may become very slow, the TCP window could drop to
10-20KB/s in the 40-60% packet loss rate.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102263924
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
I have some scenes need to take snapshot, for example
First, our major Zookeeper cluster was deployed in an AWS zone, some
observers running at a dozen IDC. We use this topological structure because
Zookeeper cluster is not friendly to multi-IDC deployment. Besize, our
zookeeper snapshot and transaction logs are huge, because some wrong client
usage that hard to fix in short time :(
Sometimes, we plan to maintains the major cluster, we have to start another
mirror cluster in same DC, and switch from the major cluster to the mirror
cluster. If we do it faster enough, the observer and client will not concern
the changes. That's why we need take snapshot to speed up the migration. If
something got wrong, we could switch back to the old cluster, lost some
transaction better than the whole system down.
Second, our backup policy need a daily/hourly offline backup, to AWS S3 or
other DC. I would like to take and upload a latest and clean snapshot, instead
of tar an old snapshot with a number of transaction logs.
Third, sometimes we need to deploy a new observer or a testing cluster in
different DC, we have to copy the latest snapshot offline, because Zookeeper
observer sync progress may become very slow, the TCP window could drop to
10-20KB/s in the 40-60% packet loss rate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Reopened] (ZOOKEEPER-2692) Fix race condition in testWatchAutoResetWithPending
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Han reopened ZOOKEEPER-2692:
Looks like this introduces a regression on one watcher test.
{noformat}
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged
Error Message:
expected: but was:
Stack Trace:
junit.framework.AssertionFailedError: expected: but
was:
at
org.apache.zookeeper.test.WatchEventWhenAutoResetTest$EventsWatcher.assertEvent(WatchEventWhenAutoResetTest.java:67)
at
org.apache.zookeeper.test.WatchEventWhenAutoResetTest.testNodeDataChanged(WatchEventWhenAutoResetTest.java:117)
at
org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
{noformat}
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/951/
https://builds.apache.org/job/ZooKeeper-trunk-jdk8/946/
[~abrahamfine] Could you please look at why this test fail and patch it?
> Fix race condition in testWatchAutoResetWithPending
> ---
>
> Key: ZOOKEEPER-2692
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2692
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: tests
>Affects Versions: 3.4.9, 3.5.3, 3.6.0
>Reporter: Abraham Fine
>Assignee: Abraham Fine
> Fix For: 3.4.10, 3.5.3, 3.6.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876199#comment-15876199
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102245094
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@eribeiro An AtomicBoolean is not enough as the SyncRequestProcessor
ignores it, only the JMX and admin commands use it.
@flier there are several things going on here and I don't know the
reasoning for all of them but I can guess.
1. Only take one snapshot at a time. I don't think this is super
critical, because it is not a correctness problem. Having multiple snapshots
happening at the same time should work correctly even in the face of crashes,
but it becomes a performance problem. There is a limited amount of CPU,
Memory, and most importantly disk bandwidth and IOps. The last successful
snapshot wins. So having multiple snapshots all going at the same time means
we are likely to be doing a lot of wasted work if everything does happen
successfully. If we can schedule the snapshots so there is only one going on
at a time then the full bandwidth and IOps can go to that snapshot. Even
better would be to space them out, so if we force a snapshot it makes
SyncRequestProcessor reset its counter so we don't take another one until X
more transactions have been processed.
2. Taking a snapshot at the wrong time and then crashing can corrupt the DB
on that node. This is potentially critical. The probability of this happening
is very very small, but if we can fix it so it can never happen, I would be
much happier.
I think we can fix both issues at the same time, by making JMX and Admin
use SyncRequestProcessor instead of going to the ZookeeperServer directly. If
no SyncRequestProcessor is ready then we can return such to the user so they
know the node is not ready to take a snapshot.
I also really would like to understand the use case for this command,
because I just don't see much value to a user to force another snapshot if one
is already in progress. I also would like to understand when you would want to
take a snapshot and if these changes would too severely limit that.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102245094
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
@eribeiro An AtomicBoolean is not enough as the SyncRequestProcessor
ignores it, only the JMX and admin commands use it.
@flier there are several things going on here and I don't know the
reasoning for all of them but I can guess.
1. Only take one snapshot at a time. I don't think this is super
critical, because it is not a correctness problem. Having multiple snapshots
happening at the same time should work correctly even in the face of crashes,
but it becomes a performance problem. There is a limited amount of CPU,
Memory, and most importantly disk bandwidth and IOps. The last successful
snapshot wins. So having multiple snapshots all going at the same time means
we are likely to be doing a lot of wasted work if everything does happen
successfully. If we can schedule the snapshots so there is only one going on
at a time then the full bandwidth and IOps can go to that snapshot. Even
better would be to space them out, so if we force a snapshot it makes
SyncRequestProcessor reset its counter so we don't take another one until X
more transactions have been processed.
2. Taking a snapshot at the wrong time and then crashing can corrupt the DB
on that node. This is potentially critical. The probability of this happening
is very very small, but if we can fix it so it can never happen, I would be
much happier.
I think we can fix both issues at the same time, by making JMX and Admin
use SyncRequestProcessor instead of going to the ZookeeperServer directly. If
no SyncRequestProcessor is ready then we can return such to the user so they
know the node is not ready to take a snapshot.
I also really would like to understand the use case for this command,
because I just don't see much value to a user to force another snapshot if one
is already in progress. I also would like to understand when you would want to
take a snapshot and if these changes would too severely limit that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876175#comment-15876175
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102238878
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
If we are not expect to take snapshot on same time, the most easy way is to
use `AtomicBoolean ` protect the `takeSnapshot` from all the code path, which
may block the `SyncRequestProcessor` a while if a manual task is ongoing. My
current code is assuming the background `SyncRequestProcessor` have higher
priority.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user flier commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102238878
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
If we are not expect to take snapshot on same time, the most easy way is to
use `AtomicBoolean ` protect the `takeSnapshot` from all the code path, which
may block the `SyncRequestProcessor` a while if a manual task is ongoing. My
current code is assuming the background `SyncRequestProcessor` have higher
priority.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876137#comment-15876137 ] ASF GitHub Bot commented on ZOOKEEPER-2700: --- Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/180 Oh I also had the idea that it might be nice to provide feedback on the snap command for both JMX and the Admin command. You could provide more then just the last zxid, but you could also indicate if we are going to do the snapshot or not, and why. > Force ZooKeeper to generate snapshot > > > Key: ZOOKEEPER-2700 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Flier Lu >Priority: Minor > > When cold backup or remote offline sync Zookeeper instances, we need the > latest snapshot. > Add a four letter `snap` command to force Zookeeper to generate snapshot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user eribeiro commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102232941
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
+1. This is the reason why I advocated the use of `AtomicBoolean` instead
of `AtomicInteger` to provide mutual exclusion on this code snippet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876147#comment-15876147
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user eribeiro commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102232941
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
+1. This is the reason why I advocated the use of `AtomicBoolean` instead
of `AtomicInteger` to provide mutual exclusion on this code snippet.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper issue #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method and Jetty...
Github user revans2 commented on the issue: https://github.com/apache/zookeeper/pull/180 Oh I also had the idea that it might be nice to provide feedback on the snap command for both JMX and the Admin command. You could provide more then just the last zxid, but you could also indicate if we are going to do the snapshot or not, and why. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876134#comment-15876134
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r10749
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
try {
+lastSnapshotZxid = zkDb.getDataTreeLastProcessedZxid();
+isGeneratingSnapshot.incrementAndGet();
+
txnLogFactory.save(zkDb.getDataTree(),
zkDb.getSessionWithTimeOuts());
} catch (IOException e) {
LOG.error("Severe unrecoverable error, exiting", e);
// This is a severe error that we cannot recover from,
// so we need to exit
System.exit(10);
+} finally {
+isGeneratingSnapshot.decrementAndGet();
}
}
+public boolean maybeTakeSnapshot() {
--- End diff --
Could you add some javadocs here? It would be nice to explain the
difference between takeSnapshot and maybeTakeSnapshot.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r10749
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
try {
+lastSnapshotZxid = zkDb.getDataTreeLastProcessedZxid();
+isGeneratingSnapshot.incrementAndGet();
+
txnLogFactory.save(zkDb.getDataTree(),
zkDb.getSessionWithTimeOuts());
} catch (IOException e) {
LOG.error("Severe unrecoverable error, exiting", e);
// This is a severe error that we cannot recover from,
// so we need to exit
System.exit(10);
+} finally {
+isGeneratingSnapshot.decrementAndGet();
}
}
+public boolean maybeTakeSnapshot() {
--- End diff --
Could you add some javadocs here? It would be nice to explain the
difference between takeSnapshot and maybeTakeSnapshot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (ZOOKEEPER-2700) Force ZooKeeper to generate snapshot
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876133#comment-15876133
]
ASF GitHub Bot commented on ZOOKEEPER-2700:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102229805
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
We now have a lot of potential paths to take a snapshot. JMX,
SyncRequestProcessor, Admin, and as part of the sync stage in ZABP. In the
past it was just the SyncRequestProcessor and ZABP that would trigger
snapshots. We didn't have much concern about collisions, because
SyncRequestProcessor would only run as edits were being sent, and the edits
would only be sent after the ZABP sync phase had completed. Now we have Admin
and JMX possibly doing a snapshot in the background.
Now if a snapshot request comes in during the ZABP sync phase after we have
clear out the in memory DB and not yet applied the new snapshot and then we
crash before we can write out the new snapshot we could end up with data
corruption. This should be super super rare so I don't really know if we care
all that much about it, but I think it is something that we can fix.
I am not an expert on the code, so I am not sure of the cleanest way to fix
this, but it feels like having maybeTakeSnapshot be a part of the
SyncRequestProcessor instead of ZooKeeperServer. I don't know how simple it is
to get to the SyncRequestProcessor from JMX/Admin but if you can then it means
that we can reset the edit count so we are not taking a lot of snapshots all at
once, one right after another. It also means that we can truly avoid having
more then one snapshot be taken at any time.
> Force ZooKeeper to generate snapshot
>
>
> Key: ZOOKEEPER-2700
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2700
> Project: ZooKeeper
> Issue Type: Improvement
>Reporter: Flier Lu
>Priority: Minor
>
> When cold backup or remote offline sync Zookeeper instances, we need the
> latest snapshot.
> Add a four letter `snap` command to force Zookeeper to generate snapshot.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] zookeeper pull request #180: ZOOKEEPER-2700 add JMX `takeSnapshot` method an...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/180#discussion_r102229805
--- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
@@ -303,15 +305,38 @@ public void loadData() throws IOException,
InterruptedException {
public void takeSnapshot(){
--- End diff --
We now have a lot of potential paths to take a snapshot. JMX,
SyncRequestProcessor, Admin, and as part of the sync stage in ZABP. In the
past it was just the SyncRequestProcessor and ZABP that would trigger
snapshots. We didn't have much concern about collisions, because
SyncRequestProcessor would only run as edits were being sent, and the edits
would only be sent after the ZABP sync phase had completed. Now we have Admin
and JMX possibly doing a snapshot in the background.
Now if a snapshot request comes in during the ZABP sync phase after we have
clear out the in memory DB and not yet applied the new snapshot and then we
crash before we can write out the new snapshot we could end up with data
corruption. This should be super super rare so I don't really know if we care
all that much about it, but I think it is something that we can fix.
I am not an expert on the code, so I am not sure of the cleanest way to fix
this, but it feels like having maybeTakeSnapshot be a part of the
SyncRequestProcessor instead of ZooKeeperServer. I don't know how simple it is
to get to the SyncRequestProcessor from JMX/Admin but if you can then it means
that we can reset the edit count so we are not taking a lot of snapshots all at
once, one right after another. It also means that we can truly avoid having
more then one snapshot be taken at any time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
ZooKeeper_branch34_openjdk7 - Build # 1400 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/1400/
###
## LAST 60 LINES OF THE CONSOLE
###
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-5 (Ubuntu yahoo-not-h2 ubuntu ubuntu5 docker) in
workspace /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url git://git.apache.org/zookeeper.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching upstream changes from git://git.apache.org/zookeeper.git
> git --version # timeout=10
> git fetch --tags --progress git://git.apache.org/zookeeper.git
> +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/branch-3.4^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/branch-3.4^{commit} # timeout=10
Checking out Revision 373b82bab843cc84c22b99f6511f8fea974fd2b4
(refs/remotes/origin/branch-3.4)
> git config core.sparsecheckout # timeout=10
> git checkout -f 373b82bab843cc84c22b99f6511f8fea974fd2b4
> git rev-list 373b82bab843cc84c22b99f6511f8fea974fd2b4 # timeout=10
No emails were triggered.
[ZooKeeper_branch34_openjdk7] $ /home/jenkins/tools/ant/latest/bin/ant
-Dtest.output=yes -Dtest.junit.threads=8 -Dtest.junit.output.format=xml
-Djavac.target=1.7 clean test-core-java
Exception in thread "main" java.lang.UnsupportedClassVersionError:
org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)
at java.net.URLClassLoader.access$100(URLClassLoader.java:64)
at java.net.URLClassLoader$1.run(URLClassLoader.java:354)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Build step 'Invoke Ant' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
ZooKeeper_branch34_solaris - Build # 1481 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_solaris/1481/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 25.43 MB...] [junit] 2017-02-21 13:59:22,191 [myid:] - INFO [main:PrepRequestProcessor@764] - Shutting down [junit] 2017-02-21 13:59:22,191 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2017-02-21 13:59:22,191 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2017-02-21 13:59:22,191 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2017-02-21 13:59:22,191 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2017-02-21 13:59:22,192 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-21 13:59:22,192 [myid:] - INFO [main:JMXEnv@147] - ensureOnly:[] [junit] 2017-02-21 13:59:22,193 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2017-02-21 13:59:22,193 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2017-02-21 13:59:22,194 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2017-02-21 13:59:22,194 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2017-02-21 13:59:22,194 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/build/test/tmp/test8150496328335636615.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/build/test/tmp/test8150496328335636615.junit.dir/version-2 [junit] 2017-02-21 13:59:22,197 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-21 13:59:22,197 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-21 13:59:22,198 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:58128 [junit] 2017-02-21 13:59:22,198 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@838] - Processing stat command from /127.0.0.1:58128 [junit] 2017-02-21 13:59:22,198 [myid:] - INFO [Thread-5:NIOServerCnxn$StatCommand@674] - Stat command output [junit] 2017-02-21 13:59:22,199 [myid:] - INFO [Thread-5:NIOServerCnxn@1019] - Closed socket connection for client /127.0.0.1:58128 (no session established for client) [junit] 2017-02-21 13:59:22,199 [myid:] - INFO [main:JMXEnv@230] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-21 13:59:22,200 [myid:] - INFO [main:JMXEnv@247] - expect:InMemoryDataTree [junit] 2017-02-21 13:59:22,200 [myid:] - INFO [main:JMXEnv@251] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2017-02-21 13:59:22,201 [myid:] - INFO [main:JMXEnv@247] - expect:StandaloneServer_port [junit] 2017-02-21 13:59:22,201 [myid:] - INFO [main:JMXEnv@251] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2017-02-21 13:59:22,201 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 9194 [junit] 2017-02-21 13:59:22,201 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2017-02-21 13:59:22,201 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2017-02-21 13:59:22,201 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2017-02-21 13:59:23,052 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x15a60f81e9d closed [junit] 2017-02-21 13:59:23,052 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x15a60f81e9d [junit] 2017-02-21 13:59:23,052 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2017-02-21 13:59:23,054 [myid:] - INFO [main:ZooKeeperServer@505] - shutting down [junit] 2017-02-21 13:59:23,054 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-21 13:59:23,054 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2017-02-21 13:59:23,054 [myid:] - INFO [main:PrepRequestProcessor@764] - Shutting down [junit] 2017-02-21 13:59:23,054 [myid:] -
ZooKeeper_branch35_jdk8 - Build # 428 - Failure
See https://builds.apache.org/job/ZooKeeper_branch35_jdk8/428/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 61.42 MB...]
[junit] 2017-02-21 12:08:56,960 [myid:] - INFO [main:ZooKeeperServer@541]
- shutting down
[junit] 2017-02-21 12:08:56,960 [myid:] - ERROR [main:ZooKeeperServer@505]
- ZKShutdownHandler is not registered, so ZooKeeper server won't take any
action on ERROR or SHUTDOWN server state changes
[junit] 2017-02-21 12:08:56,960 [myid:] - INFO
[main:SessionTrackerImpl@232] - Shutting down
[junit] 2017-02-21 12:08:56,960 [myid:] - INFO
[main:PrepRequestProcessor@973] - Shutting down
[junit] 2017-02-21 12:08:56,961 [myid:] - INFO
[main:SyncRequestProcessor@191] - Shutting down
[junit] 2017-02-21 12:08:56,961 [myid:] - INFO
[SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
[junit] 2017-02-21 12:08:56,961 [myid:] - INFO [ProcessThread(sid:0
cport:30319)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
[junit] 2017-02-21 12:08:56,961 [myid:] - INFO
[main:FinalRequestProcessor@479] - shutdown of request processor complete
[junit] 2017-02-21 12:08:56,961 [myid:] - INFO [main:MBeanRegistry@128] -
Unregister MBean
[org.apache.ZooKeeperService:name0=StandaloneServer_port30319,name1=InMemoryDataTree]
[junit] 2017-02-21 12:08:56,961 [myid:] - INFO [main:MBeanRegistry@128] -
Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port30319]
[junit] 2017-02-21 12:08:56,961 [myid:] - INFO
[main:FourLetterWordMain@85] - connecting to 127.0.0.1 30319
[junit] 2017-02-21 12:08:56,962 [myid:] - INFO [main:JMXEnv@146] -
ensureOnly:[]
[junit] 2017-02-21 12:08:56,965 [myid:] - INFO [main:ClientBase@568] -
fdcount after test is: 1470 at start it was 1470
[junit] 2017-02-21 12:08:56,966 [myid:] - INFO [main:ZKTestCase$1@65] -
SUCCEEDED testWatcherAutoResetWithLocal
[junit] 2017-02-21 12:08:56,966 [myid:] - INFO [main:ZKTestCase$1@60] -
FINISHED testWatcherAutoResetWithLocal
[junit] Tests run: 103, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
439.212 sec, Thread: 8, Class: org.apache.zookeeper.test.NettyNettySuiteTest
[junit] 2017-02-21 12:08:57,359 [myid:127.0.0.1:30199] - INFO
[main-SendThread(127.0.0.1:30199):ClientCnxn$SendThread@1113] - Opening socket
connection to server 127.0.0.1/127.0.0.1:30199. Will not attempt to
authenticate using SASL (unknown error)
[junit] 2017-02-21 12:08:57,360 [myid:] - INFO [New I/O boss
#2772:ClientCnxnSocketNetty$1@127] - future isn't success, cause: {}
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30199
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[junit] at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit] 2017-02-21 12:08:57,360 [myid:] - WARN [New I/O boss
#2772:ClientCnxnSocketNetty$ZKClientHandler@439] - Exception caught: [id:
0x4c89a971] EXCEPTION: java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30199
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30199
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
ZooKeeper-trunk-jdk8 - Build # 951 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/951/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 62.96 MB...] [junit] 2017-02-21 11:57:56,061 [myid:] - INFO [New I/O worker #6701:ZooKeeperServer@1019] - Client attempting to renew session 0x1027813ce37 at /127.0.0.1:52304 [junit] 2017-02-21 11:57:56,061 [myid:] - INFO [New I/O worker #6701:ZooKeeperServer@728] - Established session 0x1027813ce37 with negotiated timeout 6000 for client /127.0.0.1:52304 [junit] 2017-02-21 11:57:56,061 [myid:127.0.0.1:11468] - INFO [main-SendThread(127.0.0.1:11468):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:11468, sessionid = 0x1027813ce37, negotiated timeout = 6000 [junit] 2017-02-21 11:57:56,068 [myid:] - INFO [SyncThread:0:FileTxnLog@204] - Creating new log file: log.7 [junit] 2017-02-21 11:57:56,700 [myid:127.0.0.1:11222] - INFO [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:11222. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-02-21 11:57:56,700 [myid:127.0.0.1:11222] - WARN [main-SendThread(127.0.0.1:11222):ClientCnxn$SendThread@1235] - Session 0x102780bbce5 for server 127.0.0.1/127.0.0.1:11222, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-02-21 11:57:57,122 [myid:] - INFO [ProcessThread(sid:0 cport:11468)::PrepRequestProcessor@617] - Processed session termination for sessionid: 0x1027813ce37 [junit] 2017-02-21 11:57:57,130 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11468,name1=Connections,name2=127.0.0.1,name3=0x1027813ce37] [junit] 2017-02-21 11:57:57,130 [myid:] - INFO [main:ZooKeeper@1324] - Session: 0x1027813ce37 closed [junit] 2017-02-21 11:57:57,130 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x1027813ce37 [junit] 2017-02-21 11:57:57,131 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 173050 [junit] 2017-02-21 11:57:57,131 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 1643 [junit] 2017-02-21 11:57:57,131 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testWatcherAutoResetWithLocal [junit] 2017-02-21 11:57:57,131 [myid:] - INFO [main:ClientBase@558] - tearDown starting [junit] 2017-02-21 11:57:57,131 [myid:] - INFO [main:ClientBase@528] - STOPPING server [junit] 2017-02-21 11:57:57,131 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:11468 [junit] 2017-02-21 11:57:57,136 [myid:] - INFO [main:ZooKeeperServer@542] - shutting down [junit] 2017-02-21 11:57:57,136 [myid:] - ERROR [main:ZooKeeperServer@506] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-21 11:57:57,136 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down [junit] 2017-02-21 11:57:57,136 [myid:] - INFO [main:PrepRequestProcessor@1008] - Shutting down [junit] 2017-02-21 11:57:57,137 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down [junit] 2017-02-21 11:57:57,137 [myid:] - INFO [ProcessThread(sid:0 cport:11468)::PrepRequestProcessor@157] - PrepRequestProcessor exited loop! [junit] 2017-02-21 11:57:57,137 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited! [junit] 2017-02-21 11:57:57,137 [myid:] - INFO [main:FinalRequestProcessor@481] - shutdown of request processor complete [junit] 2017-02-21 11:57:57,137 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11468,name1=InMemoryDataTree] [junit] 2017-02-21 11:57:57,137 [myid:] - INFO [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11468] [junit] 2017-02-21 11:57:57,138 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11468 [junit] 2017-02-21 11:57:57,138 [myid:] - INFO [main:JMXEnv@146] - ensureOnly:[] [junit] 2017-02-21 11:57:57,143 [myid:] - INFO [main:ClientBase@583] - fdcount after test is: 4842 at start it was
ZooKeeper_branch35_openjdk7 - Build # 426 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_openjdk7/426/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 233.88 KB...] [junit] 2017-02-21 09:57:10,495 [myid:] - INFO [main:ZKTestCase$1@65] - SUCCEEDED testCreateSequentialCommandWithoutPath [junit] 2017-02-21 09:57:10,495 [myid:] - INFO [main:ZKTestCase$1@60] - FINISHED testCreateSequentialCommandWithoutPath [junit] 2017-02-21 09:57:10,496 [myid:] - INFO [main:PortAssignment@85] - Assigned port 21998 from range 21993 - 24685. [junit] 2017-02-21 09:57:10,496 [myid:] - INFO [main:ZKTestCase$1@55] - STARTING testCreateNodeWithoutData [junit] 2017-02-21 09:57:10,496 [myid:] - INFO [main:ClientBase@448] - Initial fdcount is: 50 [junit] 2017-02-21 09:57:10,506 [myid:] - INFO [main:ClientBase@466] - STARTING server [junit] 2017-02-21 09:57:10,506 [myid:] - INFO [main:ClientBase@386] - CREATING server instance 127.0.0.1:21998 [junit] 2017-02-21 09:57:10,506 [myid:] - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. [junit] 2017-02-21 09:57:10,506 [myid:] - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:21998 [junit] 2017-02-21 09:57:10,507 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:21998 [junit] 2017-02-21 09:57:10,507 [myid:] - INFO [main:ZooKeeperServer@907] - minSessionTimeout set to 6000 [junit] 2017-02-21 09:57:10,507 [myid:] - INFO [main:ZooKeeperServer@916] - maxSessionTimeout set to 6 [junit] 2017-02-21 09:57:10,507 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_openjdk7/build/test/tmp/test9086588243072711581.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_openjdk7/build/test/tmp/test9086588243072711581.junit.dir/version-2 [junit] 2017-02-21 09:57:10,508 [myid:] - INFO [main:FileTxnSnapLog@320] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_openjdk7/build/test/tmp/test9086588243072711581.junit.dir/version-2/snapshot.0 [junit] 2017-02-21 09:57:10,508 [myid:] - INFO [main:FileTxnSnapLog@320] - Snapshotting: 0x0 to /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_openjdk7/build/test/tmp/test9086588243072711581.junit.dir/version-2/snapshot.0 [junit] 2017-02-21 09:57:10,510 [myid:] - ERROR [main:ZooKeeperServer@505] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-21 09:57:10,510 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 21998 [junit] 2017-02-21 09:57:10,510 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:21998:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:53051 [junit] 2017-02-21 09:57:10,505 [myid:] - INFO [main:PortAssignment@151] - Test process 6/8 using ports from 24686 - 27378. [junit] create [-s] [-e] [-c] path [data] [acl] [junit] 2017-02-21 09:57:10,542 [myid:] - INFO [main:PortAssignment@151] - Test process 2/8 using ports from 13914 - 16606. [junit] 2017-02-21 09:57:10,563 [myid:] - INFO [main:PortAssignment@85] - Assigned port 13915 from range 13914 - 16606. [junit] 2017-02-21 09:57:10,537 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:53051 [junit] 2017-02-21 09:57:10,537 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output [junit] 2017-02-21 09:57:10,538 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@614] - Closed socket connection for client /127.0.0.1:53051 (no session established for client) [junit] 2017-02-21 09:57:10,538 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-21 09:57:10,540 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree [junit] 2017-02-21 09:57:10,541 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port21998,name1=InMemoryDataTree [junit] 2017-02-21 09:57:10,541 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port [junit] 2017-02-21 09:57:10,541 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port21998 [junit] 2017-02-21 09:57:10,541 [myid:] - INFO [main:ClientBase@462] - Client test setup finished [junit] 2017-02-21 09:57:10,541 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@77] - RUNNING TEST METHOD testCreateNodeWithoutData
ZooKeeper-trunk-solaris - Build # 1515 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-solaris/1515/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 59.99 MB...] [junit] 2017-02-21 08:39:59,116 [myid:] - INFO [main:ClientBase@376] - STARTING server instance 127.0.0.1:11222 [junit] 2017-02-21 08:39:59,116 [myid:] - INFO [main:ZooKeeperServer@908] - minSessionTimeout set to 6000 [junit] 2017-02-21 08:39:59,117 [myid:] - INFO [main:ZooKeeperServer@917] - maxSessionTimeout set to 6 [junit] 2017-02-21 08:39:59,117 [myid:] - INFO [main:ZooKeeperServer@160] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test9097261130983239221.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test9097261130983239221.junit.dir/version-2 [junit] 2017-02-21 08:39:59,117 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test9097261130983239221.junit.dir/version-2/snapshot.b [junit] 2017-02-21 08:39:59,120 [myid:] - INFO [main:FileTxnSnapLog@346] - Snapshotting: 0xb to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test9097261130983239221.junit.dir/version-2/snapshot.b [junit] 2017-02-21 08:39:59,121 [myid:] - ERROR [main:ZooKeeperServer@506] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-21 08:39:59,121 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 [junit] 2017-02-21 08:39:59,122 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:49505 [junit] 2017-02-21 08:39:59,122 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:49505 [junit] 2017-02-21 08:39:59,122 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output [junit] 2017-02-21 08:39:59,123 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@614] - Closed socket connection for client /127.0.0.1:49505 (no session established for client) [junit] 2017-02-21 08:39:59,123 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-21 08:39:59,124 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree [junit] 2017-02-21 08:39:59,124 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree [junit] 2017-02-21 08:39:59,125 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port [junit] 2017-02-21 08:39:59,125 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222 [junit] 2017-02-21 08:39:59,125 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 17910 [junit] 2017-02-21 08:39:59,125 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 24 [junit] 2017-02-21 08:39:59,125 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota [junit] 2017-02-21 08:39:59,125 [myid:] - INFO [main:ClientBase@558] - tearDown starting [junit] 2017-02-21 08:39:59,202 [myid:] - INFO [main:ZooKeeper@1324] - Session: 0x126dccc9c7d closed [junit] 2017-02-21 08:39:59,202 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x126dccc9c7d [junit] 2017-02-21 08:39:59,202 [myid:] - INFO [main:ClientBase@528] - STOPPING server [junit] 2017-02-21 08:39:59,202 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method [junit] 2017-02-21 08:39:59,203 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted [junit] 2017-02-21 08:39:59,203 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-21 08:39:59,203 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-21 08:39:59,203 [myid:] - INFO [main:ZooKeeperServer@542] - shutting down [junit] 2017-02-21 08:39:59,203 [myid:] - ERROR [main:ZooKeeperServer@506] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR
ZooKeeper_branch35_jdk7 - Build # 859 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_jdk7/859/
###
## LAST 60 LINES OF THE CONSOLE
###
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on proserpina-test (ubuntu) in workspace
/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk7
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url git://git.apache.org/zookeeper.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching upstream changes from git://git.apache.org/zookeeper.git
> git --version # timeout=10
> git fetch --tags --progress git://git.apache.org/zookeeper.git
> +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/branch-3.5^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/branch-3.5^{commit} # timeout=10
Checking out Revision 1912fa8d63e6bbfdb5bb96bec85efa7c94e131e9
(refs/remotes/origin/branch-3.5)
> git config core.sparsecheckout # timeout=10
> git checkout -f 1912fa8d63e6bbfdb5bb96bec85efa7c94e131e9
> git rev-list 1912fa8d63e6bbfdb5bb96bec85efa7c94e131e9 # timeout=10
No emails were triggered.
[ZooKeeper_branch35_jdk7] $ /home/jenkins/tools/ant/latest/bin/ant
-Dtest.output=yes -Dtest.junit.threads=8 -Dtest.junit.output.format=xml
-Djavac.target=1.7 clean test-core-java
Exception in thread "main" java.lang.UnsupportedClassVersionError:
org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Build step 'Invoke Ant' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
