[jira] [Commented] (ZOOKEEPER-2693) DOS attack on wchp/wchc four letter words (4lw)
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881932#comment-15881932
]
Rakesh R commented on ZOOKEEPER-2693:
-
I'm trying an attempt to unblock 3.4.10 and 3.5.3 releases. Following are few
proposals to make the release happen by including the reported issue.
*3.4.10 requirement:*
Expose whitelist configuration with the default values,
{{4lw.commands.whitelist=ruok,srvr,crst,srst,isro,mntr}}
Also, properly documenting the {{Publicly accessible deployment}} part from the
current PR_179 changes.
*3.5.3 requirement:*
IMHO, there are two possible proposals:
# +Proposal-1)+
Expose whitelist configuration with the default values,
{{4lw.commands.whitelist=srvr,isro}}
Also, recommend users to use admin server rather than 4lw cmds
considering that 4lw will deprecated in future.
Then, later in 3.5.4, we could make this whitelist and while
deprecating will replace srvr, isro calls with better solution. That way, we
would get enough time to think about better solutions.
# +Proposal-2)+
Expose whitelist configuration with the empty default value,
{{4lw.commands.whitelist=}}
* (a) How about zkServer.sh usage of "srvr" can be achieved like,
Introduce a new admin API, admin#getServerStatus("host:port"), which
will return running stat of that server(probably, the same string format where
srvr command is returning)
{code}
"version=
read_only=false
server_stats=
node_count="
{code}
* (b) Client#pingRwServer=> Just a plain thought, probably, readonly
client code can internally tries to establish new client session with all other
servers round robin fashion. Then, check whether that the connected server is
in {{rw}} mode and act upon.
> DOS attack on wchp/wchc four letter words (4lw)
> ---
>
> Key: ZOOKEEPER-2693
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2693
> Project: ZooKeeper
> Issue Type: Bug
> Components: security, server
>Affects Versions: 3.4.0, 3.5.1, 3.5.2
>Reporter: Patrick Hunt
>Assignee: Michael Han
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2693-01.patch
>
>
> The wchp/wchc four letter words can be exploited in a DOS attack on the ZK
> client port - typically 2181. The following POC attack was recently published
> on the web:
> https://webcache.googleusercontent.com/search?q=cache:_CNGIz10PRYJ:https://www.exploit-db.com/exploits/41277/+=14=en=clnk=us
> The most straightforward way to block this attack is to not allow access to
> the client port to non-trusted clients - i.e. firewall the ZooKeeper service
> and only allow access to trusted applications using it for coordination.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
ZooKeeper_branch34_jdk7 - Build # 1417 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1417/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 31.45 MB...] [junit] 2017-02-24 03:18:48,202 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2017-02-24 03:18:48,202 [myid:] - INFO [main:PrepRequestProcessor@764] - Shutting down [junit] 2017-02-24 03:18:48,202 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2017-02-24 03:18:48,203 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2017-02-24 03:18:48,203 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2017-02-24 03:18:48,203 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2017-02-24 03:18:48,204 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-24 03:18:48,205 [myid:] - INFO [main:JMXEnv@147] - ensureOnly:[] [junit] 2017-02-24 03:18:48,206 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2017-02-24 03:18:48,207 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2017-02-24 03:18:48,207 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2017-02-24 03:18:48,207 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2017-02-24 03:18:48,208 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/build/test/tmp/test8154036818694650824.junit.dir/version-2 snapdir /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/build/test/tmp/test8154036818694650824.junit.dir/version-2 [junit] 2017-02-24 03:18:48,213 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-24 03:18:48,213 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-24 03:18:48,214 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:42378 [junit] 2017-02-24 03:18:48,214 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@838] - Processing stat command from /127.0.0.1:42378 [junit] 2017-02-24 03:18:48,215 [myid:] - INFO [Thread-4:NIOServerCnxn$StatCommand@674] - Stat command output [junit] 2017-02-24 03:18:48,215 [myid:] - INFO [Thread-4:NIOServerCnxn@1019] - Closed socket connection for client /127.0.0.1:42378 (no session established for client) [junit] 2017-02-24 03:18:48,215 [myid:] - INFO [main:JMXEnv@230] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-24 03:18:48,218 [myid:] - INFO [main:JMXEnv@247] - expect:InMemoryDataTree [junit] 2017-02-24 03:18:48,218 [myid:] - INFO [main:JMXEnv@251] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2017-02-24 03:18:48,218 [myid:] - INFO [main:JMXEnv@247] - expect:StandaloneServer_port [junit] 2017-02-24 03:18:48,218 [myid:] - INFO [main:JMXEnv@251] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2017-02-24 03:18:48,219 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 35675 [junit] 2017-02-24 03:18:48,219 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2017-02-24 03:18:48,219 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2017-02-24 03:18:48,219 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2017-02-24 03:18:48,284 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x15a6e20cd7e closed [junit] 2017-02-24 03:18:48,284 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2017-02-24 03:18:48,284 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x15a6e20cd7e [junit] 2017-02-24 03:18:48,284 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method [junit] 2017-02-24 03:18:48,285 [myid:] - INFO [main:ZooKeeperServer@505] - shutting down [junit] 2017-02-24 03:18:48,285 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-24 03:18:48,285
[jira] [Commented] (ZOOKEEPER-2696) Eclipse ant task no longer determines correct classpath for tests after ZOOKEEPER-2689
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881821#comment-15881821 ] ASF GitHub Bot commented on ZOOKEEPER-2696: --- Github user rakeshadr commented on the issue: https://github.com/apache/zookeeper/pull/178 As per the offline discussion with @afine, identified that `checkstyle ant task` is not working in branch-3.4 and will be raising a new jira task to correct this. ``` checkstyle.check: set-checkstyle-classpath: checkstyle: [checkstyle] log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.BeanUtils). [checkstyle] log4j:WARN Please initialize the log4j system properly. [checkstyle] log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. BUILD FAILED $ZK\git\b-3-4-new\build.xml:1595: Unable to create a Checker: cannot initialize module PackageHtml - Unable to instantiate PackageHtml ``` Thanks @afine for the patience. +1 for the simple fix. I will merge this PR to branch-3.4, tomorrow if there is no objection from anyone. > Eclipse ant task no longer determines correct classpath for tests after > ZOOKEEPER-2689 > -- > > Key: ZOOKEEPER-2696 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2696 > Project: ZooKeeper > Issue Type: Bug >Affects Versions: 3.4.10 >Reporter: Abraham Fine >Assignee: Abraham Fine > > Following the changes made in ZOOKEEPER-2689 IDE's using the .classpath file > generated by the eclipse ant task (I tested both idea and eclipse) cannot > compile the tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] zookeeper issue #178: ZOOKEEPER-2696: Eclipse ant ask no longer determines c...
Github user rakeshadr commented on the issue: https://github.com/apache/zookeeper/pull/178 As per the offline discussion with @afine, identified that `checkstyle ant task` is not working in branch-3.4 and will be raising a new jira task to correct this. ``` checkstyle.check: set-checkstyle-classpath: checkstyle: [checkstyle] log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.BeanUtils). [checkstyle] log4j:WARN Please initialize the log4j system properly. [checkstyle] log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. BUILD FAILED $ZK\git\b-3-4-new\build.xml:1595: Unable to create a Checker: cannot initialize module PackageHtml - Unable to instantiate PackageHtml ``` Thanks @afine for the patience. +1 for the simple fix. I will merge this PR to branch-3.4, tomorrow if there is no objection from anyone. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2707) Fix "Unexpected bean exists!" issue in WatcherTests
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881605#comment-15881605
]
Abraham Fine commented on ZOOKEEPER-2707:
-
This appears to be related to ZOOKEEPER-1858.
The problem looks like it comes from`Assert.assertEquals("Unexpected bean
exists!", 0, children.size());` in `ClientBase#verifyUnexpectedBeans` which I
think causes a race condition.
I think the first occurrence of this failure can be found here:
http://apache.markmail.org/thread/5nuvqfufpvkymlli dating 2/24/14 a month after
ZOOKEEPER-1858 was merged in.
I have been trying to reproduce locally but have not been successful yet.
I'm wondering how valuable this assertion is. Is this something we can just
remove or do we need to find a better solution?
Would appreciate your insight on this issue [~rakeshr].
> Fix "Unexpected bean exists!" issue in WatcherTests
> ---
>
> Key: ZOOKEEPER-2707
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2707
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: tests
>Affects Versions: 3.4.10, 3.5.3
>Reporter: Abraham Fine
>Assignee: Abraham Fine
>
> All the WatcherTests occasionally fail with:
> {code}
> Error Message:
> Unexpected bean exists! expected:<0> but was:<1>
> Stack Trace:
> junit.framework.AssertionFailedError: Unexpected bean exists! expected:<0>
> but was:<1>
> at
> org.apache.zookeeper.test.ClientBase.verifyUnexpectedBeans(ClientBase.java:498)
> at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:477)
> at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
> at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:76)
> {code}
> Here is an example:
> https://builds.apache.org/job/ZooKeeper_branch35_openjdk7/422/
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
ZooKeeper_branch34 - Build # 1846 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34/1846/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 3.62 KB...] collisions:0 txqueuelen:0 RX bytes:695758 (695.7 KB) TX bytes:27785671 (27.7 MB) core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 386178 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 6 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 10240 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 6 /home/jenkins/tools/ant/latest/bin/ant clean Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) mv: cannot stat 'build/*.tar.gz': No such file or directory mv: cannot stat 'build/*.jar': No such file or directory mv: cannot stat 'build/test/findbugs': No such file or directory mv: cannot stat 'build/docs/api': No such file or directory Build Failed Build step 'Execute shell' marked build as failure Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
ZooKeeper-trunk - Build # 3290 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk/3290/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 4.10 KB...] POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 10240 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 6 /home/jenkins/tools/ant/latest/bin/ant clean Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) mv: cannot stat 'build/*.tar.gz': No such file or directory mv: cannot stat 'build/*.jar': No such file or directory mv: cannot stat 'build/test/findbugs': No such file or directory mv: cannot stat 'build/docs/api': No such file or directory Build Failed Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE [WARNINGS] Skipping publisher since build result is FAILURE Archiving artifacts Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording fingerprints Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Recording test results Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Publishing Javadoc Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7 ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Updated] (ZOOKEEPER-2707) Fix "Unexpected bean exists!" issue in WatcherTests
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abraham Fine updated ZOOKEEPER-2707:
Affects Version/s: 3.5.3
3.4.10
> Fix "Unexpected bean exists!" issue in WatcherTests
> ---
>
> Key: ZOOKEEPER-2707
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2707
> Project: ZooKeeper
> Issue Type: Sub-task
> Components: tests
>Affects Versions: 3.4.10, 3.5.3
>Reporter: Abraham Fine
>Assignee: Abraham Fine
>
> All the WatcherTests occasionally fail with:
> {code}
> Error Message:
> Unexpected bean exists! expected:<0> but was:<1>
> Stack Trace:
> junit.framework.AssertionFailedError: Unexpected bean exists! expected:<0>
> but was:<1>
> at
> org.apache.zookeeper.test.ClientBase.verifyUnexpectedBeans(ClientBase.java:498)
> at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:477)
> at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
> at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:76)
> {code}
> Here is an example:
> https://builds.apache.org/job/ZooKeeper_branch35_openjdk7/422/
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (ZOOKEEPER-2707) Fix "Unexpected bean exists!" issue in WatcherTests
Abraham Fine created ZOOKEEPER-2707:
---
Summary: Fix "Unexpected bean exists!" issue in WatcherTests
Key: ZOOKEEPER-2707
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2707
Project: ZooKeeper
Issue Type: Sub-task
Reporter: Abraham Fine
Assignee: Abraham Fine
All the WatcherTests occasionally fail with:
{code}
Error Message:
Unexpected bean exists! expected:<0> but was:<1>
Stack Trace:
junit.framework.AssertionFailedError: Unexpected bean exists! expected:<0> but
was:<1>
at
org.apache.zookeeper.test.ClientBase.verifyUnexpectedBeans(ClientBase.java:498)
at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:477)
at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:76)
{code}
Here is an example:
https://builds.apache.org/job/ZooKeeper_branch35_openjdk7/422/
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (ZOOKEEPER-2706) checkstyle broken on branch-3.4
Abraham Fine created ZOOKEEPER-2706:
---
Summary: checkstyle broken on branch-3.4
Key: ZOOKEEPER-2706
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2706
Project: ZooKeeper
Issue Type: Bug
Reporter: Abraham Fine
Assignee: Abraham Fine
While working on ZOOKEEPER-2696, [~rakeshr] and I noticed that checkstyle is
failing to execute on branch-3.4 with the following error:
{code}
BUILD FAILED
/Users/abefine/cloudera_code/zookeeper/build.xml:1595: Unable to create a
Checker: cannot initialize module PackageHtml - Unable to instantiate
PackageHtml
{code}
This seems similar to ZOOKEEPER-412
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2704) Run Jepsen against branch-3.5 / master of ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880876#comment-15880876 ] Shawn Heisey commented on ZOOKEEPER-2704: - If there's any way to automate jepsen testing so new commits are tested without any manual action, that would help ensure that the software remains stable. > Run Jepsen against branch-3.5 / master of ZooKeeper > --- > > Key: ZOOKEEPER-2704 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2704 > Project: ZooKeeper > Issue Type: Sub-task > Components: tests >Affects Versions: 3.5.2 >Reporter: Michael Han > Labels: test > Fix For: 3.6.0 > > > The [Jepsen report|https://aphyr.com/posts/291-jepsen-zookeeper] on ZooKeeper > was using an old version of ZooKeeper (3.4.5). It would be good to run Jepsen > on trunk / branch-3.5 and see what happens. This will also give our > confidence on the quality of the upcoming 3.5 stable release. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
ZooKeeper_branch35_solaris - Build # 449 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_solaris/449/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 58.54 MB...] [junit] 2017-02-23 17:16:03,793 [myid:] - INFO [main:ClientBase@361] - STARTING server instance 127.0.0.1:11222 [junit] 2017-02-23 17:16:03,793 [myid:] - INFO [main:ZooKeeperServer@907] - minSessionTimeout set to 6000 [junit] 2017-02-23 17:16:03,793 [myid:] - INFO [main:ZooKeeperServer@916] - maxSessionTimeout set to 6 [junit] 2017-02-23 17:16:03,793 [myid:] - INFO [main:ZooKeeperServer@159] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test6297383373482920872.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test6297383373482920872.junit.dir/version-2 [junit] 2017-02-23 17:16:03,794 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test6297383373482920872.junit.dir/version-2/snapshot.b [junit] 2017-02-23 17:16:03,796 [myid:] - INFO [main:FileTxnSnapLog@320] - Snapshotting: 0xb to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch35_solaris/build/test/tmp/test6297383373482920872.junit.dir/version-2/snapshot.b [junit] 2017-02-23 17:16:03,797 [myid:] - ERROR [main:ZooKeeperServer@505] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-23 17:16:03,797 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 [junit] 2017-02-23 17:16:03,798 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:39596 [junit] 2017-02-23 17:16:03,798 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:39596 [junit] 2017-02-23 17:16:03,798 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output [junit] 2017-02-23 17:16:03,799 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@614] - Closed socket connection for client /127.0.0.1:39596 (no session established for client) [junit] 2017-02-23 17:16:03,799 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-23 17:16:03,800 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree [junit] 2017-02-23 17:16:03,801 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree [junit] 2017-02-23 17:16:03,801 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port [junit] 2017-02-23 17:16:03,801 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222 [junit] 2017-02-23 17:16:03,801 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 17890 [junit] 2017-02-23 17:16:03,801 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 24 [junit] 2017-02-23 17:16:03,801 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota [junit] 2017-02-23 17:16:03,802 [myid:] - INFO [main:ClientBase@543] - tearDown starting [junit] 2017-02-23 17:16:03,882 [myid:] - INFO [main:ZooKeeper@1324] - Session: 0x126e8f1c974 closed [junit] 2017-02-23 17:16:03,882 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x126e8f1c974 [junit] 2017-02-23 17:16:03,882 [myid:] - INFO [main:ClientBase@513] - STOPPING server [junit] 2017-02-23 17:16:03,882 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted [junit] 2017-02-23 17:16:03,883 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-23 17:16:03,882 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-23 17:16:03,882 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method [junit] 2017-02-23 17:16:03,883 [myid:] - INFO [main:ZooKeeperServer@541] - shutting down [junit] 2017-02-23 17:16:03,883 [myid:] - ERROR [main:ZooKeeperServer@505] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any
[jira] [Commented] (ZOOKEEPER-2705) Container node remains indefinitely after session has long expired!
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880780#comment-15880780
]
Steve Fitzgerald commented on ZOOKEEPER-2705:
-
Yeah this looks like the same issue.
Is there any timeline for releasing 3.5.3?
> Container node remains indefinitely after session has long expired!
> ---
>
> Key: ZOOKEEPER-2705
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2705
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
>Affects Versions: 3.5.1
> Environment: 5 x RHEL 2.6.32-431.29.2.el6.x86_64
>Reporter: Steve Fitzgerald
>
> Zookeeper version: 3.5.1-alpha
> Curator Framework version: 3.2.0
> We have a 5 node cluster. When we register a service instance everything is
> created within zookeeper successfully, e.g. for a service names
> "fake-test-service" I can see the following created:
> 1. /api/enablement/fake-test-service
> 2. /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5
> When I abnormally kill (kill -9) the process that the service is registered
> from I expect both of the above to get removed by zookeeper when it expires
> the session. But only
> /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5 gets
> removed successfully.
> Here is a snippet of the log file:
> {noformat}
> 2017-02-23 05:50:00,977 [myid:5] - TRACE
> [SessionTracker:SessionTrackerImpl@208][] - Session closing: 0x502dbce4df6
> 2017-02-23 05:50:00,977 [myid:5] - INFO
> [SessionTracker:ZooKeeperServer@384][] - Expiring session 0x502dbce4df6,
> timeout of 4ms exceeded
> 2017-02-23 05:50:00,977 [myid:5] - INFO
> [SessionTracker:QuorumZooKeeperServer@132][] - Submitting global closeSession
> request for session 0x502dbce4df6
> 2017-02-23 05:50:00,977 [myid:5] - TRACE [ProcessThread(sid:5
> cport:-1)::ZooTrace@90][] - :Psessionid:0x502dbce4df6 type:closeSession
> cxid:0x0 zxid:0xfffe txntype:unknown reqpath:n/a
> 2017-02-23 05:50:00,978 [myid:5] - TRACE [ProcessThread(sid:5
> cport:-1)::SessionTrackerImpl@208][] - Session closing: 0x502dbce4df6
> 2017-02-23 05:50:00,978 [myid:5] - INFO [ProcessThread(sid:5
> cport:-1)::PrepRequestProcessor@649][] - Processed session termination for
> sessionid: 0x502dbce4df6
> 2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5
> cport:-1)::CommitProcessor@340][] - Processing request::
> sessionid:0x502dbce4df6 type:closeSession cxid:0x0 zxid:0x1d0003
> txntype:-11 reqpath:n/a
> 2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5
> cport:-1)::Leader@1066][] - Proposing:: sessionid:0x502dbce4df6
> type:closeSession cxid:0x0 zxid:0x1d0003 txntype:-11 reqpath:n/a
> 2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@787][] - Ack
> zxid: 0x1d0003
> 2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@790][] -
> outstanding proposal: 0x1d0003
> 2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@793][] -
> outstanding proposals all
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.164:38716:Leader@787][] - Ack zxid: 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.164:38716:Leader@790][] - outstanding proposal:
> 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.164:38716:Leader@793][] - outstanding proposals all
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.161:55588:Leader@787][] - Ack zxid: 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.161:55588:Leader@790][] - outstanding proposal:
> 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.161:55588:Leader@793][] - outstanding proposals all
> 2017-02-23 05:50:00,982 [myid:5] - DEBUG
> [LearnerHandler-/10.24.128.161:55588:CommitProcessor@327][] - Committing
> request:: sessionid:0x502dbce4df6 type:closeSession cxid:0x0
> zxid:0x1d0003 txntype:-11 reqpath:n/a
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.162:47580:Leader@787][] - Ack zxid: 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.162:47580:Leader@793][] - outstanding proposals all
> 2017-02-23 05:50:00,983 [myid:5] - DEBUG
> [LearnerHandler-/10.24.128.162:47580:Leader@808][] - outstanding is 0
> 2017-02-23 05:50:00,983 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.160:41119:Leader@787][] - Ack zxid: 0x1d0003
> 2017-02-23 05:50:00,983 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.160:41119:Leader@793][] - outstanding proposals all
> 2017-02-23 05:50:00,983 [myid:5] - DEBUG
> [LearnerHandler-/10.24.128.160:41119:Leader@808][] - outstanding is 0
> 2017-02-23 05:50:00,983
[jira] [Commented] (ZOOKEEPER-2705) Container node remains indefinitely after session has long expired!
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880700#comment-15880700
]
Edward Ribeiro commented on ZOOKEEPER-2705:
---
Hi [~stondofitz],
Looks like you hit https://issues.apache.org/jira/browse/ZOOKEEPER-2464 that
was recently fixed by https://issues.apache.org/jira/browse/ZOOKEEPER-2680
TL;DR: {{DataNode.getChildren()}} could either return an empty collection or
null, and this broke ContainerManager at line 151, as shown in your log.
Could you confirm it is the same issue, please?
> Container node remains indefinitely after session has long expired!
> ---
>
> Key: ZOOKEEPER-2705
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2705
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
>Affects Versions: 3.5.1
> Environment: 5 x RHEL 2.6.32-431.29.2.el6.x86_64
>Reporter: Steve Fitzgerald
>
> Zookeeper version: 3.5.1-alpha
> Curator Framework version: 3.2.0
> We have a 5 node cluster. When we register a service instance everything is
> created within zookeeper successfully, e.g. for a service names
> "fake-test-service" I can see the following created:
> 1. /api/enablement/fake-test-service
> 2. /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5
> When I abnormally kill (kill -9) the process that the service is registered
> from I expect both of the above to get removed by zookeeper when it expires
> the session. But only
> /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5 gets
> removed successfully.
> Here is a snippet of the log file:
> {noformat}
> 2017-02-23 05:50:00,977 [myid:5] - TRACE
> [SessionTracker:SessionTrackerImpl@208][] - Session closing: 0x502dbce4df6
> 2017-02-23 05:50:00,977 [myid:5] - INFO
> [SessionTracker:ZooKeeperServer@384][] - Expiring session 0x502dbce4df6,
> timeout of 4ms exceeded
> 2017-02-23 05:50:00,977 [myid:5] - INFO
> [SessionTracker:QuorumZooKeeperServer@132][] - Submitting global closeSession
> request for session 0x502dbce4df6
> 2017-02-23 05:50:00,977 [myid:5] - TRACE [ProcessThread(sid:5
> cport:-1)::ZooTrace@90][] - :Psessionid:0x502dbce4df6 type:closeSession
> cxid:0x0 zxid:0xfffe txntype:unknown reqpath:n/a
> 2017-02-23 05:50:00,978 [myid:5] - TRACE [ProcessThread(sid:5
> cport:-1)::SessionTrackerImpl@208][] - Session closing: 0x502dbce4df6
> 2017-02-23 05:50:00,978 [myid:5] - INFO [ProcessThread(sid:5
> cport:-1)::PrepRequestProcessor@649][] - Processed session termination for
> sessionid: 0x502dbce4df6
> 2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5
> cport:-1)::CommitProcessor@340][] - Processing request::
> sessionid:0x502dbce4df6 type:closeSession cxid:0x0 zxid:0x1d0003
> txntype:-11 reqpath:n/a
> 2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5
> cport:-1)::Leader@1066][] - Proposing:: sessionid:0x502dbce4df6
> type:closeSession cxid:0x0 zxid:0x1d0003 txntype:-11 reqpath:n/a
> 2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@787][] - Ack
> zxid: 0x1d0003
> 2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@790][] -
> outstanding proposal: 0x1d0003
> 2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@793][] -
> outstanding proposals all
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.164:38716:Leader@787][] - Ack zxid: 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.164:38716:Leader@790][] - outstanding proposal:
> 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.164:38716:Leader@793][] - outstanding proposals all
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.161:55588:Leader@787][] - Ack zxid: 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.161:55588:Leader@790][] - outstanding proposal:
> 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.161:55588:Leader@793][] - outstanding proposals all
> 2017-02-23 05:50:00,982 [myid:5] - DEBUG
> [LearnerHandler-/10.24.128.161:55588:CommitProcessor@327][] - Committing
> request:: sessionid:0x502dbce4df6 type:closeSession cxid:0x0
> zxid:0x1d0003 txntype:-11 reqpath:n/a
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.162:47580:Leader@787][] - Ack zxid: 0x1d0003
> 2017-02-23 05:50:00,982 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.162:47580:Leader@793][] - outstanding proposals all
> 2017-02-23 05:50:00,983 [myid:5] - DEBUG
> [LearnerHandler-/10.24.128.162:47580:Leader@808][] - outstanding is 0
> 2017-02-23 05:50:00,983 [myid:5] - TRACE
> [LearnerHandler-/10.24.128.160:41119:Leader@787][] -
ZooKeeper_branch34_openjdk7 - Build # 1402 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/1402/
###
## LAST 60 LINES OF THE CONSOLE
###
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H15 (ubuntu) in workspace
/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url git://git.apache.org/zookeeper.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching upstream changes from git://git.apache.org/zookeeper.git
> git --version # timeout=10
> git fetch --tags --progress git://git.apache.org/zookeeper.git
> +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/branch-3.4^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/branch-3.4^{commit} # timeout=10
Checking out Revision 373b82bab843cc84c22b99f6511f8fea974fd2b4
(refs/remotes/origin/branch-3.4)
> git config core.sparsecheckout # timeout=10
> git checkout -f 373b82bab843cc84c22b99f6511f8fea974fd2b4
> git rev-list 373b82bab843cc84c22b99f6511f8fea974fd2b4 # timeout=10
No emails were triggered.
[ZooKeeper_branch34_openjdk7] $ /home/jenkins/tools/ant/latest/bin/ant
-Dtest.output=yes -Dtest.junit.threads=8 -Dtest.junit.output.format=xml
-Djavac.target=1.7 clean test-core-java
Error: JAVA_HOME is not defined correctly.
We cannot execute /usr/lib/jvm/java-7-openjdk-amd64//bin/java
Build step 'Invoke Ant' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
ZooKeeper_branch34_solaris - Build # 1483 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_solaris/1483/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 25.08 MB...] [junit] 2017-02-23 13:58:22,413 [myid:] - INFO [main:PrepRequestProcessor@764] - Shutting down [junit] 2017-02-23 13:58:22,413 [myid:] - INFO [main:SyncRequestProcessor@208] - Shutting down [junit] 2017-02-23 13:58:22,414 [myid:] - INFO [ProcessThread(sid:0 cport:11221)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop! [junit] 2017-02-23 13:58:22,414 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@186] - SyncRequestProcessor exited! [junit] 2017-02-23 13:58:22,414 [myid:] - INFO [main:FinalRequestProcessor@402] - shutdown of request processor complete [junit] 2017-02-23 13:58:22,414 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-23 13:58:22,415 [myid:] - INFO [main:JMXEnv@147] - ensureOnly:[] [junit] 2017-02-23 13:58:22,415 [myid:] - INFO [main:ClientBase@445] - STARTING server [junit] 2017-02-23 13:58:22,416 [myid:] - INFO [main:ClientBase@366] - CREATING server instance 127.0.0.1:11221 [junit] 2017-02-23 13:58:22,416 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2017-02-23 13:58:22,416 [myid:] - INFO [main:ClientBase@341] - STARTING server instance 127.0.0.1:11221 [junit] 2017-02-23 13:58:22,417 [myid:] - INFO [main:ZooKeeperServer@173] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/build/test/tmp/test1524573733394292484.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper_branch34_solaris/build/test/tmp/test1524573733394292484.junit.dir/version-2 [junit] 2017-02-23 13:58:22,419 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-23 13:58:22,419 [myid:] - INFO [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11221 [junit] 2017-02-23 13:58:22,419 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:53533 [junit] 2017-02-23 13:58:22,420 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11221:NIOServerCnxn@838] - Processing stat command from /127.0.0.1:53533 [junit] 2017-02-23 13:58:22,420 [myid:] - INFO [Thread-5:NIOServerCnxn$StatCommand@674] - Stat command output [junit] 2017-02-23 13:58:22,420 [myid:] - INFO [Thread-5:NIOServerCnxn@1019] - Closed socket connection for client /127.0.0.1:53533 (no session established for client) [junit] 2017-02-23 13:58:22,420 [myid:] - INFO [main:JMXEnv@230] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-23 13:58:22,426 [myid:] - INFO [main:JMXEnv@247] - expect:InMemoryDataTree [junit] 2017-02-23 13:58:22,426 [myid:] - INFO [main:JMXEnv@251] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11221,name1=InMemoryDataTree [junit] 2017-02-23 13:58:22,426 [myid:] - INFO [main:JMXEnv@247] - expect:StandaloneServer_port [junit] 2017-02-23 13:58:22,426 [myid:] - INFO [main:JMXEnv@251] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11221 [junit] 2017-02-23 13:58:22,426 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@58] - Memory used 9183 [junit] 2017-02-23 13:58:22,426 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@63] - Number of threads 20 [junit] 2017-02-23 13:58:22,427 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@78] - FINISHED TEST METHOD testQuota [junit] 2017-02-23 13:58:22,427 [myid:] - INFO [main:ClientBase@522] - tearDown starting [junit] 2017-02-23 13:58:22,510 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x15a6b43fb1b closed [junit] 2017-02-23 13:58:22,510 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x15a6b43fb1b [junit] 2017-02-23 13:58:22,510 [myid:] - INFO [main:ClientBase@492] - STOPPING server [junit] 2017-02-23 13:58:22,510 [myid:] - INFO [main:ZooKeeperServer@505] - shutting down [junit] 2017-02-23 13:58:22,511 [myid:] - ERROR [main:ZooKeeperServer@472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-23 13:58:22,511 [myid:] - INFO [main:SessionTrackerImpl@225] - Shutting down [junit] 2017-02-23 13:58:22,511 [myid:] - INFO [main:PrepRequestProcessor@764] - Shutting down [junit] 2017-02-23 13:58:22,511 [myid:] -
[jira] [Commented] (ZOOKEEPER-832) Invalid session id causes infinite loop during automatic reconnect
[ https://issues.apache.org/jira/browse/ZOOKEEPER-832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880411#comment-15880411 ] Roman Gavryliuk commented on ZOOKEEPER-832: --- Just for a general info - we've also faced the same issue with Zookeeper 3.4.5 and Hive 1.2. Because this patch in not yet committed we were able to handle this problem by increasing zxid value via external script, which updates some dummy Znode value. > Invalid session id causes infinite loop during automatic reconnect > -- > > Key: ZOOKEEPER-832 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-832 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.5, 3.5.0 > Environment: All >Reporter: Ryan Holmes >Assignee: Mohammad Arshad >Priority: Critical > Fix For: 3.5.3, 3.6.0, 3.4.11 > > Attachments: ZOOKEEPER-832.patch, ZOOKEEPER-832.patch, > ZOOKEEPER-832.patch, ZOOKEEPER-832.patch, ZOOKEEPER-832.patch, > ZOOKEEPER-832.patch, ZOOKEEPER-832.patch, ZOOKEEPER-832.patch, > ZOOKEEPER-832.patch, ZOOKEEPER-832.patch > > > Steps to reproduce: > 1.) Connect to a standalone server using the Java client. > 2.) Stop the server. > 3.) Delete the contents of the data directory (i.e. the persisted session > data). > 4.) Start the server. > The client now automatically tries to reconnect but the server refuses the > connection because the session id is invalid. The client and server are now > in an infinite loop of attempted and rejected connections. While this > situation represents a catastrophic failure and the current behavior is not > incorrect, it appears that there is no way to detect this situation on the > client and therefore no way to recover. > The suggested improvement is to send an event to the default watcher > indicating that the current state is "session invalid", similar to how the > "session expired" state is handled. > Server log output (repeats indefinitely): > 2010-08-05 11:48:08,283 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@250] - > Accepted socket connection from /127.0.0.1:63292 > 2010-08-05 11:48:08,284 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@751] - Refusing > session request for client /127.0.0.1:63292 as it has seen zxid 0x44 our last > zxid is 0x0 client must try another server > 2010-08-05 11:48:08,284 - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1434] - Closed > socket connection for client /127.0.0.1:63292 (no session established for > client) > Client log output (repeats indefinitely): > 11:47:17 org.apache.zookeeper.ClientCnxn startConnect INFO line 1000 - > Opening socket connection to server localhost/127.0.0.1:2181 > 11:47:17 org.apache.zookeeper.ClientCnxn run WARN line 1120 - Session > 0x12a3ae4e893000a for server null, unexpected error, closing socket > connection and attempting reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > 11:47:17 org.apache.zookeeper.ClientCnxn cleanup DEBUG line 1167 - Ignoring > exception during shutdown input > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1129) > 11:47:17 org.apache.zookeeper.ClientCnxn cleanup DEBUG line 1174 - Ignoring > exception during shutdown output > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1171) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1129) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (ZOOKEEPER-2705) Container node remains indefinitely after session has long expired!
Steve Fitzgerald created ZOOKEEPER-2705:
---
Summary: Container node remains indefinitely after session has
long expired!
Key: ZOOKEEPER-2705
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2705
Project: ZooKeeper
Issue Type: Bug
Components: quorum
Affects Versions: 3.5.1
Environment: 5 x RHEL 2.6.32-431.29.2.el6.x86_64
Reporter: Steve Fitzgerald
Zookeeper version: 3.5.1-alpha
Curator Framework version: 3.2.0
We have a 5 node cluster. When we register a service instance everything is
created within zookeeper successfully, e.g. for a service names
"fake-test-service" I can see the following created:
1. /api/enablement/fake-test-service
2. /api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5
When I abnormally kill (kill -9) the process that the service is registered
from I expect both of the above to get removed by zookeeper when it expires the
session. But only
/api/enablement/fake-test-service/bb831396-5c55-4456-a7c0-5950ba294fd5 gets
removed successfully.
Here is a snippet of the log file:
{noformat}
2017-02-23 05:50:00,977 [myid:5] - TRACE
[SessionTracker:SessionTrackerImpl@208][] - Session closing: 0x502dbce4df6
2017-02-23 05:50:00,977 [myid:5] - INFO [SessionTracker:ZooKeeperServer@384][]
- Expiring session 0x502dbce4df6, timeout of 4ms exceeded
2017-02-23 05:50:00,977 [myid:5] - INFO
[SessionTracker:QuorumZooKeeperServer@132][] - Submitting global closeSession
request for session 0x502dbce4df6
2017-02-23 05:50:00,977 [myid:5] - TRACE [ProcessThread(sid:5
cport:-1)::ZooTrace@90][] - :Psessionid:0x502dbce4df6 type:closeSession
cxid:0x0 zxid:0xfffe txntype:unknown reqpath:n/a
2017-02-23 05:50:00,978 [myid:5] - TRACE [ProcessThread(sid:5
cport:-1)::SessionTrackerImpl@208][] - Session closing: 0x502dbce4df6
2017-02-23 05:50:00,978 [myid:5] - INFO [ProcessThread(sid:5
cport:-1)::PrepRequestProcessor@649][] - Processed session termination for
sessionid: 0x502dbce4df6
2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5
cport:-1)::CommitProcessor@340][] - Processing request::
sessionid:0x502dbce4df6 type:closeSession cxid:0x0 zxid:0x1d0003
txntype:-11 reqpath:n/a
2017-02-23 05:50:00,978 [myid:5] - DEBUG [ProcessThread(sid:5
cport:-1)::Leader@1066][] - Proposing:: sessionid:0x502dbce4df6
type:closeSession cxid:0x0 zxid:0x1d0003 txntype:-11 reqpath:n/a
2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@787][] - Ack
zxid: 0x1d0003
2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@790][] -
outstanding proposal: 0x1d0003
2017-02-23 05:50:00,981 [myid:5] - TRACE [SyncThread:5:Leader@793][] -
outstanding proposals all
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.164:38716:Leader@787][] - Ack zxid: 0x1d0003
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.164:38716:Leader@790][] - outstanding proposal:
0x1d0003
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.164:38716:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.161:55588:Leader@787][] - Ack zxid: 0x1d0003
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.161:55588:Leader@790][] - outstanding proposal:
0x1d0003
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.161:55588:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,982 [myid:5] - DEBUG
[LearnerHandler-/10.24.128.161:55588:CommitProcessor@327][] - Committing
request:: sessionid:0x502dbce4df6 type:closeSession cxid:0x0
zxid:0x1d0003 txntype:-11 reqpath:n/a
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.162:47580:Leader@787][] - Ack zxid: 0x1d0003
2017-02-23 05:50:00,982 [myid:5] - TRACE
[LearnerHandler-/10.24.128.162:47580:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,983 [myid:5] - DEBUG
[LearnerHandler-/10.24.128.162:47580:Leader@808][] - outstanding is 0
2017-02-23 05:50:00,983 [myid:5] - TRACE
[LearnerHandler-/10.24.128.160:41119:Leader@787][] - Ack zxid: 0x1d0003
2017-02-23 05:50:00,983 [myid:5] - TRACE
[LearnerHandler-/10.24.128.160:41119:Leader@793][] - outstanding proposals all
2017-02-23 05:50:00,983 [myid:5] - DEBUG
[LearnerHandler-/10.24.128.160:41119:Leader@808][] - outstanding is 0
2017-02-23 05:50:00,983 [myid:5] - DEBUG
[CommitProcWorkThread-1:FinalRequestProcessor@91][] - Processing request::
sessionid:0x502dbce4df6 type:closeSession cxid:0x0 zxid:0x1d0003
txntype:-11 reqpath:n/a
2017-02-23 05:50:00,983 [myid:5] - TRACE [CommitProcWorkThread-1:ZooTrace@90][]
- :Esessionid:0x502dbce4df6 type:closeSession cxid:0x0 zxid:0x1d0003
txntype:-11 reqpath:n/a
2017-02-23 05:50:00,983 [myid:5] - DEBUG
ZooKeeper-trunk-solaris - Build # 1517 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-solaris/1517/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 58.60 MB...] [junit] 2017-02-23 08:39:05,558 [myid:] - INFO [main:ClientBase@376] - STARTING server instance 127.0.0.1:11222 [junit] 2017-02-23 08:39:05,559 [myid:] - INFO [main:ZooKeeperServer@908] - minSessionTimeout set to 6000 [junit] 2017-02-23 08:39:05,559 [myid:] - INFO [main:ZooKeeperServer@917] - maxSessionTimeout set to 6 [junit] 2017-02-23 08:39:05,559 [myid:] - INFO [main:ZooKeeperServer@160] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test6452893461893143809.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test6452893461893143809.junit.dir/version-2 [junit] 2017-02-23 08:39:05,560 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test6452893461893143809.junit.dir/version-2/snapshot.b [junit] 2017-02-23 08:39:05,562 [myid:] - INFO [main:FileTxnSnapLog@346] - Snapshotting: 0xb to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/build/test/tmp/test6452893461893143809.junit.dir/version-2/snapshot.b [junit] 2017-02-23 08:39:05,563 [myid:] - ERROR [main:ZooKeeperServer@506] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-02-23 08:39:05,563 [myid:] - INFO [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222 [junit] 2017-02-23 08:39:05,564 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44975 [junit] 2017-02-23 08:39:05,564 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:44975 [junit] 2017-02-23 08:39:05,565 [myid:] - INFO [NIOWorkerThread-1:StatCommand@49] - Stat command output [junit] 2017-02-23 08:39:05,565 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@614] - Closed socket connection for client /127.0.0.1:44975 (no session established for client) [junit] 2017-02-23 08:39:05,565 [myid:] - INFO [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port] [junit] 2017-02-23 08:39:05,566 [myid:] - INFO [main:JMXEnv@245] - expect:InMemoryDataTree [junit] 2017-02-23 08:39:05,566 [myid:] - INFO [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree [junit] 2017-02-23 08:39:05,567 [myid:] - INFO [main:JMXEnv@245] - expect:StandaloneServer_port [junit] 2017-02-23 08:39:05,567 [myid:] - INFO [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222 [junit] 2017-02-23 08:39:05,567 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 17905 [junit] 2017-02-23 08:39:05,567 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 24 [junit] 2017-02-23 08:39:05,567 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota [junit] 2017-02-23 08:39:05,567 [myid:] - INFO [main:ClientBase@558] - tearDown starting [junit] 2017-02-23 08:39:05,642 [myid:] - INFO [main:ZooKeeper@1324] - Session: 0x126e7187d52 closed [junit] 2017-02-23 08:39:05,642 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x126e7187d52 [junit] 2017-02-23 08:39:05,642 [myid:] - INFO [main:ClientBase@528] - STOPPING server [junit] 2017-02-23 08:39:05,643 [myid:] - INFO [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted [junit] 2017-02-23 08:39:05,643 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-23 08:39:05,643 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2017-02-23 08:39:05,643 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method [junit] 2017-02-23 08:39:05,643 [myid:] - INFO [main:ZooKeeperServer@542] - shutting down [junit] 2017-02-23 08:39:05,643 [myid:] - ERROR [main:ZooKeeperServer@506] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR
ZooKeeper_branch35_jdk7 - Build # 861 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_jdk7/861/
###
## LAST 60 LINES OF THE CONSOLE
###
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on proserpina-test (ubuntu) in workspace
/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch35_jdk7
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url git://git.apache.org/zookeeper.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching upstream changes from git://git.apache.org/zookeeper.git
> git --version # timeout=10
> git fetch --tags --progress git://git.apache.org/zookeeper.git
> +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/branch-3.5^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/branch-3.5^{commit} # timeout=10
Checking out Revision 1912fa8d63e6bbfdb5bb96bec85efa7c94e131e9
(refs/remotes/origin/branch-3.5)
> git config core.sparsecheckout # timeout=10
> git checkout -f 1912fa8d63e6bbfdb5bb96bec85efa7c94e131e9
> git rev-list 1912fa8d63e6bbfdb5bb96bec85efa7c94e131e9 # timeout=10
No emails were triggered.
[ZooKeeper_branch35_jdk7] $ /home/jenkins/tools/ant/latest/bin/ant
-Dtest.output=yes -Dtest.junit.threads=8 -Dtest.junit.output.format=xml
-Djavac.target=1.7 clean test-core-java
Exception in thread "main" java.lang.UnsupportedClassVersionError:
org/apache/tools/ant/launch/Launcher : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Build step 'Invoke Ant' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
