Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20605/ Java: 32bit/jdk1.8.0_144 -client -XX:+UseConcMarkSweepGC
2 tests failed. FAILED: org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry Error Message: Captured an uncaught exception in thread: Thread[id=19517, name=OverseerAutoScalingTriggerThread-98769953126350851-dummy.host.com:8984_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=19517, name=OverseerAutoScalingTriggerThread-98769953126350851-dummy.host.com:8984_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers] at __randomizedtesting.SeedInfo.seed([E3C6A345C2B6F468:3519821FF4ABE7F0]:0) Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events at __randomizedtesting.SeedInfo.seed([E3C6A345C2B6F468]:0) at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107) at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:30) at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398) at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149) at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:306) at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:303) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:303) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:512) at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:97) at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78) at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105) ... 5 more FAILED: org.apache.solr.cloud.HealthCheckHandlerTest.testHealthCheckHandlerSolrJ Error Message: Error from server at https://127.0.0.1:33941/solr: Host Unavailable: Not connected to zk Stack Trace: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:33941/solr: Host Unavailable: Not connected to zk at __randomizedtesting.SeedInfo.seed([E3C6A345C2B6F468:49357C51B950BC69]:0) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195) at org.apache.solr.cloud.HealthCheckHandlerTest.testHealthCheckHandlerSolrJ(HealthCheckHandlerTest.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) Build Log: [...truncated 13171 lines...] [junit4] Suite: org.apache.solr.cloud.HealthCheckHandlerTest [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HealthCheckHandlerTest_E3C6A345C2B6F468-001/init-core-data-001 [junit4] 2> 1866487 WARN (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=66 numCloses=66 [junit4] 2> 1866488 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false [junit4] 2> 1866489 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN) [junit4] 2> 1866491 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> 1866491 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HealthCheckHandlerTest_E3C6A345C2B6F468-001/tempDir-001 [junit4] 2> 1866492 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 1866492 INFO (Thread-2064) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1866492 INFO (Thread-2064) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 1866496 ERROR (Thread-2064) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> 1866592 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:43673 [junit4] 2> 1866601 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception [junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ee6ba820f0000, likely client has closed socket [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239) [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 1866605 INFO (jetty-launcher-3363-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531 [junit4] 2> 1866606 INFO (jetty-launcher-3363-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@14a58a9{/solr,null,AVAILABLE} [junit4] 2> 1866619 INFO (jetty-launcher-3363-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@19e8ae6{SSL,[ssl, http/1.1]}{127.0.0.1:33941} [junit4] 2> 1866620 INFO (jetty-launcher-3363-thread-1) [ ] o.e.j.s.Server Started @1868073ms [junit4] 2> 1866620 INFO (jetty-launcher-3363-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33941} [junit4] 2> 1866620 ERROR (jetty-launcher-3363-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1866620 INFO (jetty-launcher-3363-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0 [junit4] 2> 1866620 INFO (jetty-launcher-3363-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1866620 INFO (jetty-launcher-3363-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null [junit4] 2> 1866620 INFO (jetty-launcher-3363-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-10-04T09:32:16.398Z [junit4] 2> 1866624 INFO (jetty-launcher-3363-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 1866635 INFO (jetty-launcher-3363-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43673/solr [junit4] 2> 1866718 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 1866718 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33941_solr [junit4] 2> 1866719 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.c.Overseer Overseer (id=98769930569056259-127.0.0.1:33941_solr-n_0000000000) starting [junit4] 2> 1866726 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33941_solr [junit4] 2> 1866732 INFO (zkCallback-3370-thread-1-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1866864 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33941.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@fea315 [junit4] 2> 1866877 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33941.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@fea315 [junit4] 2> 1866877 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33941.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@fea315 [junit4] 2> 1866879 INFO (jetty-launcher-3363-thread-1) [n:127.0.0.1:33941_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HealthCheckHandlerTest_E3C6A345C2B6F468-001/tempDir-001/node1/. [junit4] 2> 1866917 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1866918 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43673/solr ready [junit4] 2> 1866993 INFO (TEST-HealthCheckHandlerTest.testHealthCheckHandlerWithCloudClient-seed#[E3C6A345C2B6F468]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testHealthCheckHandlerWithCloudClient [junit4] 2> 1866995 INFO (TEST-HealthCheckHandlerTest.testHealthCheckHandlerWithCloudClient-seed#[E3C6A345C2B6F468]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testHealthCheckHandlerWithCloudClient [junit4] 2> 1867039 INFO (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[E3C6A345C2B6F468]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testHealthCheckHandler [junit4] 2> 1867119 INFO (qtp2511437-14066) [n:127.0.0.1:33941_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/health params={wt=javabin&version=2} status=0 QTime=0 [junit4] 2> 1867120 INFO (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[E3C6A345C2B6F468]) [ ] o.a.s.c.MiniSolrCloudCluster Expired zookeeper session 98769930569056259 from node https://127.0.0.1:33941/solr [junit4] 2> 1867123 INFO (zkCallback-3375-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 1867223 WARN (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@aee56c name: ZooKeeperConnection Watcher:127.0.0.1:43673/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1867224 WARN (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1867233 INFO (qtp2511437-14073) [n:127.0.0.1:33941_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/health params={wt=javabin&version=2} status=0 QTime=0 [junit4] 2> 1867234 INFO (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[E3C6A345C2B6F468]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testHealthCheckHandler [junit4] 2> 1867274 INFO (TEST-HealthCheckHandlerTest.testHealthCheckHandlerSolrJ-seed#[E3C6A345C2B6F468]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testHealthCheckHandlerSolrJ [junit4] 2> 1867283 INFO (qtp2511437-14066) [n:127.0.0.1:33941_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/health params={wt=javabin&version=2} status=503 QTime=0 [junit4] 2> 1867284 INFO (TEST-HealthCheckHandlerTest.testHealthCheckHandlerSolrJ-seed#[E3C6A345C2B6F468]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testHealthCheckHandlerSolrJ [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=HealthCheckHandlerTest -Dtests.method=testHealthCheckHandlerSolrJ -Dtests.seed=E3C6A345C2B6F468 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sq-AL -Dtests.timezone=Europe/Astrakhan -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 0.05s J0 | HealthCheckHandlerTest.testHealthCheckHandlerSolrJ <<< [junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:33941/solr: Host Unavailable: Not connected to zk [junit4] > at __randomizedtesting.SeedInfo.seed([E3C6A345C2B6F468:49357C51B950BC69]:0) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242) [junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178) [junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195) [junit4] > at org.apache.solr.cloud.HealthCheckHandlerTest.testHealthCheckHandlerSolrJ(HealthCheckHandlerTest.java:78) [junit4] > at java.lang.Thread.run(Thread.java:748) [junit4] 2> 1867287 INFO (jetty-closer-3364-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@19e8ae6{SSL,[ssl, http/1.1]}{127.0.0.1:0} [junit4] 2> 1867288 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=714100 [junit4] 2> 1867288 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null [junit4] 2> 1867288 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1eef96: rootName = solr_33941, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@957a14 [junit4] 2> 1867293 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null [junit4] 2> 1867293 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1382214: rootName = solr_33941, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@f45eed [junit4] 2> 1867298 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null [junit4] 2> 1867298 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@c4cc55: rootName = solr_33941, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@40328e [junit4] 2> 1868277 WARN (jetty-launcher-3363-thread-1-SendThread(127.0.0.1:43673)) [n:127.0.0.1:33941_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6ba820f0003 has expired [junit4] 2> 1868277 WARN (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@aee56c name: ZooKeeperConnection Watcher:127.0.0.1:43673/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1868277 WARN (OverseerCollectionConfigSetProcessor-98769930569056259-127.0.0.1:33941_solr-n_0000000000) [n:127.0.0.1:33941_solr ] o.a.s.c.OverseerTaskProcessor Overseer cannot talk to ZK [junit4] 2> 1868277 WARN (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1868277 WARN (OverseerStateUpdate-98769930569056259-127.0.0.1:33941_solr-n_0000000000) [n:127.0.0.1:33941_solr ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:313) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:290) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstElement(ZkDistributedQueue.java:406) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:165) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:142) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:185) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 1868277 INFO (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.Overseer Overseer (id=98769930569056259-127.0.0.1:33941_solr-n_0000000000) closing [junit4] 2> 1868277 INFO (OverseerStateUpdate-98769930569056259-127.0.0.1:33941_solr-n_0000000000) [n:127.0.0.1:33941_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33941_solr [junit4] 2> 1868280 WARN (OverseerAutoScalingTriggerThread-98769930569056259-127.0.0.1:33941_solr-n_0000000000) [n:127.0.0.1:33941_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting. [junit4] 2> 1868282 INFO (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores [junit4] 2> 1868283 WARN (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1868284 INFO (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1868284 INFO (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1868377 ERROR (jetty-closer-3364-thread-1) [ ] o.a.s.c.Overseer Could not create Overseer node [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:392) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:389) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:389) [junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:755) [junit4] 2> at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:628) [junit4] 2> at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:615) [junit4] 2> at org.apache.solr.cloud.ZkController.publishNodeAsDown(ZkController.java:2596) [junit4] 2> at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:750) [junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:317) [junit4] 2> at org.eclipse.jetty.servlet.FilterHolder.destroyInstance(FilterHolder.java:178) [junit4] 2> at org.eclipse.jetty.servlet.FilterHolder.doStop(FilterHolder.java:155) [junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:238) [junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:142) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:160) [junit4] 2> at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) [junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:127) [junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:142) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:160) [junit4] 2> at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) [junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:825) [junit4] 2> at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:356) [junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:880) [junit4] 2> at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:272) [junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:142) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:160) [junit4] 2> at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) [junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:142) [junit4] 2> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:160) [junit4] 2> at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) [junit4] 2> at org.eclipse.jetty.server.Server.doStop(Server.java:493) [junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.stop(JettySolrRunner.java:428) [junit4] 2> at org.apache.solr.cloud.MiniSolrCloudCluster.stopJettySolrRunner(MiniSolrCloudCluster.java:423) [junit4] 2> at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$shutdown$1(MiniSolrCloudCluster.java:460) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 1868378 WARN (jetty-closer-3364-thread-1) [ ] o.a.s.c.ZkController Could not publish node as down: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer [junit4] 2> 1868378 INFO (jetty-closer-3364-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null [junit4] 2> 1868722 WARN (zkCallback-3370-thread-1-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 1868723 ERROR (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:716) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:433) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> [junit4] 2> 1868723 ERROR (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:716) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:433) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322) [junit4] 2> ... 10 more [junit4] 2> [junit4] 2> 1868723 WARN (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 1868725 INFO (zkCallback-3370-thread-2-processing-n:127.0.0.1:33941_solr) [n:127.0.0.1:33941_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> 1868725 INFO (jetty-closer-3364-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@14a58a9{/solr,null,UNAVAILABLE} [junit4] 2> 1868725 ERROR (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> 1868726 INFO (SUITE-HealthCheckHandlerTest-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43673 43673 [junit4] 2> 1870001 INFO (Thread-2064) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43673 43673 [junit4] 2> 1870002 WARN (Thread-2064) [ ] o.a.s.c.ZkTestServer Watch limit violations: [junit4] 2> Maximum concurrent create/delete watches above limit: [junit4] 2> [junit4] 2> 2 /solr/aliases.json [junit4] 2> [junit4] 2> Maximum concurrent data watches above limit: [junit4] 2> [junit4] 2> 3 /solr/clusterprops.json [junit4] 2> 2 /solr/clusterstate.json [junit4] 2> [junit4] 2> Maximum concurrent children watches above limit: [junit4] 2> [junit4] 2> 2 /solr/live_nodes [junit4] 2> 2 /solr/collections [junit4] 2> [junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HealthCheckHandlerTest_E3C6A345C2B6F468-001 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1458, maxMBSortInHeap=5.935118690107852, sim=RandomSimilarity(queryNorm=false): {}, locale=sq-AL, timezone=Europe/Astrakhan [junit4] 2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=93627736,total=401948672 [junit4] 2> NOTE: All tests run in this JVM: [TestDynamicFieldResource, WordBreakSolrSpellCheckerTest, LeaderInitiatedRecoveryOnShardRestartTest, TestBulkSchemaAPI, V2StandaloneTest, MergeStrategyTest, SystemLogListenerTest, TestPKIAuthenticationPlugin, TestOverriddenPrefixQueryForCustomFieldType, TestOmitPositions, TestFilteredDocIdSet, AutoAddReplicasIntegrationTest, TestCloudRecovery, ExitableDirectoryReaderTest, TestSizeLimitedDistributedMap, CollectionTooManyReplicasTest, CursorPagingTest, DistributedSpellCheckComponentTest, TestCSVLoader, TestGeoJSONResponseWriter, TestIntervalFaceting, TestFieldTypeCollectionResource, TestDocumentBuilder, TestComponentsName, TestJsonFacets, OverriddenZkACLAndCredentialsProvidersTest, TestReloadAndDeleteDocs, TestXmlQParser, DistribDocExpirationUpdateProcessorTest, SpellCheckCollatorTest, ManagedSchemaRoundRobinCloudTest, TestUseDocValuesAsStored, TestDefaultStatsCache, TestDocTermOrds, DistributedQueryComponentOptimizationTest, TestSearcherReuse, IndexBasedSpellCheckerTest, SegmentsInfoRequestHandlerTest, TestCollectionAPIs, ParsingFieldUpdateProcessorsTest, HighlighterMaxOffsetTest, SolrPluginUtilsTest, TestIBSimilarityFactory, RequestHandlersTest, TestSolrConfigHandlerConcurrent, TestReplicationHandlerBackup, TriLevelCompositeIdRoutingTest, TestPhraseSuggestions, TestSolrCloudWithKerberosAlt, TestDeleteCollectionOnDownNodes, TestRetrieveFieldsOptimizer, TestNumericTerms64, ExternalFileFieldSortTest, TestFunctionQuery, BufferStoreTest, BooleanFieldTest, FieldAnalysisRequestHandlerTest, AddSchemaFieldsUpdateProcessorFactoryTest, DocumentAnalysisRequestHandlerTest, TestCSVResponseWriter, TestSolrQueryParser, BasicDistributedZkTest, TestDistributedStatsComponentCardinality, TestReRankQParserPlugin, TestCloudSchemaless, TestManagedSchema, TestBlendedInfixSuggestions, TestExclusionRuleCollectionAccess, RAMDirectoryFactoryTest, AssignTest, TestInitQParser, SolrIndexSplitterTest, TestInfoStreamLogging, SuggesterWFSTTest, TestSSLRandomization, SpatialRPTFieldTypeTest, RollingRestartTest, HdfsChaosMonkeySafeLeaderTest, DebugComponentTest, TestFastOutputStream, TestNonDefinedSimilarityFactory, TestConfigSetProperties, DistributedQueryElevationComponentTest, TestZkAclsWithHadoopAuth, TestManagedResource, ExplicitHLLTest, InfixSuggestersTest, TestFieldCacheWithThreads, TestBadConfig, SolrCoreTest, CdcrBootstrapTest, ConcurrentDeleteAndCreateCollectionTest, TestEmbeddedSolrServerSchemaAPI, TestSQLHandler, TestAddFieldRealTimeGet, BigEndianAscendingWordDeserializerTest, TestTolerantUpdateProcessorCloud, ActionThrottleTest, ShardSplitTest, SolrCLIZkUtilsTest, TestDistributedSearch, HealthCheckHandlerTest] [junit4] Completed [501/745 (1!)] on J0 in 3.54s, 3 tests, 1 error <<< FAILURES! [...truncated 147 lines...] [junit4] Suite: org.apache.solr.cloud.TestLeaderElectionZkExpiry [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestLeaderElectionZkExpiry_E3C6A345C2B6F468-001/init-core-data-001 [junit4] 2> 2210538 WARN (SUITE-TestLeaderElectionZkExpiry-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4 [junit4] 2> 2210539 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false [junit4] 2> 2210540 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN) [junit4] 2> 2210541 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[E3C6A345C2B6F468]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> 2210543 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testLeaderElectionWithZkExpiry [junit4] 2> 2210548 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5f4d5, but no JMX reporters were configured - adding default JMX reporter. [junit4] 2> 2210582 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f4d5 [junit4] 2> 2210591 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f4d5 [junit4] 2> 2210591 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f4d5 [junit4] 2> 2210592 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestLeaderElectionZkExpiry_E3C6A345C2B6F468-001/testLeaderElectionWithZkExpiry-solr-001 [junit4] 2> 2210593 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 2210593 INFO (Thread-5549) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 2210593 INFO (Thread-5549) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 2210595 ERROR (Thread-5549) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> 2210694 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [ ] o.a.s.c.ZkTestServer start zk server on port:35021 [junit4] 2> 2210761 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 2210762 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 2210762 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=98769953126350850-dummy.host.com:8984_solr-n_0000000000) starting [junit4] 2> 2210765 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 2210765 INFO (zkCallback-3561-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 2210766 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x15ee6bfc2940002 due to java.nio.channels.CancelledKeyException [junit4] 2> 2210866 WARN (zkCallback-3561-thread-3-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 2210867 WARN (zkCallback-3561-thread-3-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 2212837 WARN (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]-SendThread(127.0.0.1:35021)) [n:dummy.host.com:8984_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6bfc2940002 has expired [junit4] 2> 2212841 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 2212841 WARN (OverseerStateUpdate-98769953126350850-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:313) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:290) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstElement(ZkDistributedQueue.java:406) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:165) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:142) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:185) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 2212841 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 2212842 INFO (OverseerStateUpdate-98769953126350850-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 2212842 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=98769953126350850-dummy.host.com:8984_solr-n_0000000000) closing [junit4] 2> 2212842 INFO (OverseerCollectionConfigSetProcessor-98769953126350850-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=98769953126350850-dummy.host.com:8984_solr-n_0000000000) am no longer a leader. [junit4] 2> 2212842 ERROR (OverseerExitThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer could not read the data [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:299) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 2212938 WARN (OverseerAutoScalingTriggerThread-98769953126350850-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.a.TriggerBase Exception checking ZK path /autoscaling/triggerState [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/triggerState [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:306) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:303) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:303) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:557) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:415) [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerBase.<init>(TriggerBase.java:47) [junit4] 2> at org.apache.solr.cloud.autoscaling.NodeLostTrigger.<init>(NodeLostTrigger.java:72) [junit4] 2> at org.apache.solr.cloud.autoscaling.AutoScaling$TriggerFactory.create(AutoScaling.java:137) [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.loadTriggers(OverseerTriggerThread.java:365) [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.refreshAutoScalingConf(OverseerTriggerThread.java:312) [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:137) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 2212938 WARN (zkCallback-3561-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 2212940 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 2212941 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 2212941 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 2212942 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 2212944 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 2212944 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=98769953126350851-dummy.host.com:8984_solr-n_0000000001) starting [junit4] 2> 2212947 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 2212947 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper [junit4] 2> 2212947 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> 2212947 INFO (zkCallback-3561-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 2213050 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 2213050 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 2214709 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:35021)) [n:dummy.host.com:8984_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6bfc2940003 has expired [junit4] 2> 2214709 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 2214709 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 2214709 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=98769953126350851-dummy.host.com:8984_solr-n_0000000001) closing [junit4] 2> 2214709 INFO (OverseerStateUpdate-98769953126350851-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 2214714 ERROR (OverseerExitThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer could not read the data [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:299) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 2214809 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> ott 04, 2017 5:38:04 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> AVVERTENZA: Uncaught exception in thread: Thread[OverseerAutoScalingTriggerThread-98769953126350851-dummy.host.com:8984_solr-n_0000000001,5,Overseer autoscaling triggers] [junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events [junit4] 2> at __randomizedtesting.SeedInfo.seed([E3C6A345C2B6F468]:0) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107) [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:30) [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398) [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149) [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:306) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:303) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:303) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:512) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:97) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78) [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105) [junit4] 2> ... 5 more [junit4] 2> [junit4] 2> 2214818 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 2214819 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 2214820 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 2214820 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 2214922 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 2214923 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 2216047 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:35021)) [n:dummy.host.com:8984_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6bfc2940004 has expired [junit4] 2> 2216047 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 2216047 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 2216047 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 2216048 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 2216048 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 2216049 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception [junit4] 2> java.nio.channels.CancelledKeyException [junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) [junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87) [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:182) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 2216149 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 2216150 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 2218040 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:35021)) [n:dummy.host.com:8984_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6bfc2940005 has expired [junit4] 2> 2218041 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 2218041 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 2218041 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 2218043 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 2218043 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 2218141 ERROR (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:716) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:433) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> [junit4] 2> 2218141 ERROR (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController :org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect [junit4] 2> at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:772) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:338) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78) [junit4] 2> at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:770) [junit4] 2> ... 6 more [junit4] 2> [junit4] 2> 2218142 ERROR (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:716) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:433) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322) [junit4] 2> ... 5 more [junit4] 2> [junit4] 2> 2218142 ERROR (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect [junit4] 2> at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:772) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:338) [junit4] 2> ... 5 more [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78) [junit4] 2> at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:770) [junit4] 2> ... 6 more [junit4] 2> [junit4] 2> 2218143 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 2218142 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 2218144 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 2218144 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> 2218144 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 2218144 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> 2219321 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:35021)) [n:dummy.host.com:8984_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6bfc2940006 has expired [junit4] 2> 2219321 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 2219321 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 2219321 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 2219323 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 2219323 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 2219429 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 2219429 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 2220473 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:35021)) [n:dummy.host.com:8984_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6bfc2940007 has expired [junit4] 2> 2220473 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 2220474 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 2220474 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 2220476 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 2220476 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 2220574 ERROR (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> [junit4] 2> 2220575 ERROR (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322) [junit4] 2> ... 5 more [junit4] 2> [junit4] 2> 2220575 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 2220576 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> 2220584 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 2220584 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 2221688 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:35021)) [n:dummy.host.com:8984_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee6bfc2940008 has expired [junit4] 2> 2221688 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c66f6f name: ZooKeeperConnection Watcher:127.0.0.1:35021/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 2221689 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 2221689 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 2221689 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 2221689 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 2222085 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 2222086 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=98769953126350858-dummy.host.com:8984_solr-n_0000000002) starting [junit4] 2> 2222086 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.c.TestLeaderElectionZkExpiry Time=3028741277894901 Overseer leader is = dummy.host.com:8984_solr [junit4] 2> 2222088 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=98769953126350858-dummy.host.com:8984_solr-n_0000000002) closing [junit4] 2> 2222088 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 2222088 INFO (OverseerStateUpdate-98769953126350858-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer According to ZK I (id=98769953126350858-dummy.host.com:8984_solr-n_0000000002) am no longer a leader. [junit4] 2> 2222088 ERROR (OverseerCollectionConfigSetProcessor-98769953126350858-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer [junit4] 2> java.lang.InterruptedException [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.lang.Object.wait(Object.java:502) [junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320) [junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:61) [junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:178) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> 2222088 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper [junit4] 2> 2222088 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> 2222088 INFO (OverseerStateUpdate-98769953126350858-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 2222090 INFO (zkCallback-3561-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 2222091 WARN (OverseerExitThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer I'm exiting, but I'm still the leader [junit4] 2> 2222091 WARN (zkCallback-3561-thread-4-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 2222092 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 2222091 WARN (zkCallback-3561-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 2222092 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=18141919 [junit4] 2> 2222092 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null [junit4] 2> 2222092 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@8232c4: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@18bf72c [junit4] 2> 2222096 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null [junit4] 2> 2222097 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[E3C6A345C2B6F468]) [n:dummy.host.com:8984_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ff188e: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@1cce88 [...truncated too long message...] n.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267) [junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:347) [junit4] 2> ... 5 more [junit4] 2> [junit4] 2> 2223593 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 2223593 ERROR (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337) [junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267) [junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:347) [junit4] 2> ... 5 more [junit4] 2> [junit4] 2> 2223594 WARN (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 2223594 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> 2223595 INFO (zkCallback-3561-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager Connected:true [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=2042, maxMBSortInHeap=6.671730572399635, sim=RandomSimilarity(queryNorm=false): {}, locale=it-IT, timezone=US/Eastern [junit4] 2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=233671096,total=518979584 [junit4] 2> NOTE: All tests run in this JVM: [NotRequiredUniqueKeyTest, TestFieldCacheReopen, OutOfBoxZkACLAndCredentialsProvidersTest, MultiTermTest, DistributedQueryComponentCustomSortTest, TestLFUCache, TriggerIntegrationTest, TestSuggestSpellingConverter, TestReloadDeadlock, TestComplexPhraseLeadingWildcard, AutoScalingHandlerTest, TestConfigSetsAPI, SolrJmxReporterCloudTest, QueryResultKeyTest, TestStressCloudBlindAtomicUpdates, DeleteReplicaTest, MinimalSchemaTest, CheckHdfsIndexTest, ShowFileRequestHandlerTest, BadCopyFieldTest, TestRestManager, NumericFieldsTest, TestSolrCoreProperties, TestPayloadScoreQParserPlugin, TestScoreJoinQPNoScore, OverseerTest, TestHdfsBackupRestoreCore, GraphQueryTest, LeaderElectionContextKeyTest, BlockDirectoryTest, TestHalfAndHalfDocValues, TestFoldingMultitermQuery, TestSolrCloudWithHadoopAuthPlugin, DirectUpdateHandlerOptimizeTest, FastVectorHighlighterTest, TestSurroundQueryParser, TestFreeTextSuggestions, StatsComponentTest, SuggestComponentTest, SimplePostToolTest, TestNumericRangeQuery32, TestSolrCLIRunExample, BasicAuthIntegrationTest, TestConfigSetImmutable, DistributedFacetPivotSmallAdvancedTest, TestCoreAdminApis, UnloadDistributedZkTest, AutoCommitTest, TestTlogReplica, TestLeaderElectionWithEmptyReplica, CurrencyFieldTypeTest, HardAutoCommitTest, TestCloudManagedSchema, CloudMLTQParserTest, DataDrivenBlockJoinTest, RequiredFieldsTest, InfoHandlerTest, TolerantUpdateProcessorTest, CoreAdminCreateDiscoverTest, TestManagedStopFilterFactory, SortByFunctionTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, SolrCoreMetricManagerTest, TestSort, QueryEqualityTest, SortSpecParsingTest, TestNumericTerms32, TestLocalFSCloudBackupRestore, TestCodecSupport, CdcrReplicationHandlerTest, TestRecovery, ClassificationUpdateProcessorTest, TestSolrDeletionPolicy2, TestFieldCacheVsDocValues, TestSolrCloudWithDelegationTokens, TestSchemalessBufferedUpdates, SaslZkACLProviderTest, HdfsNNFailoverTest, TermVectorComponentDistributedTest, TestUtils, TestSha256AuthenticationProvider, TestQuerySenderListener, TestSlowCompositeReaderWrapper, SuggesterTest, EnumFieldTest, HLLSerializationTest, DocumentBuilderTest, TestJmxIntegration, TestCorePropertiesReload, ClusterStateTest, DistribCursorPagingTest, BaseCdcrDistributedZkTest, DistributedFacetExistsSmallTest, TestMaxScoreQueryParser, BitVectorTest, TlogReplayBufferedWhileIndexingTest, BasicDistributedZk2Test, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, SolrMetricsIntegrationTest, TestGroupingSearch, LeaderInitiatedRecoveryOnCommitTest, TestStressReorder, JavabinLoaderTest, TestExportWriter, CollectionStateFormat2Test, TestCloudInspectUtil, TestReload, DateMathParserTest, TestCloudPseudoReturnFields, RecoveryZkTest, TestPullReplicaErrorHandling, TestSchemaVersionResource, SecurityConfHandlerTest, ResponseHeaderTest, TestPseudoReturnFields, TestDFRSimilarityFactory, MissingSegmentRecoveryTest, TestTrie, RegexBoostProcessorTest, ReturnFieldsTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, PingRequestHandlerTest, SearchHandlerTest, HighlighterConfigTest, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, TestMergePolicyConfig, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestCharFilters, EchoParamsTest, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TimeZoneUtilsTest, ScriptEngineTest, URLClassifyProcessorTest, TestFastLRUCache, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, DateFieldTest, SpellingQueryConverterTest, TestSolrJ, TestLRUCache, ZkNodePropsTest, SystemInfoHandlerTest, UUIDFieldTest, CircularListTest, DistributedMLTComponentTest, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestTolerantSearch, TestLuceneIndexBackCompat, TestEmbeddedSolrServerAdminHandler, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ConnectionReuseTest, AssignBackwardCompatibilityTest, CdcrReplicationDistributedZkTest, CdcrVersionReplicationTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CollectionReloadTest, CollectionsAPISolrJTest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DeleteNodeTest, DistributedQueueTest, ForceLeaderTest, LeaderFailureAfterFreshStartTest, MoveReplicaHDFSTest, MultiThreadedOCPTest, PeerSyncReplicationTest, TestClusterProperties, TestCollectionAPI, TestDownShardTolerantSearch, TestLeaderElectionZkExpiry] [junit4] Completed [546/745 (2!)] on J2 in 13.16s, 1 test, 1 error <<< FAILURES! [...truncated 22214 lines...] BUILD FAILED /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:826: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:117: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:393: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:2014: Rat problems were found! Total time: 85 minutes 22 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts [WARNINGS] Skipping publisher since build result is FAILURE Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org