Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11312/ Java: 32bit/jdk1.8.0_40-ea-b09 -client -XX:+UseG1GC
2 tests failed. REGRESSION: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch Error Message: No live SolrServers available to handle this request:[https://127.0.0.1:52857, https://127.0.0.1:34173, https://127.0.0.1:34253] Stack Trace: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:52857, https://127.0.0.1:34173, https://127.0.0.1:34253] at __randomizedtesting.SeedInfo.seed([5E39969D5E5D7335:DFDF188529021309]:0) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333) at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:1015) at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:793) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:736) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not remove replica : customcollreplicadeletion/a/core_node2 at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:354) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:313) ... 46 more REGRESSION: org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch Error Message: No live SolrServers available to handle this request:[https://127.0.0.1:42767, https://127.0.0.1:54724, https://127.0.0.1:51404, https://127.0.0.1:59677, https://127.0.0.1:42110] Stack Trace: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:42767, https://127.0.0.1:54724, https://127.0.0.1:51404, https://127.0.0.1:59677, https://127.0.0.1:42110] at __randomizedtesting.SeedInfo.seed([5E39969D5E5D7335:DFDF188529021309]:0) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333) at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:1015) at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:793) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:736) at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:172) at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:145) at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not remove replica : delLiveColl/shard1/core_node1 at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:354) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:313) ... 46 more Build Log: [...truncated 11057 lines...] [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest [junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/init-core-data-001 [junit4] 2> 284432 T585 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false) [junit4] 2> 284433 T585 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: / [junit4] 2> 284435 T585 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch [junit4] 2> 284436 T585 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 284437 T586 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 284536 T585 oasc.ZkTestServer.run start zk server on port:37422 [junit4] 2> 284537 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 284538 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 284541 T592 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fad4 name:ZooKeeperConnection Watcher:127.0.0.1:37422 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 284541 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 284542 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 284542 T585 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 284546 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 284547 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 284548 T594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@106ebe1 name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 284548 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 284549 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 284549 T585 oascc.SolrZkClient.makePath makePath: /collections/collection1 [junit4] 2> 284552 T585 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards [junit4] 2> 284554 T585 oascc.SolrZkClient.makePath makePath: /collections/control_collection [junit4] 2> 284556 T585 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards [junit4] 2> 284558 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 284559 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml [junit4] 2> 284562 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml [junit4] 2> 284562 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml [junit4] 2> 284565 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 284566 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 284568 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 284569 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt [junit4] 2> 284571 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 284571 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt [junit4] 2> 284574 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4] 2> 284574 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml [junit4] 2> 284577 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 284577 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml [junit4] 2> 284579 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json [junit4] 2> 284580 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json [junit4] 2> 284582 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 284583 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 284585 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 284585 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt [junit4] 2> 284588 T585 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 284588 T585 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt [junit4] 2> 284591 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 284592 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 284594 T596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bc51a name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 284594 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 284595 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 284930 T585 oejs.Server.doStart jetty-8.1.10.v20130312 [junit4] 2> 284952 T585 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] [junit4] 2> 284991 T585 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:34173 [junit4] 2> 284994 T585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() [junit4] 2> 284994 T585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 284995 T585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002 [junit4] 2> 284995 T585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/' [junit4] 2> 285014 T585 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/solr.xml [junit4] 2> 285023 T585 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002 [junit4] 2> 285024 T585 oasc.CoreContainer.<init> New CoreContainer 25912555 [junit4] 2> 285024 T585 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/] [junit4] 2> 285025 T585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 285026 T585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 285026 T585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 285026 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 285026 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 285027 T585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 285027 T585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 285027 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 285028 T585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 285028 T585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 285030 T585 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false [junit4] 2> 285032 T585 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 285032 T585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 285033 T585 oasc.CoreContainer.load Host Name: 127.0.0.1 [junit4] 2> 285033 T585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37422/solr [junit4] 2> 285033 T585 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 285034 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 285035 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 285036 T607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83241c name:ZooKeeperConnection Watcher:127.0.0.1:37422 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 285037 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 285037 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 285045 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 285047 T609 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@951ad6 name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 285047 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 285049 T585 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 285051 T585 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 285053 T585 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 285055 T585 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 285057 T585 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 285060 T585 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 285061 T585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34173_ [junit4] 2> 285061 T585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34173_ [junit4] 2> 285063 T585 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 285064 T585 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 285066 T585 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 285067 T585 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34173_ [junit4] 2> 285067 T585 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 285069 T585 oasc.Overseer.start Overseer (id=92639312997056516-127.0.0.1:34173_-n_0000000000) starting [junit4] 2> 285071 T585 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 285076 T585 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000 [junit4] 2> 285077 T611 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 285077 T585 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 285079 T585 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 285081 T585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 285083 T610 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 285084 T585 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002 [junit4] 2> 285091 T585 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/conf/ [junit4] 2> 285220 T585 oasc.CorePropertiesLocator.discover Found 1 core definitions [junit4] 2> 285221 T585 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1 [junit4] 2> 285222 T585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 285225 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 285226 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 285227 T615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dcb600 name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 285227 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 285227 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 285229 T585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 285232 T585 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false [junit4] 2> 285233 T585 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2 [junit4] 2> 285233 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 285233 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 285235 T617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ae8012 name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 285235 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 285235 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 285238 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 285241 T610 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1 [junit4] 2> 285242 T610 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2] [junit4] 2> 285242 T610 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2 [junit4] 2> 285245 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json [junit4] 2> 285246 T618 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 285247 T619 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 285539 T585 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 [junit4] 2> 285540 T585 oejs.Server.doStart jetty-8.1.10.v20130312 [junit4] 2> 285544 T585 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] [junit4] 2> 285546 T585 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:52857 [junit4] 2> 285555 T585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() [junit4] 2> 285556 T585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 285556 T585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003 [junit4] 2> 285556 T585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/' [junit4] 2> 285573 T585 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/solr.xml [junit4] 2> 285582 T585 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003 [junit4] 2> 285583 T585 oasc.CoreContainer.<init> New CoreContainer 15129105 [junit4] 2> 285583 T585 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/] [junit4] 2> 285584 T585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 285584 T585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 285585 T585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 285585 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 285585 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 285586 T585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 285586 T585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 285586 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 285586 T585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 285587 T585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 285588 T585 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false [junit4] 2> 285590 T585 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 285590 T585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 285590 T585 oasc.CoreContainer.load Host Name: 127.0.0.1 [junit4] 2> 285591 T585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37422/solr [junit4] 2> 285591 T585 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 285591 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 285592 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 285594 T630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dada04 name:ZooKeeperConnection Watcher:127.0.0.1:37422 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 285594 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 285595 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 285597 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 285598 T632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1770dde name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 285598 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 285604 T585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 286609 T585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52857_ [junit4] 2> 286610 T585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52857_ [junit4] 2> 286614 T585 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 286617 T585 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003 [junit4] 2> 286624 T585 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/conf/ [junit4] 2> 286789 T585 oasc.CorePropertiesLocator.discover Found 1 core definitions [junit4] 2> 286790 T585 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1 [junit4] 2> 286790 T585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 287060 T585 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 [junit4] 2> 287061 T585 oejs.Server.doStart jetty-8.1.10.v20130312 [junit4] 2> 287064 T585 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] [junit4] 2> 287066 T585 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:34253 [junit4] 2> 287068 T585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() [junit4] 2> 287068 T585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 287069 T585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-004 [junit4] 2> 287069 T585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-004/' [junit4] 2> 287086 T585 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-004/solr.xml [junit4] 2> 287094 T585 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-004 [junit4] 2> 287095 T585 oasc.CoreContainer.<init> New CoreContainer 12759181 [junit4] 2> 287095 T585 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-004/] [junit4] 2> 287096 T585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 287096 T585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 287097 T585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 287097 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 287097 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 287098 T585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 287098 T585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 287098 T585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 287099 T585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 287099 T585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 287101 T585 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false [junit4] 2> 287102 T585 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 287102 T585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 287102 T585 oasc.CoreContainer.load Host Name: 127.0.0.1 [junit4] 2> 287103 T585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37422/solr [junit4] 2> 287103 T585 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 287104 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 287104 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 287106 T645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dc2a7e name:ZooKeeperConnection Watcher:127.0.0.1:37422 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 287106 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 287107 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 287111 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 287112 T647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bf02e name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 287112 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 287118 T585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 288123 T585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34253_ [junit4] 2> 288124 T585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34253_ [junit4] 2> 288128 T585 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 288131 T585 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-004 [junit4] 2> 288137 T585 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-004/conf/ [junit4] 2> 288262 T585 oasc.CorePropertiesLocator.discover Found 1 core definitions [junit4] 2> 288263 T585 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1 [junit4] 2> 288263 T585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 288267 T585 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion [junit4] 2> 288267 T585 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 288268 T585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 288269 T651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d0e48 name:ZooKeeperConnection Watcher:127.0.0.1:37422/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 288270 T585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 288270 T585 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 288272 T585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 288348 T628 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2 [junit4] 2> 288349 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged [junit4] 2> 288351 T611 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{ [junit4] 2> "operation":"create", [junit4] 2> "fromApi":"true", [junit4] 2> "name":"customcollreplicadeletion", [junit4] 2> "replicationFactor":"1", [junit4] 2> "numShards":"1", [junit4] 2> "maxShardsPerNode":"5", [junit4] 2> "shards":"a,b", [junit4] 2> "stateFormat":"2", [junit4] 2> "router.name":"implicit"} [junit4] 2> 288352 T652 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , { [junit4] 2> "operation":"create", [junit4] 2> "fromApi":"true", [junit4] 2> "name":"customcollreplicadeletion", [junit4] 2> "replicationFactor":"1", [junit4] 2> "numShards":"1", [junit4] 2> "maxShardsPerNode":"5", [junit4] 2> "shards":"a,b", [junit4] 2> "stateFormat":"2", [junit4] 2> "router.name":"implicit"} [junit4] 2> 288353 T652 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1 [junit4] 2> 288353 T652 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion [junit4] 2> 288354 T652 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion [junit4] 2> 288357 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 288359 T610 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion [junit4] 2> 288359 T610 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b] [junit4] 2> 288360 T610 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2 [junit4] 2> 288361 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 288362 T619 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 288363 T653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 288362 T633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 288363 T618 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 288363 T648 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 288457 T652 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1 [junit4] 2> 288457 T652 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:34173_ [junit4] 2> 288459 T652 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:52857_ [junit4] 2> 288583 T625 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2 [junit4] 2> 288583 T603 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2 [junit4] 2> 288586 T625 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion [junit4] 2> 288587 T603 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion [junit4] 2> 288588 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 288589 T625 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion [junit4] 2> 288589 T625 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion [junit4] 2> 288590 T603 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion [junit4] 2> 288591 T610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={ [junit4] 2> "core":"customcollreplicadeletion_b_replica1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"https://127.0.0.1:52857";, [junit4] 2> "node_name":"127.0.0.1:52857_", [junit4] 2> "numShards":"2", [junit4] 2> "state":"down", [junit4] 2> "shard":"b", [junit4] 2> "collection":"customcollreplicadeletion", [junit4] 2> "operation":"state"} [junit4] 2> 288592 T625 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... [junit4] 2> 288593 T625 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 [junit4] 2> 288593 T625 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 288597 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 288598 T633 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 288598 T633 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 [junit4] 2> 288599 T603 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion [junit4] 2> 288600 T603 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... [junit4] 2> 288601 T603 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 [junit4] 2> 288602 T603 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 288604 T610 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 [junit4] 2> 288604 T610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={ [junit4] 2> "core":"customcollreplicadeletion_a_replica1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"https://127.0.0.1:34173";, [junit4] 2> "node_name":"127.0.0.1:34173_", [junit4] 2> "numShards":"2", [junit4] 2> "state":"down", [junit4] 2> "shard":"a", [junit4] 2> "collection":"customcollreplicadeletion", [junit4] 2> "operation":"state"} [junit4] 2> 288606 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 288606 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 288607 T618 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 288607 T633 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 288608 T633 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 [junit4] 2> 288609 T618 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 [junit4] 2> 289594 T625 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1 [junit4] 2> 289594 T625 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion [junit4] 2> 289595 T625 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 289595 T625 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion [junit4] 2> 289596 T625 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper [junit4] 2> 289596 T625 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/customcollreplicadeletion_b_replica1/' [junit4] 2> 289602 T603 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1 [junit4] 2> 289603 T603 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion [junit4] 2> 289603 T603 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 289604 T603 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion [junit4] 2> 289605 T603 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper [junit4] 2> 289605 T603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/customcollreplicadeletion_a_replica1/' [junit4] 2> 289645 T625 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0 [junit4] 2> 289653 T603 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0 [junit4] 2> 289665 T625 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 289667 T625 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml [junit4] 2> 289672 T603 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 289673 T603 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml [junit4] 2> 289681 T625 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test [junit4] 2> 289687 T603 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test [junit4] 2> 289856 T603 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440. [junit4] 2> 289858 T625 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440. [junit4] 2> 289862 T603 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 289864 T603 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 289864 T625 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 289867 T625 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 289876 T603 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 289878 T625 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 289879 T603 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 289880 T625 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 289882 T603 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json [junit4] 2> 289883 T603 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 289884 T603 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 289883 T625 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json [junit4] 2> 289884 T603 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json [junit4] 2> 289885 T625 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 289886 T625 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 289886 T603 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 289886 T603 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 289886 T625 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json [junit4] 2> 289887 T603 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion [junit4] 2> 289887 T603 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory [junit4] 2> 289887 T625 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 289888 T625 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 289888 T603 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/customcollreplicadeletion_a_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/customcollreplicadeletion_a_replica1/data/ [junit4] 2> 289888 T625 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion [junit4] 2> 289889 T603 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6caf66 [junit4] 2> 289889 T625 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory [junit4] 2> 289890 T625 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/customcollreplicadeletion_b_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/customcollreplicadeletion_b_replica1/data/ [junit4] 2> 289890 T603 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/customcollreplicadeletion_a_replica1/data [junit4] 2> 289890 T625 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6caf66 [junit4] 2> 289891 T603 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index/ [junit4] 2> 289891 T603 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index' doesn't exist. Creating new index... [junit4] 2> 289891 T625 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/customcollreplicadeletion_b_replica1/data [junit4] 2> 289892 T603 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index [junit4] 2> 289892 T625 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index/ [junit4] 2> 289893 T603 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.734375, noCFSRatio=0.6544615927991775] [junit4] 2> 289893 T625 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_b_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index' doesn't exist. Creating new index... [junit4] 2> 289894 T625 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5E39969D5E5D7335-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index [junit4] 2> 289895 T625 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.734375, noCFSRatio=0.6544615927991775] [junit4] 2> 289895 T603 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-82c20a)),segFN=segments_1,generation=1} [junit4] 2> 289895 T625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-bb3d28)),segFN=segments_1,generation=1} [junit4] 2> 289896 T625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 289895 T603 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 289901 T625 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib" [junit4] 2> 289902 T625 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe" [junit4] 2> 289902 T603 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib" [junit4] 2> 289902 T625 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 289903 T603 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe" [junit4] 2> 289903 T625 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig" [junit4] 2> 289904 T603 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 289904 T625 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 289905 T603 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig" [junit4] 2> 289905 T625 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 289906 T603 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 289907 T625 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 289907 T603 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 289907 T625 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 289908 T603 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 289908 T625 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 289908 T603 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 289909 T603 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 289909 T625 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 289910 T603 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 289910 T625 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 289910 T603 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 289911 T625 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 289911 T603 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 289912 T625 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler [junit4] 2> 289912 T603 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler [junit4] 2> 289913 T625 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler [junit4] 2> 289913 T603 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler [junit4] 2> 289914 T603 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler [junit4] 2> 289913 T625 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler [junit4] 2> 289914 T603 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler [junit4] 2> 289914 T625 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler [junit4] 2> 289915 T603 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler [junit4] 2> 289915 T625 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler [junit4] 2> 289915 T603 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4] 2> 289916 T625 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4] 2> 289916 T603 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler [junit4] 2> 289917 T625 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler [junit4] 2> 289917 T603 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler [junit4] 2> 289918 T625 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler [junit4] 2> 289920 T603 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289920 T625 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289921 T603 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289921 T625 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289923 T603 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289923 T625 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289924 T603 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289925 T625 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 289926 T603 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4] 2> 289927 T603 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4] 2> 289927 T625 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4] 2> 289927 T625 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4] 2> 289927 T603 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=49.18359375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=28.956381379206576, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.420001690669511 [junit4] 2> 289928 T625 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=49.18359375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=28.956381379206576, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.420001690669511 [junit4] 2> 289929 T603 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-82c20a)),segFN=segments_1,generation=1} [junit4] 2> 289929 T625 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-bb3d28)),segFN=segments_1,generation=1} [junit4] 2> 289929 T625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 289929 T603 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 289930 T625 oass.SolrIndexSearcher.<init> Opening Searcher@1353e97[customcollreplicadeletion_b_replica1] main [junit4] 2> 289930 T603 oass.SolrIndexSearcher.<init> Opening Searcher@13c05df[customcollreplicadeletion_a_replica1] main [junit4] 2> 289931 T625 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion [junit4] 2> 289931 T603 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion [junit4] 2> 289932 T625 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper [junit4] 2> 289932 T625 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 289932 T603 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper [junit4] 2> 289933 T625 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 289933 T625 oasr.RestManager.init Initializing RestManager with initArgs: {} [junit4] 2> 289933 T603 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 289933 T625 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 289934 T603 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 289934 T625 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json [junit4] 2> 289934 T603 oasr.RestManager.init Initializing RestManager with initArgs: {} [junit4] 2> 289935 T625 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 289935 T603 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 289936 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json [junit4] 2> 289936 T603 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json [junit4] 2> 289936 T603 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 289937 T625 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json [junit4] 2> 289938 T625 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 289938 T603 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to existing znode /configs/conf1/_rest_managed.json [junit4] 2> 289938 T625 oasr.RestManager.init Initializing 0 registered ManagedResources [junit4] 2> 289938 T603 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 289939 T603 oasr.RestManager.init Initializing 0 registered ManagedResources [junit4] 2> 289942 T657 oasc.SolrCore.registerSearcher [customcollreplicadeletion_b_replica1] Registered new searcher Searcher@1353e97[customcollreplicadeletion_b_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 289943 T656 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@13c05df[customcollreplicadeletion_a_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 289946 T603 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1 [junit4] 2> 289946 T603 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:https://127.0.0.1:34173 collection:customcollreplicadeletion shard:a [junit4] 2> 289946 T625 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_b_replica1 [junit4] 2> 289947 T625 oasc.ZkController.register Register replica - core:customcollreplicadeletion_b_replica1 address:https://127.0.0.1:52857 collection:customcollreplicadeletion shard:b [junit4] 2> 289947 T603 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election [junit4] 2> 289948 T625 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/b/election [junit4] 2> 289951 T625 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard b [junit4] 2> 289952 T603 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a [junit4] 2> 289953 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 289953 T625 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue. [junit4] 2> 289953 T625 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync [junit4] 2> ASYNC NEW_CORE C145 name=customcollreplicadeletion_b_replica1 org.apache.solr.core.SolrCore@5e3923 url=https://127.0.0.1:52857/customcollreplicadeletion_b_replica1 node=127.0.0.1:52857_ C145_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_b_replica1 props:{core=customcollreplicadeletion_b_replica1, base_url=https://127.0.0.1:52857, node_name=127.0.0.1:52857_, state=down} [junit4] 2> 289954 T625 C145 P52857 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:52857/customcollreplicadeletion_b_replica1/ [junit4] 2> 289954 T603 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue. [junit4] 2> 289955 T603 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync [junit4] 2> 289954 T625 C145 P52857 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me [junit4] 2> ASYNC NEW_CORE C146 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@1bc9743 url=https://127.0.0.1:34173/customcollreplicadeletion_a_replica1 node=127.0.0.1:34173_ C146_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=https://127.0.0.1:34173, node_name=127.0.0.1:34173_, state=down} [junit4] 2> 289955 T603 C146 P34173 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:34173/customcollreplicadeletion_a_replica1/ [junit4] 2> 289955 T625 C145 P52857 oasc.SyncStrategy.syncToMe https://127.0.0.1:52857/customcollreplicadeletion_b_replica1/ has no replicas [junit4] 2> 289956 T603 C146 P34173 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me [junit4] 2> 289956 T625 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:52857/customcollreplicadeletion_b_replica1/ b [junit4] 2> 289957 T610 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 [junit4] 2> 289956 T603 C146 P34173 oasc.SyncStrategy.syncToMe https://127.0.0.1:34173/customcollreplicadeletion_a_replica1/ has no replicas [junit4] 2> 289957 T625 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/b [junit4] 2> 289957 T603 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:34173/customcollreplicadeletion_a_replica1/ a [junit4] 2> 289958 T603 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a [junit4] 2> 289959 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 289960 T633 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289960 T618 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289961 T633 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 [junit4] 2> 289963 T618 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 [junit4] 2> 289963 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 289966 T610 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 [junit4] 2> 289968 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 289969 T618 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289969 T633 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289970 T633 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 [junit4] 2> 289971 T618 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 [junit4] 2> 289974 T610 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 [junit4] 2> 289975 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 289976 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 289976 T633 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289976 T618 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289977 T633 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 [junit4] 2> 289978 T618 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 [junit4] 2> 289981 T610 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 [junit4] 2> 289983 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 289983 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 289984 T618 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289984 T633 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 289985 T633 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 [junit4] 2> 289985 T618 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 [junit4] 2> 290012 T625 oasc.ZkController.register We are https://127.0.0.1:52857/customcollreplicadeletion_b_replica1/ and leader is https://127.0.0.1:52857/customcollreplicadeletion_b_replica1/ [junit4] 2> 290012 T625 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_b_replica1 baseURL=https://127.0.0.1:52857 [junit4] 2> 290013 T625 oasc.ZkController.checkRecovery I am the leader, no recovery necessary [junit4] 2> 290013 T625 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=active collection=customcollreplicadeletion [junit4] 2> 290014 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 290014 T603 oasc.ZkController.register We are https://127.0.0.1:34173/customcollreplicadeletion_a_replica1/ and leader is https://127.0.0.1:34173/customcollreplicadeletion_a_replica1/ [junit4] 2> 290015 T603 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=https://127.0.0.1:34173 [junit4] 2> 290015 T603 oasc.ZkController.checkRecovery I am the leader, no recovery necessary [junit4] 2> 290015 T603 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion [junit4] 2> 290020 T625 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 [junit4] 2> 290021 T625 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2} status=0 QTime=1438 [junit4] 2> 290022 T610 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 [junit4] 2> 290022 T610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={ [junit4] 2> "core":"customcollreplicadeletion_b_replica1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"https://127.0.0.1:52857";, [junit4] 2> "node_name":"127.0.0.1:52857_", [junit4] 2> "numShards":"2", [junit4] 2> "state":"active", [junit4] 2> "shard":"b", [junit4] 2> "collection":"customcollreplicadeletion", [junit4] 2> "operation":"state"} [junit4] 2> 290025 T610 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json [junit4] 2> 290026 T618 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 290026 T633 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... [junit4] 2> 290027 T633 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 [junit4] 2> 290027 T618 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 [junit4] 2> 290030 T603 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 [junit4] 2> 290031 T603 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=1448 [junit4] 2> 290032 T652 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: customcollreplicadeletion [junit4] 2> 290034 T652 oasc.OverseerCollectionProcessor$Runner.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=1438},core=customcollreplicadeletion_b_replica1},null={responseHeader={status=0,QTime=1448},core=customcollreplicadeletion_a_replica1}}} [junit4] 2> 290035 T633 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged [junit4] 2> 290036 T610 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 [junit4] 2> 290036 T610 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={ [junit4] 2> "core":"customcollreplicadeletion_a_replica1", [junit4] 2> "core_node_name":"core_node2", [junit4] 2> "roles":null, [junit4] 2> "base_url":"https://127.0.0.1:34173";, [junit4] 2> "node_name":"127.0.0.1:34173_", [junit4] 2> "numShards":"2", [junit4] 2> "state":"active", [junit4] 2> "shard":"a", [junit4] 2> "collection":"customcollreplicadeletion", [junit4] 2> "operation":"state"} [junit4] 2> 290036 T618 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged [junit4] 2> 290039 T628 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2} status=0 QTime=1691 [junit4] 2> 290039 T618 oasc.DistributedQueue [...truncated too long message...] 032352782-127.0.0.1:42110_-n_0000000004) starting [junit4] 2> 2414283 T7168 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000 [junit4] 2> 2414283 T7170 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 2414284 T7169 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 2414298 T7017 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null} [junit4] 2> 2419287 T7017 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 42110 [junit4] 2> 2419288 T7017 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=32972360 [junit4] 2> 2419288 T7017 oasc.Overseer.close Overseer (id=92639439032352782-127.0.0.1:42110_-n_0000000004) closing [junit4] 2> 2419289 T7169 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:42110_ [junit4] 2> 2420791 T7168 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 2420815 T7017 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null} [junit4] 2> 2425804 T7017 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch [junit4] 2> 2425806 T7017 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43361 43361 [junit4] 2> 2431222 T7018 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43361 43361 [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=DeleteReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=5E39969D5E5D7335 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el_CY -Dtests.timezone=America/Rio_Branco -Dtests.file.encoding=UTF-8 [junit4] ERROR 224s J0 | DeleteReplicaTest.testDistribSearch <<< [junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:42767, https://127.0.0.1:54724, https://127.0.0.1:51404, https://127.0.0.1:59677, https://127.0.0.1:42110] [junit4] > at __randomizedtesting.SeedInfo.seed([5E39969D5E5D7335:DFDF188529021309]:0) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:1015) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:793) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:736) [junit4] > at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:172) [junit4] > at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:145) [junit4] > at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not remove replica : delLiveColl/shard1/core_node1 [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:354) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:313) [junit4] > ... 46 more [junit4] 2> 2431249 T7017 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-5E39969D5E5D7335-001 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=el_CY, timezone=America/Rio_Branco [junit4] 2> NOTE: Linux 3.13.0-37-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=163910376,total=293601280 [junit4] 2> NOTE: All tests run in this JVM: [TestExpandComponent, EchoParamsTest, CustomCollectionTest, IndexBasedSpellCheckerTest, TestReplicationHandlerBackup, AliasIntegrationTest, TestBinaryResponseWriter, ReplicationFactorTest, TestFuzzyAnalyzedSuggestions, XmlUpdateRequestHandlerTest, TestWordDelimiterFilterFactory, SuggestComponentTest, FileUtilsTest, CoreAdminHandlerTest, TestDistribDocBasedVersion, ExitableDirectoryReaderTest, DocumentBuilderTest, OverseerTest, CoreAdminCreateDiscoverTest, PathHierarchyTokenizerFactoryTest, TestFieldTypeCollectionResource, StatelessScriptUpdateProcessorFactoryTest, ZkControllerTest, TestDefaultSearchFieldResource, TestComponentsName, TestLMJelinekMercerSimilarityFactory, TestBulkSchemaAPI, AnalyticsMergeStrategyTest, MinimalSchemaTest, SolrTestCaseJ4Test, TestChildDocTransformer, TestSolr4Spatial2, TestAddFieldRealTimeGet, StressHdfsTest, TestSolrCoreProperties, TestAnalyzeInfixSuggestions, ResponseLogComponentTest, LeaderElectionTest, TestSolrDeletionPolicy2, TestManagedSchemaDynamicFieldResource, OpenExchangeRatesOrgProviderTest, CoreAdminRequestStatusTest, TestCollectionAPI, TestAnalyzedSuggestions, FastVectorHighlighterTest, TestSuggestSpellingConverter, TestRTGBase, HighlighterConfigTest, TestRandomDVFaceting, SuggesterTSTTest, DisMaxRequestHandlerTest, HighlighterTest, BadComponentTest, TestFreeTextSuggestions, ShardRoutingTest, TestModifyConfFiles, TestDocSet, SharedFSAutoReplicaFailoverTest, RecoveryZkTest, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerTest, BJQParserTest, PluginInfoTest, SuggesterTest, TestDocumentBuilder, TestCloudManagedSchema, VMParamsZkACLAndCredentialsProvidersTest, TestSolrXml, TestConfig, AtomicUpdatesTest, NotRequiredUniqueKeyTest, WordBreakSolrSpellCheckerTest, TestManagedSchema, TestCursorMarkWithoutUniqueKey, TestInitQParser, SolrCoreCheckLockOnStartupTest, SpellPossibilityIteratorTest, TestFastOutputStream, TestReload, TestSolrJ, TestRemoteStreaming, DocValuesMultiTest, TestCloudSchemaless, SolrCmdDistributorTest, TestStressRecovery, PingRequestHandlerTest, BasicDistributedZkTest, BadIndexSchemaTest, TestPivotHelperCode, HdfsLockFactoryTest, DateRangeFieldTest, IndexSchemaTest, AssignTest, BasicFunctionalityTest, ClusterStateUpdateTest, TestSortingResponseWriter, ReturnFieldsTest, TestSchemaNameResource, DeleteShardTest, BlockDirectoryTest, DistributedIntervalFacetingTest, TestConfigSets, TestHighlightDedupGrouping, TestSystemIdResolver, UnloadDistributedZkTest, SyncSliceTest, FullSolrCloudDistribCmdsTest, ZkCLITest, TestDistributedSearch, AutoCommitTest, TestDistributedGrouping, TestFaceting, TestRecovery, TestHashPartitioner, DistributedSpellCheckComponentTest, TestStressVersions, TestRangeQuery, SpellCheckComponentTest, PeerSyncTest, TestLazyCores, TestBadConfig, OverseerCollectionProcessorTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, TestAtomicUpdateErrorCases, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, SortByFunctionTest, TestSolrDeletionPolicy1, DebugComponentTest, TestSurroundQueryParser, LukeRequestHandlerTest, TestQueryUtils, StandardRequestHandlerTest, TestWriterPerf, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, TermsComponentTest, TestValueSourceCache, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, TestCollationField, MBeansHandlerTest, QueryParsingTest, TestPartialUpdateDeduplication, CoreContainerCoreInitFailuresTest, TestQuerySenderListener, TestSolrIndexConfig, SolrIndexConfigTest, TestSearchPerf, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, TestLuceneMatchVersion, TestCodecSupport, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, TestTolerantSearch, CloudExitableDirectoryReaderTest, CollectionsAPIAsyncDistributedZkTest, DeleteReplicaTest] [junit4] Completed on J0 in 223.62s, 1 test, 1 error <<< FAILURES! [...truncated 299 lines...] BUILD FAILED /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:524: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:472: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build.xml:189: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1360: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:962: There were test failures: 437 suites, 1795 tests, 2 errors, 44 ignored (21 assumptions) Total time: 118 minutes 42 seconds Build step 'Invoke Ant' marked build as failure [description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b09 -client -XX:+UseG1GC Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
