Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4597/ Java: 64bit/jdk1.8.0_40 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed. FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test Error Message: There were too many update fails (30 > 20) - we expect it can happen, but shouldn't easily Stack Trace: java.lang.AssertionError: There were too many update fails (30 > 20) - we expect it can happen, but shouldn't easily at __randomizedtesting.SeedInfo.seed([D021BCBB66EB4B4C:58758361C81726B4]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertFalse(Assert.java:68) at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:230) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 10161 lines...] [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest [junit4] 2> 1622859 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! [junit4] 2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\init-core-data-001 [junit4] 2> 1622872 T10152 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_h/ [junit4] 2> 1622885 T10152 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1622887 T10153 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 1623002 T10152 oasc.ZkTestServer.run start zk server on port:57899 [junit4] 2> 1623002 T10152 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 1623003 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1623008 T10160 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@717757f0 name:ZooKeeperConnection Watcher:127.0.0.1:57899 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1623008 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1623008 T10152 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 1623008 T10152 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 1623012 T10152 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 1623013 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1623014 T10163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9c3dd0c name:ZooKeeperConnection Watcher:127.0.0.1:57899/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1623015 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1623015 T10152 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 1623015 T10152 oascc.SolrZkClient.makePath makePath: /collections/collection1 [junit4] 2> 1623018 T10152 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards [junit4] 2> 1623020 T10152 oascc.SolrZkClient.makePath makePath: /collections/control_collection [junit4] 2> 1623022 T10152 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards [junit4] 2> 1623023 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 1623024 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml [junit4] 2> 1623028 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema15.xml to /configs/conf1/schema.xml [junit4] 2> 1623028 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml [junit4] 2> 1623030 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 1623031 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 1623033 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 1623033 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt [junit4] 2> 1623036 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 1623036 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt [junit4] 2> 1623039 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml [junit4] 2> 1623039 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml [junit4] 2> 1623041 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 1623041 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml [junit4] 2> 1623043 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json [junit4] 2> 1623043 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json [junit4] 2> 1623046 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 1623047 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 1623049 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 1623049 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt [junit4] 2> 1623051 T10152 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 1623052 T10152 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt [junit4] 2> 1623341 T10152 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1 [junit4] 2> 1623343 T10152 oejs.Server.doStart jetty-9.2.9.v20150224 [junit4] 2> 1623350 T10152 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4e5294a5{/_h,null,AVAILABLE} [junit4] 2> 1623350 T10152 oejs.AbstractConnector.doStart Started ServerConnector@2d669af6{HTTP/1.1}{127.0.0.1:57906} [junit4] 2> 1623351 T10152 oejs.Server.doStart Started @1633944ms [junit4] 2> 1623351 T10152 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\tempDir-001/control/data, hostContext=/_h, hostPort=57906, coreRootDirectory=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores} [junit4] 2> 1623351 T10152 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a [junit4] 2> 1623351 T10152 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\' [junit4] 2> 1623380 T10152 oasc.SolrXmlConfig.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\solr.xml [junit4] 2> 1623390 T10152 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores [junit4] 2> 1623390 T10152 oasc.CoreContainer.<init> New CoreContainer 636548459 [junit4] 2> 1623390 T10152 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\] [junit4] 2> 1623390 T10152 oasc.CoreContainer.load loading shared library: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\lib [junit4] 2> 1623391 T10152 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\lib). [junit4] 2> 1623399 T10152 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 1623399 T10152 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 1623399 T10152 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 1623399 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 1623399 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 1623399 T10152 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 1623400 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 1623400 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 1623400 T10152 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 1623400 T10152 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 1623400 T10152 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false [junit4] 2> 1623400 T10152 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 1623401 T10152 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 1623401 T10152 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 1623401 T10152 oasc.CoreContainer.load Node Name: 127.0.0.1 [junit4] 2> 1623401 T10152 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57899/solr [junit4] 2> 1623401 T10152 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 1623402 T10152 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 1623405 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1623406 T10177 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c3ed555 name:ZooKeeperConnection Watcher:127.0.0.1:57899 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1623407 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1623407 T10152 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 1623410 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1623411 T10180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bcf4e8a name:ZooKeeperConnection Watcher:127.0.0.1:57899/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1623411 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1623412 T10152 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 1623416 T10152 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 1623418 T10152 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 1623420 T10152 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 1623424 T10152 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 1623426 T10152 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 1623428 T10152 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 1623430 T10152 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 1623432 T10152 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57906__h [junit4] 2> 1623432 T10152 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57906__h [junit4] 2> 1623435 T10152 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 1623437 T10152 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 1623438 T10152 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 1623440 T10152 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57906__h [junit4] 2> 1623440 T10152 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 1623443 T10152 oasc.Overseer.start Overseer (id=93544537104384003-127.0.0.1:57906__h-n_0000000000) starting [junit4] 2> 1623445 T10152 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 1623453 T10152 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000 [junit4] 2> 1623454 T10152 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1623454 T10182 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 1623455 T10181 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 1623460 T10152 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores [junit4] 2> 1623462 T10152 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1, collection=control_collection, absoluteInstDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\, coreNodeName=, dataDir=data\, shard=} [junit4] 2> 1623462 T10152 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\ [junit4] 2> 1623462 T10152 oasc.CorePropertiesLocator.discover Found 1 core definitions [junit4] 2> 1623464 T10184 C:control_collection c:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection [junit4] 2> 1623464 T10184 C:control_collection c:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 1623464 T10184 C:control_collection c:collection1 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 1623464 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1623465 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57906/_h", [junit4] 2> "node_name":"127.0.0.1:57906__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} current state version: 0 [junit4] 2> 1623466 T10181 oasco.ReplicaMutator.updateState Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57906/_h", [junit4] 2> "node_name":"127.0.0.1:57906__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} [junit4] 2> 1623466 T10181 oasco.ClusterStateMutator.createCollection building a new cName: control_collection [junit4] 2> 1623466 T10181 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1 [junit4] 2> 1623467 T10180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1624610 T10184 C:control_collection c:collection1 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1 [junit4] 2> 1624610 T10184 C:control_collection c:collection1 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection [junit4] 2> 1624611 T10184 C:control_collection c:collection1 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 1624611 T10184 C:control_collection c:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection [junit4] 2> 1624612 T10184 C:control_collection c:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper [junit4] 2> 1624612 T10184 C:control_collection c:collection1 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\' [junit4] 2> 1624629 T10184 C:control_collection c:collection1 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1 [junit4] 2> 1624631 T10184 C:control_collection c:collection1 oasc.Config.<init> loaded config solrconfig.xml with version 0 [junit4] 2> 1624638 T10184 C:control_collection c:collection1 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1 [junit4] 2> 1624640 T10184 C:control_collection c:collection1 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0 [junit4] 2> 1624655 T10184 C:control_collection c:collection1 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 1624658 T10184 C:control_collection c:collection1 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml [junit4] 2> 1624661 T10184 C:control_collection c:collection1 oass.IndexSchema.readSchema [collection1] Schema name=test [junit4] 2> 1624738 T10184 C:control_collection c:collection1 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 1624739 T10184 C:control_collection c:collection1 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 1624739 T10184 C:control_collection c:collection1 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 1624743 T10184 C:control_collection c:collection1 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 1624763 T10184 C:control_collection c:collection1 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection [junit4] 2> 1624763 T10184 C:control_collection c:collection1 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory [junit4] 2> 1624764 T10184 C:control_collection c:collection1 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\data\ [junit4] 2> 1624764 T10184 C:control_collection c:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a4bcf7 [junit4] 2> 1624765 T10184 C:control_collection c:collection1 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\data [junit4] 2> 1624765 T10184 C:control_collection c:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\data\index/ [junit4] 2> 1624766 T10184 C:control_collection c:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\data\index' doesn't exist. Creating new index... [junit4] 2> 1624767 T10184 C:control_collection c:collection1 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\data\index [junit4] 2> 1624767 T10184 C:control_collection c:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=37.736328125, floorSegmentMB=1.9677734375, forceMergeDeletesPctAllowed=18.634962535671566, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33981433401806516 [junit4] 2> 1624775 T10184 C:control_collection c:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\data\index,segFN=segments_1,generation=1} [junit4] 2> 1624775 T10184 C:control_collection c:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 1624780 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib" [junit4] 2> 1624780 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe" [junit4] 2> 1624781 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 1624781 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig" [junit4] 2> 1624781 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 1624781 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 1624781 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 1624781 T10184 C:control_collection c:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 1624782 T10184 C:control_collection c:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 1624785 T10184 C:control_collection c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1624787 T10184 C:control_collection c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1624788 T10184 C:control_collection c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1624790 T10184 C:control_collection c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1624793 T10184 C:control_collection c:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file [junit4] 2> 1624793 T10184 C:control_collection c:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache [junit4] 2> 1624794 T10184 C:control_collection c:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1624794 T10184 C:control_collection c:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 [junit4] 2> 1624795 T10184 C:control_collection c:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4] 2> 1624795 T10184 C:control_collection c:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4] 2> 1624795 T10184 C:control_collection c:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15539120955552743] [junit4] 2> 1624796 T10184 C:control_collection c:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\control-001\cores\collection1\data\index,segFN=segments_1,generation=1} [junit4] 2> 1624796 T10184 C:control_collection c:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 1624796 T10184 C:control_collection c:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@2c448bb5[collection1] main [junit4] 2> 1624796 T10184 C:control_collection c:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection [junit4] 2> 1624797 T10184 C:control_collection c:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper [junit4] 2> 1624797 T10184 C:control_collection c:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 1624799 T10184 C:control_collection c:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1624800 T10184 C:control_collection c:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {} [junit4] 2> 1624800 T10184 C:control_collection c:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1624800 T10184 C:control_collection c:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json [junit4] 2> 1624800 T10184 C:control_collection c:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1624800 T10184 C:control_collection c:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources [junit4] 2> 1624800 T10184 C:control_collection c:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000 [junit4] 2> 1624801 T10184 C:control_collection c:collection1 oasc.CoreContainer.registerCore registering core: collection1 [junit4] 2> 1624801 T10185 C:control_collection c:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c448bb5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1624802 T10188 C:control_collection S:shard1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57906/_h collection:control_collection shard:shard1 [junit4] 2> 1624802 T10152 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1 [junit4] 2> 1624802 T10152 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 1624802 T10188 C:control_collection S:shard1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election [junit4] 2> 1624802 T10152 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 1624803 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1624805 T10191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14ba493c name:ZooKeeperConnection Watcher:127.0.0.1:57899/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1624805 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1624805 T10152 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 1624806 T10152 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1624806 T10188 C:control_collection S:shard1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1 [junit4] 2> 1624807 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1624809 T10188 C:control_collection S:shard1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue. [junit4] 2> 1624809 T10188 C:control_collection S:shard1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync [junit4] 2> 1624809 T10152 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false [junit4] 2> 1624809 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection"} current state version: 1 [junit4] 2> ASYNC NEW_CORE C4621 name=collection1 org.apache.solr.core.SolrCore@11f970b7 url=http://127.0.0.1:57906/_h/collection1 node=127.0.0.1:57906__h C4621_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:57906/_h, node_name=127.0.0.1:57906__h, state=down} [junit4] 2> 1624809 T10188 C:control_collection S:shard1 C4621 P57906 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57906/_h/collection1/ [junit4] 2> 1624810 T10188 C:control_collection S:shard1 C4621 P57906 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me [junit4] 2> 1624810 T10188 C:control_collection S:shard1 C4621 P57906 oasc.SyncStrategy.syncToMe http://127.0.0.1:57906/_h/collection1/ has no replicas [junit4] 2> 1624810 T10188 C:control_collection S:shard1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57906/_h/collection1/ shard1 [junit4] 2> 1624810 T10188 C:control_collection S:shard1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1 [junit4] 2> 1624819 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1624820 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "base_url":"http://127.0.0.1:57906/_h", [junit4] 2> "core":"collection1", [junit4] 2> "state":"active"} current state version: 1 [junit4] 2> 1624937 T10191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1624938 T10180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1624995 T10188 C:control_collection S:shard1 oasc.ZkController.register We are http://127.0.0.1:57906/_h/collection1/ and leader is http://127.0.0.1:57906/_h/collection1/ [junit4] 2> 1624995 T10188 C:control_collection S:shard1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57906/_h [junit4] 2> 1624995 T10188 C:control_collection S:shard1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary [junit4] 2> 1624995 T10188 C:control_collection S:shard1 c:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection [junit4] 2> 1624995 T10188 C:control_collection S:shard1 c:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 1624998 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1625000 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57906/_h", [junit4] 2> "node_name":"127.0.0.1:57906__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} current state version: 2 [junit4] 2> 1625001 T10181 oasco.ReplicaMutator.updateState Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57906/_h", [junit4] 2> "node_name":"127.0.0.1:57906__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} [junit4] 2> 1625118 T10152 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1 [junit4] 2> 1625118 T10180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1625118 T10191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1625121 T10152 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001 [junit4] 2> 1625122 T10152 oejs.Server.doStart jetty-9.2.9.v20150224 [junit4] 2> 1625124 T10152 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5996e67d{/_h,null,AVAILABLE} [junit4] 2> 1625125 T10152 oejs.AbstractConnector.doStart Started ServerConnector@1a95045b{HTTP/1.1}{127.0.0.1:57918} [junit4] 2> 1625126 T10152 oejs.Server.doStart Started @1635498ms [junit4] 2> 1625126 T10152 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/_h, hostPort=57918, coreRootDirectory=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores} [junit4] 2> 1625126 T10152 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a [junit4] 2> 1625126 T10152 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\' [junit4] 2> 1625151 T10152 oasc.SolrXmlConfig.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\solr.xml [junit4] 2> 1625157 T10152 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores [junit4] 2> 1625157 T10152 oasc.CoreContainer.<init> New CoreContainer 410628844 [junit4] 2> 1625159 T10152 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\] [junit4] 2> 1625159 T10152 oasc.CoreContainer.load loading shared library: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\lib [junit4] 2> 1625159 T10152 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\lib). [junit4] 2> 1625166 T10152 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 1625166 T10152 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 1625166 T10152 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 1625166 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 1625167 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 1625167 T10152 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 1625167 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 1625167 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 1625167 T10152 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 1625169 T10152 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 1625169 T10152 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false [junit4] 2> 1625169 T10152 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 1625169 T10152 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 1625169 T10152 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 1625170 T10152 oasc.CoreContainer.load Node Name: 127.0.0.1 [junit4] 2> 1625170 T10152 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57899/solr [junit4] 2> 1625170 T10152 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 1625170 T10152 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 1625171 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1625173 T10208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ab5da35 name:ZooKeeperConnection Watcher:127.0.0.1:57899 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1625173 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1625173 T10152 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 1625176 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1625178 T10211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@311bc732 name:ZooKeeperConnection Watcher:127.0.0.1:57899/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1625178 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1625183 T10152 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1626332 T10152 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57918__h [junit4] 2> 1626333 T10152 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57918__h [junit4] 2> 1626337 T10152 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 1626339 T10152 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores [junit4] 2> 1626341 T10152 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1, collection=collection1, absoluteInstDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\, coreNodeName=, dataDir=data\, shard=} [junit4] 2> 1626341 T10152 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\ [junit4] 2> 1626341 T10152 oasc.CorePropertiesLocator.discover Found 1 core definitions [junit4] 2> 1626342 T10212 C:collection1 c:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1 [junit4] 2> 1626342 T10212 C:collection1 c:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 1626343 T10212 C:collection1 c:collection1 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 1626343 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1626345 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57918/_h", [junit4] 2> "node_name":"127.0.0.1:57918__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} current state version: 3 [junit4] 2> 1626346 T10181 oasco.ReplicaMutator.updateState Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57918/_h", [junit4] 2> "node_name":"127.0.0.1:57918__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} [junit4] 2> 1626346 T10181 oasco.ClusterStateMutator.createCollection building a new cName: collection1 [junit4] 2> 1626346 T10181 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1 [junit4] 2> 1626463 T10180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1626463 T10211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1626463 T10191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1627488 T10212 C:collection1 c:collection1 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1 [junit4] 2> 1627488 T10212 C:collection1 c:collection1 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1 [junit4] 2> 1627489 T10212 C:collection1 c:collection1 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 1627489 T10212 C:collection1 c:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1 [junit4] 2> 1627490 T10212 C:collection1 c:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper [junit4] 2> 1627490 T10212 C:collection1 c:collection1 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\' [junit4] 2> 1627515 T10212 C:collection1 c:collection1 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1 [junit4] 2> 1627517 T10212 C:collection1 c:collection1 oasc.Config.<init> loaded config solrconfig.xml with version 0 [junit4] 2> 1627524 T10212 C:collection1 c:collection1 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1 [junit4] 2> 1627526 T10212 C:collection1 c:collection1 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0 [junit4] 2> 1627533 T10212 C:collection1 c:collection1 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 1627535 T10212 C:collection1 c:collection1 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml [junit4] 2> 1627539 T10212 C:collection1 c:collection1 oass.IndexSchema.readSchema [collection1] Schema name=test [junit4] 2> 1627610 T10212 C:collection1 c:collection1 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 1627611 T10212 C:collection1 c:collection1 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 1627611 T10212 C:collection1 c:collection1 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 1627615 T10212 C:collection1 c:collection1 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 1627634 T10212 C:collection1 c:collection1 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1 [junit4] 2> 1627634 T10212 C:collection1 c:collection1 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory [junit4] 2> 1627635 T10212 C:collection1 c:collection1 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\data\ [junit4] 2> 1627635 T10212 C:collection1 c:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a4bcf7 [junit4] 2> 1627636 T10212 C:collection1 c:collection1 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\data [junit4] 2> 1627636 T10212 C:collection1 c:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\data\index/ [junit4] 2> 1627637 T10212 C:collection1 c:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\data\index' doesn't exist. Creating new index... [junit4] 2> 1627638 T10212 C:collection1 c:collection1 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\data\index [junit4] 2> 1627638 T10212 C:collection1 c:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=37.736328125, floorSegmentMB=1.9677734375, forceMergeDeletesPctAllowed=18.634962535671566, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33981433401806516 [junit4] 2> 1627641 T10212 C:collection1 c:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\data\index,segFN=segments_1,generation=1} [junit4] 2> 1627641 T10212 C:collection1 c:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 1627645 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib" [junit4] 2> 1627646 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe" [junit4] 2> 1627646 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 1627646 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig" [junit4] 2> 1627646 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 1627646 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 1627647 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 1627647 T10212 C:collection1 c:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 1627647 T10212 C:collection1 c:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 1627650 T10212 C:collection1 c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1627651 T10212 C:collection1 c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1627653 T10212 C:collection1 c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1627654 T10212 C:collection1 c:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 1627658 T10212 C:collection1 c:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file [junit4] 2> 1627659 T10212 C:collection1 c:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache [junit4] 2> 1627659 T10212 C:collection1 c:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1627659 T10212 C:collection1 c:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 [junit4] 2> 1627660 T10212 C:collection1 c:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4] 2> 1627660 T10212 C:collection1 c:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4] 2> 1627660 T10212 C:collection1 c:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15539120955552743] [junit4] 2> 1627661 T10212 C:collection1 c:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-1-001\cores\collection1\data\index,segFN=segments_1,generation=1} [junit4] 2> 1627661 T10212 C:collection1 c:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 1627661 T10212 C:collection1 c:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@3ca37909[collection1] main [junit4] 2> 1627661 T10212 C:collection1 c:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1 [junit4] 2> 1627663 T10212 C:collection1 c:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper [junit4] 2> 1627663 T10212 C:collection1 c:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 1627664 T10212 C:collection1 c:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1627664 T10212 C:collection1 c:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {} [junit4] 2> 1627664 T10212 C:collection1 c:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1627665 T10212 C:collection1 c:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json [junit4] 2> 1627665 T10212 C:collection1 c:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1627665 T10212 C:collection1 c:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources [junit4] 2> 1627665 T10212 C:collection1 c:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000 [junit4] 2> 1627674 T10213 C:collection1 c:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ca37909[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1627675 T10212 C:collection1 c:collection1 oasc.CoreContainer.registerCore registering core: collection1 [junit4] 2> 1627676 T10216 C:collection1 S:shard1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57918/_h collection:collection1 shard:shard1 [junit4] 2> 1627676 T10152 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1 [junit4] 2> 1627676 T10152 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 1627677 T10216 C:collection1 S:shard1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 1627682 T10216 C:collection1 S:shard1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1 [junit4] 2> 1627683 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1627684 T10216 C:collection1 S:shard1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue. [junit4] 2> 1627684 T10216 C:collection1 S:shard1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync [junit4] 2> ASYNC NEW_CORE C4622 name=collection1 org.apache.solr.core.SolrCore@372d394b url=http://127.0.0.1:57918/_h/collection1 node=127.0.0.1:57918__h C4622_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:57918/_h, node_name=127.0.0.1:57918__h, state=down} [junit4] 2> 1627685 T10216 C:collection1 S:shard1 C4622 P57918 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57918/_h/collection1/ [junit4] 2> 1627685 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1"} current state version: 4 [junit4] 2> 1627685 T10216 C:collection1 S:shard1 C4622 P57918 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me [junit4] 2> 1627685 T10216 C:collection1 S:shard1 C4622 P57918 oasc.SyncStrategy.syncToMe http://127.0.0.1:57918/_h/collection1/ has no replicas [junit4] 2> 1627685 T10216 C:collection1 S:shard1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57918/_h/collection1/ shard1 [junit4] 2> 1627685 T10216 C:collection1 S:shard1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1627689 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1627691 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1", [junit4] 2> "base_url":"http://127.0.0.1:57918/_h", [junit4] 2> "core":"collection1", [junit4] 2> "state":"active"} current state version: 4 [junit4] 2> 1627809 T10211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1627809 T10180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1627810 T10191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1627865 T10216 C:collection1 S:shard1 oasc.ZkController.register We are http://127.0.0.1:57918/_h/collection1/ and leader is http://127.0.0.1:57918/_h/collection1/ [junit4] 2> 1627865 T10216 C:collection1 S:shard1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57918/_h [junit4] 2> 1627865 T10216 C:collection1 S:shard1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary [junit4] 2> 1627865 T10216 C:collection1 S:shard1 c:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1 [junit4] 2> 1627865 T10216 C:collection1 S:shard1 c:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 1627866 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1627867 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57918/_h", [junit4] 2> "node_name":"127.0.0.1:57918__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} current state version: 5 [junit4] 2> 1627868 T10181 oasco.ReplicaMutator.updateState Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57918/_h", [junit4] 2> "node_name":"127.0.0.1:57918__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} [junit4] 2> 1627954 T10152 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1 [junit4] 2> 1627957 T10152 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001 [junit4] 2> 1627958 T10152 oejs.Server.doStart jetty-9.2.9.v20150224 [junit4] 2> 1627961 T10152 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7f003451{/_h,null,AVAILABLE} [junit4] 2> 1627962 T10152 oejs.AbstractConnector.doStart Started ServerConnector@185470d1{HTTP/1.1}{127.0.0.1:57927} [junit4] 2> 1627962 T10152 oejs.Server.doStart Started @1637981ms [junit4] 2> 1627962 T10152 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/_h, hostPort=57927, coreRootDirectory=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores} [junit4] 2> 1627962 T10152 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a [junit4] 2> 1627963 T10152 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\' [junit4] 2> 1627985 T10152 oasc.SolrXmlConfig.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\solr.xml [junit4] 2> 1627987 T10211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1627987 T10180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1627988 T10191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 1627992 T10152 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores [junit4] 2> 1627994 T10152 oasc.CoreContainer.<init> New CoreContainer 1393725004 [junit4] 2> 1627994 T10152 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\] [junit4] 2> 1627994 T10152 oasc.CoreContainer.load loading shared library: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\lib [junit4] 2> 1627994 T10152 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\lib). [junit4] 2> 1628001 T10152 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 1628001 T10152 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 1628001 T10152 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 1628001 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 1628002 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 1628002 T10152 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 1628002 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 1628002 T10152 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 1628002 T10152 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 1628002 T10152 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 1628004 T10152 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false [junit4] 2> 1628004 T10152 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 1628004 T10152 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 1628004 T10152 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 1628004 T10152 oasc.CoreContainer.load Node Name: 127.0.0.1 [junit4] 2> 1628005 T10152 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57899/solr [junit4] 2> 1628005 T10152 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 1628005 T10152 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 1628006 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1628008 T10230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43d823b2 name:ZooKeeperConnection Watcher:127.0.0.1:57899 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1628008 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1628008 T10152 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 1628011 T10152 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1628013 T10233 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77ccc98c name:ZooKeeperConnection Watcher:127.0.0.1:57899/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1628013 T10152 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1628018 T10152 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1629167 T10152 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57927__h [junit4] 2> 1629168 T10152 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57927__h [junit4] 2> 1629172 T10152 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 1629175 T10152 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores [junit4] 2> 1629178 T10152 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1, collection=collection1, absoluteInstDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\, coreNodeName=, dataDir=data\, shard=} [junit4] 2> 1629178 T10152 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\ [junit4] 2> 1629178 T10152 oasc.CorePropertiesLocator.discover Found 1 core definitions [junit4] 2> 1629180 T10234 C:collection1 c:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1 [junit4] 2> 1629180 T10234 C:collection1 c:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 1629181 T10234 C:collection1 c:collection1 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 1629181 T10180 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 1629182 T10181 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57927/_h", [junit4] 2> "node_name":"127.0.0.1:57927__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} current state version: 6 [junit4] 2> 1629183 T10181 oasco.ReplicaMutator.updateState Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:57927/_h", [junit4] 2> "node_name":"127.0.0.1:57927__h", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} [junit4] 2> 1629183 T10181 oasco.ReplicaMutator.updateState Collection already exists with numShards=1 [junit4] 2> 1629183 T10181 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1 [junit4] 2> 1629301 T10211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 1629301 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 1629301 T10180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 1629301 T10191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3) [junit4] 2> 1630325 T10234 C:collection1 c:collection1 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1 [junit4] 2> 1630325 T10234 C:collection1 c:collection1 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1 [junit4] 2> 1630327 T10234 C:collection1 c:collection1 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 1630327 T10234 C:collection1 c:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1 [junit4] 2> 1630328 T10234 C:collection1 c:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper [junit4] 2> 1630328 T10234 C:collection1 c:collection1 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\' [junit4] 2> 1630342 T10234 C:collection1 c:collection1 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1 [junit4] 2> 1630344 T10234 C:collection1 c:collection1 oasc.Config.<init> loaded config solrconfig.xml with version 0 [junit4] 2> 1630352 T10234 C:collection1 c:collection1 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1 [junit4] 2> 1630356 T10234 C:collection1 c:collection1 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0 [junit4] 2> 1630362 T10234 C:collection1 c:collection1 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 1630363 T10234 C:collection1 c:collection1 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml [junit4] 2> 1630368 T10234 C:collection1 c:collection1 oass.IndexSchema.readSchema [collection1] Schema name=test [junit4] 2> 1630464 T10234 C:collection1 c:collection1 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 1630465 T10234 C:collection1 c:collection1 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 1630465 T10234 C:collection1 c:collection1 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 1630469 T10234 C:collection1 c:collection1 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 1630492 T10234 C:collection1 c:collection1 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1 [junit4] 2> 1630492 T10234 C:collection1 c:collection1 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory [junit4] 2> 1630492 T10234 C:collection1 c:collection1 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data\ [junit4] 2> 1630492 T10234 C:collection1 c:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a4bcf7 [junit4] 2> 1630493 T10234 C:collection1 c:collection1 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data [junit4] 2> 1630494 T10234 C:collection1 c:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data\index/ [junit4] 2> 1630494 T10234 C:collection1 c:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data\index' doesn't exist. Creating new index... [junit4] 2> 1630495 T10234 C:collection1 c:collection1 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data\index [junit4] 2> 1630495 T10234 C:collection1 c:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=37.736328125, floorSegme [...truncated too long message...] on1 oasc.CachingDirectoryFactory.close Closing directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data [junit4] 2> 1661003 T10152 C:control_collection S:shard1 c:collection1 oasc.CachingDirectoryFactory.closeCacheValue looking to close C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data\index [CachedDir<<refCount=0;path=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data\index;done=false>>] [junit4] 2> 1661003 T10152 C:control_collection S:shard1 c:collection1 oasc.CachingDirectoryFactory.close Closing directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001\shard-2-001\cores\collection1\data\index [junit4] 2> 1661004 T10152 C:control_collection S:shard1 c:collection1 oasc.Overseer.close Overseer (id=93544537104384008-127.0.0.1:57927__h-n_0000000002) closing [junit4] 2> 1661004 T10282 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57927__h [junit4] 2> 1662132 T10255 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 1662133 T10152 C:control_collection S:shard1 c:collection1 oejs.AbstractConnector.doStop Stopped ServerConnector@185470d1{HTTP/1.1}{127.0.0.1:0} [junit4] 2> 1662134 T10152 C:control_collection S:shard1 c:collection1 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@7f003451{/_h,null,UNAVAILABLE} [junit4] 2> 1662135 T10152 C:control_collection S:shard1 c:collection1 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57899 57899 [junit4] 2> 1663826 T10265 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57899 57899 [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=test -Dtests.seed=D021BCBB66EB4B4C -Dtests.slow=true -Dtests.locale=lt -Dtests.timezone=Asia/Macao -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] FAILURE 41.0s J1 | ChaosMonkeyNothingIsSafeTest.test <<< [junit4] > Throwable #1: java.lang.AssertionError: There were too many update fails (30 > 20) - we expect it can happen, but shouldn't easily [junit4] > at __randomizedtesting.SeedInfo.seed([D021BCBB66EB4B4C:58758361C81726B4]:0) [junit4] > at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:230) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> 1663833 T10152 C:control_collection S:shard1 c:collection1 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.ChaosMonkeyNothingIsSafeTest D021BCBB66EB4B4C-001 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {rnd_b=PostingsFormat(name=Asserting), _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_i=PostingsFormat(name=Asserting), id=PostingsFormat(name=Asserting)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=lt, timezone=Asia/Macao [junit4] 2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0_40 (64-bit)/cpus=3,threads=1,free=78011272,total=279347200 [junit4] 2> NOTE: All tests run in this JVM: [HdfsUnloadDistributedZkTest, TestDistribDocBasedVersion, CustomCollectionTest, TestJsonRequest, DebugComponentTest, JsonLoaderTest, DistributedDebugComponentTest, TestJoin, TestSearchPerf, StatsComponentTest, SpellCheckCollatorTest, TestRemoteStreaming, TestPerFieldSimilarity, ConnectionManagerTest, CursorMarkTest, MoreLikeThisHandlerTest, ShardSplitTest, SolrIndexConfigTest, TestOverriddenPrefixQueryForCustomFieldType, ChaosMonkeySafeLeaderTest, UpdateParamsTest, TestMissingGroups, DeleteInactiveReplicaTest, TestBlendedInfixSuggestions, TestSolrDeletionPolicy1, UUIDUpdateProcessorFallbackTest, RegexBoostProcessorTest, SolrCmdDistributorTest, TestSolrQueryParser, UUIDFieldTest, QueryEqualityTest, CSVRequestHandlerTest, DistributedTermsComponentTest, TestSolrQueryParserDefaultOperatorResource, SOLR749Test, DistributedFacetPivotLargeTest, DistributedQueueTest, URLClassifyProcessorTest, SpellingQueryConverterTest, CursorPagingTest, TestReloadAndDeleteDocs, PrimitiveFieldTypeTest, AddBlockUpdateTest, TestReversedWildcardFilterFactory, AsyncMigrateRouteKeyTest, FastVectorHighlighterTest, OverseerStatusTest, TestReload, ConvertedLegacyTest, AlternateDirectoryTest, TestBulkSchemaAPI, AutoCommitTest, RollingRestartTest, TestSchemaSimilarityResource, TestReplicationHandlerBackup, TestEmbeddedSolrServer, DeleteShardTest, CachingDirectoryFactoryTest, TestTrie, BasicDistributedZk2Test, TestConfigReload, IgnoreCommitOptimizeUpdateProcessorFactoryTest, HdfsThreadLeakTest, TestBadConfig, TestNonDefinedSimilarityFactory, DirectSolrSpellCheckerTest, CircularListTest, TestSolrConfigHandlerCloud, LeaderElectionIntegrationTest, TestRecovery, TestCloudManagedSchemaConcurrent, SolrTestCaseJ4Test, FullSolrCloudDistribCmdsTest, TestLFUCache, SegmentsInfoRequestHandlerTest, TestAnalyzedSuggestions, TestBM25SimilarityFactory, HdfsDirectoryFactoryTest, SignatureUpdateProcessorFactoryTest, SliceStateTest, OverseerCollectionProcessorTest, TestInfoStreamLogging, SpatialFilterTest, BlockCacheTest, UpdateRequestProcessorFactoryTest, TestSerializedLuceneMatchVersion, TestPostingsSolrHighlighter, SynonymTokenizerTest, SharedFSAutoReplicaFailoverTest, DirectUpdateHandlerOptimizeTest, ResponseHeaderTest, TestSolr4Spatial, TestRebalanceLeaders, TestCollationField, TestLuceneMatchVersion, RemoteQueryErrorTest, ExternalCollectionsTest, TestCollapseQParserPlugin, TestDocSet, StatelessScriptUpdateProcessorFactoryTest, TestMiniSolrCloudCluster, TestInitQParser, TestSolrConfigHandlerConcurrent, OpenExchangeRatesOrgProviderTest, OpenCloseCoreStressTest, SimpleCollectionCreateDeleteTest, FieldMutatingUpdateProcessorTest, TestConfig, TestJsonFacets, TestDefaultStatsCache, JSONWriterTest, TestRandomMergePolicy, TestComponentsName, WordBreakSolrSpellCheckerTest, CollectionsAPIDistributedZkTest, BasicZkTest, RecoveryZkTest, TestZkChroot, ZkCLITest, TestDistributedGrouping, TermVectorComponentDistributedTest, TestRealTimeGet, TestStressReorder, TestStressVersions, HardAutoCommitTest, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, SpellCheckComponentTest, TestGroupingSearch, BadIndexSchemaTest, TestFiltering, BasicFunctionalityTest, DirectUpdateHandlerTest, HighlighterTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, TestExtendedDismaxParser, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTSTTest, TestCSVLoader, SchemaVersionSpecificBehaviorTest, SortByFunctionTest, DistanceFunctionTest, XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, TestQueryUtils, DocumentAnalysisRequestHandlerTest, TestOmitPositions, TermsComponentTest, DocumentBuilderTest, TestValueSourceCache, TestIndexingPerformance, TestArbitraryIndexDir, SolrPluginUtilsTest, ReturnFieldsTest, QueryParsingTest, PingRequestHandlerTest, HighlighterConfigTest, BadComponentTest, TestMergePolicyConfig, NumericFieldsTest, TestElisionMultitermQuery, ExternalFileFieldSortTest, NotRequiredUniqueKeyTest, TestCodecSupport, EchoParamsTest, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestFastWriter, TestFastOutputStream, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest] [junit4] Completed on J1 in 41.00s, 1 test, 1 failure <<< FAILURES! [...truncated 360 lines...] BUILD FAILED C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:519: The following error occurred while executing this line: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:467: The following error occurred while executing this line: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:61: The following error occurred while executing this line: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:191: The following error occurred while executing this line: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:510: The following error occurred while executing this line: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1351: The following error occurred while executing this line: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:958: There were test failures: 478 suites, 1899 tests, 1 failure, 58 ignored (30 assumptions) Total time: 57 minutes 27 seconds Build step 'Invoke Ant' marked build as failure [description-setter] Description set: Java: 64bit/jdk1.8.0_40 -XX:+UseCompressedOops -XX:+UseSerialGC Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org