OK, I think I've found a possible reason, and it's probably a test
failure for the BasicDistributedZkTest. The waitForDocCount method
waits until a response comes back with the, well, expected doc counts.
But then it drops out of the wait loop the first time a query works.

But then it goes out to each and every node and re-issues the request.
This looks to be a 2-shard, 2-replica situation. So here's the theory:
the second (or more) nodes haven't yet opened a new searcher. So the
wait loop is satisfied by, say, node2 but the test later looks at
node4 (both for shard2) which hasn't completed opening a searcher yet
so it fails.

I could not get this to fail locally in 20 runs. So I'll beast the
unchanged version some more to see but meanwhile commit this change
which I think is more correct anyway and monitor.

See SOLR-8658

This is unrelated to the second test failure about replication handler
which I'm not addressing.





On Mon, Feb 8, 2016 at 8:07 AM, Erick Erickson <erickerick...@gmail.com> wrote:
> Digging into this, I changed test recently.
>
>
> On Mon, Feb 8, 2016 at 6:05 AM, Policeman Jenkins Server
> <jenk...@thetaphi.de> wrote:
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/3073/
>> Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>>
>> 2 tests failed.
>> FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test
>>
>> Error Message:
>> add commitWithin did not work expected:<69> but was:<67>
>>
>> Stack Trace:
>> java.lang.AssertionError: add commitWithin did not work expected:<69> but 
>> was:<67>
>>         at 
>> __randomizedtesting.SeedInfo.seed([BA21C32BCEFAAE7F:3275FCF16006C387]:0)
>>         at org.junit.Assert.fail(Assert.java:93)
>>         at org.junit.Assert.failNotEquals(Assert.java:647)
>>         at org.junit.Assert.assertEquals(Assert.java:128)
>>         at org.junit.Assert.assertEquals(Assert.java:472)
>>         at 
>> org.apache.solr.cloud.BasicDistributedZkTest.waitForDocCount(BasicDistributedZkTest.java:385)
>>         at 
>> org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:330)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>         at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
>>         at 
>> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:964)
>>         at 
>> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:939)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>         at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>         at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>         at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>         at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>         at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>>         at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
>>         at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>         at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>         at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>         at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>         at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED:  
>> junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler
>>
>> Error Message:
>> ObjectTracker found 1 object(s) that were not released!!! 
>> [NRTCachingDirectory]
>>
>> Stack Trace:
>> java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
>> released!!! [NRTCachingDirectory]
>>         at __randomizedtesting.SeedInfo.seed([BA21C32BCEFAAE7F]:0)
>>         at org.junit.Assert.fail(Assert.java:93)
>>         at org.junit.Assert.assertTrue(Assert.java:43)
>>         at org.junit.Assert.assertNull(Assert.java:551)
>>         at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:228)
>>         at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
>>         at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>>         at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>         at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>         at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>         at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>         at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>         at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 11196 lines...]
>>    [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
>>    [junit4]   2> Creating dataDir: 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/init-core-data-001
>>    [junit4]   2> 1453544 INFO  
>> (SUITE-BasicDistributedZkTest-seed#[BA21C32BCEFAAE7F]-worker) [    ] 
>> o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
>>    [junit4]   2> 1453546 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>>    [junit4]   2> 1453547 INFO  (Thread-3401) [    ] o.a.s.c.ZkTestServer 
>> client port:0.0.0.0/0.0.0.0:0
>>    [junit4]   2> 1453547 INFO  (Thread-3401) [    ] o.a.s.c.ZkTestServer 
>> Starting server
>>    [junit4]   2> 1453648 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.ZkTestServer start zk server on port:51017
>>    [junit4]   2> 1453648 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>    [junit4]   2> 1453650 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>    [junit4]   2> 1453660 INFO  (zkCallback-1536-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@19e3befa 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017 got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1453660 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>    [junit4]   2> 1453660 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>    [junit4]   2> 1453661 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /solr
>>    [junit4]   2> 1453666 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>    [junit4]   2> 1453667 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>    [junit4]   2> 1453670 INFO  (zkCallback-1537-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@58428dd6 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017/solr got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1453670 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>    [junit4]   2> 1453670 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>    [junit4]   2> 1453670 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /collections/collection1
>>    [junit4]   2> 1453673 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
>>    [junit4]   2> 1453676 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
>>    [junit4]   2> 1453679 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
>>    [junit4]   2> 1453682 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
>>  to /configs/conf1/solrconfig.xml
>>    [junit4]   2> 1453682 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
>>    [junit4]   2> 1453687 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml
>>  to /configs/conf1/schema.xml
>>    [junit4]   2> 1453687 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
>>    [junit4]   2> 1453691 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
>>  to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>>    [junit4]   2> 1453691 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: 
>> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>>    [junit4]   2> 1453695 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
>>  to /configs/conf1/stopwords.txt
>>    [junit4]   2> 1453695 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
>>    [junit4]   2> 1453698 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt
>>  to /configs/conf1/protwords.txt
>>    [junit4]   2> 1453698 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
>>    [junit4]   2> 1453701 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml
>>  to /configs/conf1/currency.xml
>>    [junit4]   2> 1453701 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
>>    [junit4]   2> 1453704 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
>>  to /configs/conf1/enumsConfig.xml
>>    [junit4]   2> 1453705 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
>>    [junit4]   2> 1453707 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
>>  to /configs/conf1/open-exchange-rates.json
>>    [junit4]   2> 1453708 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
>>    [junit4]   2> 1453710 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
>>  to /configs/conf1/mapping-ISOLatin1Accent.txt
>>    [junit4]   2> 1453711 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>>    [junit4]   2> 1453713 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
>>  to /configs/conf1/old_synonyms.txt
>>    [junit4]   2> 1453714 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
>>    [junit4]   2> 1453716 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractZkTestCase put 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
>>  to /configs/conf1/synonyms.txt
>>    [junit4]   2> 1453716 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
>>    [junit4]   2> 1453967 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.SolrTestCaseJ4 Writing core.properties file to 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1
>>    [junit4]   2> 1453969 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.Server jetty-9.2.13.v20150730
>>    [junit4]   2> 1453972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.h.ContextHandler Started 
>> o.e.j.s.ServletContextHandler@6db9b9a5{/,null,AVAILABLE}
>>    [junit4]   2> 1453972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.ServerConnector Started 
>> ServerConnector@26033313{HTTP/1.1}{127.0.0.1:51020}
>>    [junit4]   2> 1453972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.Server Started @1457321ms
>>    [junit4]   2> 1453972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.s.e.JettySolrRunner Jetty properties: 
>> {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/tempDir-001/control/data,
>>  hostContext=/, hostPort=51020, 
>> coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores}
>>    [junit4]   2> 1453973 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
>> sun.misc.Launcher$AppClassLoader@1540e19d
>>    [junit4]   2> 1453973 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
>> '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001'
>>    [junit4]   2> 1453973 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>    [junit4]   2> 1453973 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
>> system property or JNDI)
>>    [junit4]   2> 1453973 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>    [junit4]   2> 1453974 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>    [junit4]   2> 1453975 INFO  (zkCallback-1538-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@1c8adae7 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017/solr got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1453975 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>    [junit4]   2> 1453975 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>    [junit4]   2> 1453977 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
>> ZooKeeper)
>>    [junit4]   2> 1453977 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrXmlConfig Loading container configuration from 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/solr.xml
>>    [junit4]   2> 1454025 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CorePropertiesLocator Config-defined core root directory: 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores
>>    [junit4]   2> 1454025 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CoreContainer New CoreContainer 609395164
>>    [junit4]   2> 1454025 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CoreContainer Loading cores into CoreContainer 
>> [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001]
>>    [junit4]   2> 1454026 WARN  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CoreContainer Couldn't add files from 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/lib
>>  to classpath: 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/lib
>>    [junit4]   2> 1454026 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
>> 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
>> 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
>> 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
>> false,useRetries : false,
>>    [junit4]   2> 1454028 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
>> params: socketTimeout=340000&connTimeout=45000&retry=true
>>    [junit4]   2> 1454028 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>    [junit4]   2> 1454028 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.l.LogWatcher Registering Log Listener [Log4j 
>> (org.slf4j.impl.Log4jLoggerFactory)]
>>    [junit4]   2> 1454029 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51017/solr
>>    [junit4]   2> 1454029 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.ZkController zkHost includes chroot
>>    [junit4]   2> 1454029 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>    [junit4]   2> 1454030 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>    [junit4]   2> 1454031 INFO  (zkCallback-1541-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@431c9cdb 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017 got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1454032 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>    [junit4]   2> 1454032 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>    [junit4]   2> 1454034 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
>> connect to ZooKeeper
>>    [junit4]   2> 1454036 INFO  
>> (zkCallback-1542-thread-1-processing-n:127.0.0.1:51020_) [n:127.0.0.1:51020_ 
>>    ] o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@57e86e88 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017/solr got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1454036 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.ConnectionManager Client is connected to 
>> ZooKeeper
>>    [junit4]   2> 1454037 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
>>    [junit4]   2> 1454040 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /overseer/collection-queue-work
>>    [junit4]   2> 1454046 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /overseer/collection-map-running
>>    [junit4]   2> 1454049 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /overseer/collection-map-completed
>>    [junit4]   2> 1454053 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /overseer/collection-map-failure
>>    [junit4]   2> 1454057 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>>    [junit4]   2> 1454059 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
>>    [junit4]   2> 1454061 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
>>    [junit4]   2> 1454064 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>>    [junit4]   2> 1454066 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.ZkController Register node as live in 
>> ZooKeeper:/live_nodes/127.0.0.1:51020_
>>    [junit4]   2> 1454066 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /live_nodes/127.0.0.1:51020_
>>    [junit4]   2> 1454069 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
>>    [junit4]   2> 1454071 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /overseer_elect/election
>>    [junit4]   2> 1454073 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer Overseer (id=null) closing
>>    [junit4]   2> 1454074 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.LeaderElector Joined leadership election 
>> with path: 
>> /overseer_elect/election/95350858267951108-127.0.0.1:51020_-n_0000000000
>>    [junit4]   2> 1454075 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.OverseerElectionContext I am going to be 
>> the leader 127.0.0.1:51020_
>>    [junit4]   2> 1454075 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /overseer_elect/leader
>>    [junit4]   2> 1454078 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer Overseer 
>> (id=95350858267951108-127.0.0.1:51020_-n_0000000000) starting
>>    [junit4]   2> 1454081 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /overseer/queue-work
>>    [junit4]   2> 1454090 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting 
>> OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
>> autoReplicaFailoverWaitAfterExpiration=30000 
>> autoReplicaFailoverBadNodeExpiration=60000
>>    [junit4]   2> 1454091 INFO  
>> (OverseerCollectionConfigSetProcessor-95350858267951108-127.0.0.1:51020_-n_0000000000)
>>  [n:127.0.0.1:51020_    ] o.a.s.c.OverseerTaskProcessor Process current 
>> queue of overseer operations
>>    [junit4]   2> 1454091 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
>> ZooKeeper...
>>    [junit4]   2> 1454092 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer Starting to work on the main queue
>>    [junit4]   2> 1454107 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
>> Skipping setup for authorization module.
>>    [junit4]   2> 1454107 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.CoreContainer No authentication plugin used.
>>    [junit4]   2> 1454110 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.CorePropertiesLocator Looking for core 
>> definitions underneath 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores
>>    [junit4]   2> 1454112 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: 
>> {name=collection1, config=solrconfig.xml, transient=false, 
>> schema=schema.xml, loadOnStartup=true, 
>> configSetProperties=configsetprops.json, collection=control_collection, 
>> coreNodeName=, dataDir=data/, shard=}
>>    [junit4]   2> 1454112 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.CorePropertiesLocator Found core 
>> collection1 in 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1
>>    [junit4]   2> 1454112 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.CorePropertiesLocator Found 1 core 
>> definitions
>>    [junit4]   2> 1454114 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.s.SolrDispatchFilter 
>> user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
>>    [junit4]   2> 1454114 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51020_    ] o.a.s.s.SolrDispatchFilter 
>> SolrDispatchFilter.init() done
>>    [junit4]   2> 1454114 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.ZkController publishing state=down
>>    [junit4]   2> 1454114 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.ZkController numShards not found on descriptor - reading it from 
>> system property
>>    [junit4]   2> 1454118 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.ZkController look for our core node name
>>    [junit4]   2> 1454118 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
>> message = {
>>    [junit4]   2>   "core":"collection1",
>>    [junit4]   2>   "roles":null,
>>    [junit4]   2>   "base_url":"http://127.0.0.1:51020";,
>>    [junit4]   2>   "node_name":"127.0.0.1:51020_",
>>    [junit4]   2>   "numShards":"1",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "shard":null,
>>    [junit4]   2>   "collection":"control_collection",
>>    [junit4]   2>   "operation":"state"} current state version: 0
>>    [junit4]   2> 1454119 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
>> message={
>>    [junit4]   2>   "core":"collection1",
>>    [junit4]   2>   "roles":null,
>>    [junit4]   2>   "base_url":"http://127.0.0.1:51020";,
>>    [junit4]   2>   "node_name":"127.0.0.1:51020_",
>>    [junit4]   2>   "numShards":"1",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "shard":null,
>>    [junit4]   2>   "collection":"control_collection",
>>    [junit4]   2>   "operation":"state"}
>>    [junit4]   2> 1454119 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
>> control_collection
>>    [junit4]   2> 1454119 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
>> shard shard=shard1
>>    [junit4]   2> 1454121 INFO  
>> (zkCallback-1542-thread-1-processing-n:127.0.0.1:51020_) [n:127.0.0.1:51020_ 
>>    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
>> state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has 
>> occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 1455119 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
>>    [junit4]   2> 1455120 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.ZkController Check for collection zkNode:control_collection
>>    [junit4]   2> 1455120 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.ZkController Collection zkNode exists
>>    [junit4]   2> 1455120 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.c.ZkStateReader Load collection config from: 
>> [/collections/control_collection]
>>    [junit4]   2> 1455121 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.c.ZkStateReader path=[/collections/control_collection] 
>> [configName]=[conf1] specified config exists in ZooKeeper
>>    [junit4]   2> 1455121 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
>> '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1'
>>    [junit4]   2> 1455122 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>    [junit4]   2> 1455122 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
>> system property or JNDI)
>>    [junit4]   2> 1455124 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] o.a.s.c.Config 
>> loaded config solrconfig.xml with version 0
>>    [junit4]   2> 1455132 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
>> current version of requestparams : -1
>>    [junit4]   2> 1455136 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
>> Using Lucene MatchVersion: 6.0.0
>>    [junit4]   2> 1455166 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
>> Loaded SolrConfig: solrconfig.xml
>>    [junit4]   2> 1455173 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.IndexSchema [collection1] Schema name=test
>>    [junit4]   2> 1455302 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
>> rates=open-exchange-rates.json, refreshInterval=1440.
>>    [junit4]   2> 1455312 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.IndexSchema default search field in schema is text
>>    [junit4]   2> 1455314 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.IndexSchema unique key field: id
>>    [junit4]   2> 1455330 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file 
>> currency.xml
>>    [junit4]   2> 1455334 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file 
>> currency.xml
>>    [junit4]   2> 1455338 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
>> open-exchange-rates.json
>>    [junit4]   2> 1455339 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
>> open-exchange-rates.json
>>    [junit4]   2> 1455343 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming 
>> default properties: Can't find resource 'configsetprops.json' in classpath 
>> or '/configs/conf1', 
>> cwd=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
>>    [junit4]   2> 1455343 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection   x:collection1] 
>> o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration 
>> from collection control_collection
>>    [junit4]   2> 1455343 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
>>    [junit4]   2> 1455344 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
>> [/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1],
>>  
>> dataDir=[/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1/data/]
>>    [junit4]   2> 1455344 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding 
>> Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76cd5d4e
>>    [junit4]   2> 1455344 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1/data
>>    [junit4]   2> 1455345 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrCore New index directory detected: old=null 
>> new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1/data/index/
>>    [junit4]   2> 1455345 WARN  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 
>> '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1/data/index'
>>  doesn't exist. Creating new index...
>>    [junit4]   2> 1455345 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1/data/index
>>    [junit4]   2> 1455345 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
>> org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
>> minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, 
>> maxMergeSizeForForcedMerge=9223372036854775807, 
>> calibrateSizeByDeletes=false, maxMergeDocs=2147483647, 
>> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6594206971437206]
>>    [junit4]   2> 1455346 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: 
>> commits: num=1
>>    [junit4]   2>        
>> commit{dir=MockDirectoryWrapper(RAMDirectory@35ba59b0 
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@68ae9afc),segFN=segments_1,generation=1}
>>    [junit4]   2> 1455346 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>>    [junit4]   2> 1455347 INFO  
>> (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
>> Looking for old index directories to cleanup for core collection1 in 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1/data/
>>    [junit4]   2> 1455347 WARN  
>> (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.DirectoryFactory 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/control-001/cores/collection1/data/
>>  does not point to a valid data directory; skipping clean-up of old index 
>> directories.
>>    [junit4]   2> 1455350 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
>> updateRequestProcessorChain "nodistrib"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
>> updateRequestProcessorChain "dedupe"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
>> updateRequestProcessorChain "stored_sig"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain 
>> "stored_sig"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
>> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
>> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain 
>> "distrib-dup-test-chain-implicit"
>>    [junit4]   2> 1455351 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as 
>> default, creating implicit default
>>    [junit4]   2> 1455352 WARN  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler 
>> {type = requestHandler,name = /dump,class = DumpRequestHandler,args = 
>> {defaults={a=A,b=B}}}
>>    [junit4]   2> 1455357 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>    [junit4]   2> 1455358 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>    [junit4]   2> 1455359 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>    [junit4]   2> 1455361 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>    [junit4]   2> 1455365 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.RequestHandlers Registered paths: 
>> /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
>>    [junit4]   2> 1455366 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrCore Using default statsCache cache: 
>> org.apache.solr.search.stats.LocalStatsCache
>>    [junit4]   2> 1455367 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: 
>> org.apache.solr.update.UpdateLog
>>    [junit4]   2> 1455367 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= 
>> defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
>> numVersionBuckets=65536
>>    [junit4]   2> 1455369 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
>>    [junit4]   2> 1455369 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
>>    [junit4]   2> 1455370 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
>> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
>> maxMergeAtOnce=38, maxMergeAtOnceExplicit=43, 
>> maxMergedSegmentMB=68.01171875, floorSegmentMB=2.03515625, 
>> forceMergeDeletesPctAllowed=17.75152513275686, segmentsPerTier=50.0, 
>> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8976554323863468
>>    [junit4]   2> 1455370 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: 
>> commits: num=1
>>    [junit4]   2>        
>> commit{dir=MockDirectoryWrapper(RAMDirectory@35ba59b0 
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@68ae9afc),segFN=segments_1,generation=1}
>>    [junit4]   2> 1455370 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>>    [junit4]   2> 1455370 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.s.SolrIndexSearcher Opening 
>> [Searcher@15361d70[collection1] main]
>>    [junit4]   2> 1455370 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: 
>> [/collections/control_collection]
>>    [junit4]   2> 1455372 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.c.ZkStateReader 
>> path=[/collections/control_collection] [configName]=[conf1] specified config 
>> exists in ZooKeeper
>>    [junit4]   2> 1455372 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based 
>> storage for the RestManager with znodeBase: /configs/conf1
>>    [junit4]   2> 1455373 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO 
>> with znodeBase: /configs/conf1
>>    [junit4]   2> 1455373 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
>>    [junit4]   2> 1455373 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json 
>> using ZooKeeperStorageIO:path=/configs/conf1
>>    [junit4]   2> 1455374 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode 
>> /configs/conf1/_rest_managed.json
>>    [junit4]   2> 1455374 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path 
>> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>>    [junit4]   2> 1455374 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
>>    [junit4]   2> 1455374 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
>>    [junit4]   2> 1455375 INFO  
>> (searcherExecutor-4182-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
>> [collection1] Registered new searcher Searcher@15361d70[collection1] 
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>    [junit4]   2> 1455375 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to 
>> seed version buckets
>>    [junit4]   2> 1455375 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 
>> 65536 version buckets from index
>>    [junit4]   2> 1455376 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed 
>> version bucket highest value from index
>>    [junit4]   2> 1455376 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.UpdateLog Could not find max version in index or 
>> recent updates, using new clock 1525613734201917440
>>    [junit4]   2> 1455378 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with 
>> highest version 1525613734201917440
>>    [junit4]   2> 1455378 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
>>    [junit4]   2> 1455379 INFO  
>> (coreLoadExecutor-4181-thread-1-processing-n:127.0.0.1:51020_) 
>> [n:127.0.0.1:51020_ c:control_collection s:shard1 r:core_node1 
>> x:collection1] o.a.s.c.CoreContainer registering core: collection1
>>    [junit4]   2> 1455380 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ZkController Register replica - core:collection1 
>> address:http://127.0.0.1:51020 collection:control_collection shard:shard1
>>    [junit4]   2> 1455383 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.c.SolrZkClient makePath: 
>> /collections/control_collection/leader_elect/shard1/election
>>    [junit4]   2> 1455390 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.LeaderElector Joined leadership election with path: 
>> /collections/control_collection/leader_elect/shard1/election/95350858267951108-core_node1-n_0000000000
>>    [junit4]   2> 1455391 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ShardLeaderElectionContext Running the leader process for 
>> shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
>>    [junit4]   2> 1455394 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>    [junit4]   2> 1455394 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
>>    [junit4]   2> 1455394 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:51020/collection1/
>>    [junit4]   2> 1455394 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>    [junit4]   2> 1455394 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.SyncStrategy http://127.0.0.1:51020/collection1/ has no replicas
>>    [junit4]   2> 1455395 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
>> message = {
>>    [junit4]   2>   "operation":"leader",
>>    [junit4]   2>   "shard":"shard1",
>>    [junit4]   2>   "collection":"control_collection"} current state version: 
>> 1
>>    [junit4]   2> 1455396 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.c.SolrZkClient makePath: 
>> /collections/control_collection/leaders/shard1
>>    [junit4]   2> 1455402 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
>>    [junit4]   2> 1455406 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ShardLeaderElectionContext I am the new leader: 
>> http://127.0.0.1:51020/collection1/ shard1
>>    [junit4]   2> 1455408 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
>> message = {
>>    [junit4]   2>   "operation":"leader",
>>    [junit4]   2>   "shard":"shard1",
>>    [junit4]   2>   "collection":"control_collection",
>>    [junit4]   2>   "base_url":"http://127.0.0.1:51020";,
>>    [junit4]   2>   "core":"collection1",
>>    [junit4]   2>   "state":"active"} current state version: 1
>>    [junit4]   2> 1455513 INFO  
>> (zkCallback-1542-thread-1-processing-n:127.0.0.1:51020_) [n:127.0.0.1:51020_ 
>>    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
>> state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has 
>> occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 1455562 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ZkController We are http://127.0.0.1:51020/collection1/ and leader 
>> is http://127.0.0.1:51020/collection1/
>>    [junit4]   2> 1455562 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ZkController No LogReplay needed for core=collection1 
>> baseURL=http://127.0.0.1:51020
>>    [junit4]   2> 1455562 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ZkController I am the leader, no recovery necessary
>>    [junit4]   2> 1455562 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ZkController publishing state=active
>>    [junit4]   2> 1455562 INFO  
>> (coreZkRegister-4175-thread-1-processing-n:127.0.0.1:51020_ x:collection1 
>> s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51020_ 
>> c:control_collection s:shard1 r:core_node1 x:collection1] 
>> o.a.s.c.ZkController numShards not found on descriptor - reading it from 
>> system property
>>    [junit4]   2> 1455565 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
>> message = {
>>    [junit4]   2>   "core":"collection1",
>>    [junit4]   2>   "core_node_name":"core_node1",
>>    [junit4]   2>   "roles":null,
>>    [junit4]   2>   "base_url":"http://127.0.0.1:51020";,
>>    [junit4]   2>   "node_name":"127.0.0.1:51020_",
>>    [junit4]   2>   "numShards":"1",
>>    [junit4]   2>   "state":"active",
>>    [junit4]   2>   "shard":"shard1",
>>    [junit4]   2>   "collection":"control_collection",
>>    [junit4]   2>   "operation":"state"} current state version: 2
>>    [junit4]   2> 1455566 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
>> message={
>>    [junit4]   2>   "core":"collection1",
>>    [junit4]   2>   "core_node_name":"core_node1",
>>    [junit4]   2>   "roles":null,
>>    [junit4]   2>   "base_url":"http://127.0.0.1:51020";,
>>    [junit4]   2>   "node_name":"127.0.0.1:51020_",
>>    [junit4]   2>   "numShards":"1",
>>    [junit4]   2>   "state":"active",
>>    [junit4]   2>   "shard":"shard1",
>>    [junit4]   2>   "collection":"control_collection",
>>    [junit4]   2>   "operation":"state"}
>>    [junit4]   2> 1455619 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>    [junit4]   2> 1455620 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>    [junit4]   2> 1455622 INFO  (zkCallback-1544-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@716e17b9 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017/solr got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1455622 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>    [junit4]   2> 1455622 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>    [junit4]   2> 1455622 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
>>    [junit4]   2> 1455628 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
>> loss:false
>>    [junit4]   2> 1455675 INFO  
>> (zkCallback-1542-thread-1-processing-n:127.0.0.1:51020_) [n:127.0.0.1:51020_ 
>>    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
>> state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has 
>> occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 1455683 INFO  (zkCallback-1544-thread-1) [    ] 
>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
>> state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has 
>> occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 1455937 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.SolrTestCaseJ4 Writing core.properties file to 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1
>>    [junit4]   2> 1455939 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001
>>    [junit4]   2> 1455943 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.Server jetty-9.2.13.v20150730
>>    [junit4]   2> 1455948 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.h.ContextHandler Started 
>> o.e.j.s.ServletContextHandler@5f3519d9{/,null,AVAILABLE}
>>    [junit4]   2> 1455949 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.ServerConnector Started 
>> ServerConnector@5647bcab{HTTP/1.1}{127.0.0.1:51026}
>>    [junit4]   2> 1455949 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.e.j.s.Server Started @1459298ms
>>    [junit4]   2> 1455950 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.s.e.JettySolrRunner Jetty properties: 
>> {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/tempDir-001/jetty1,
>>  solrconfig=solrconfig.xml, hostContext=/, hostPort=51026, 
>> coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores}
>>    [junit4]   2> 1455950 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
>> sun.misc.Launcher$AppClassLoader@1540e19d
>>    [junit4]   2> 1455950 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
>> '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001'
>>    [junit4]   2> 1455950 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>    [junit4]   2> 1455950 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
>> system property or JNDI)
>>    [junit4]   2> 1455951 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>    [junit4]   2> 1455953 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>    [junit4]   2> 1455954 INFO  (zkCallback-1545-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@28e3b058 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017/solr got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1455954 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>    [junit4]   2> 1455955 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>    [junit4]   2> 1455956 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
>> ZooKeeper)
>>    [junit4]   2> 1455956 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.SolrXmlConfig Loading container configuration from 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/solr.xml
>>    [junit4]   2> 1455967 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CorePropertiesLocator Config-defined core root directory: 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores
>>    [junit4]   2> 1455967 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CoreContainer New CoreContainer 1744286484
>>    [junit4]   2> 1455967 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CoreContainer Loading cores into CoreContainer 
>> [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001]
>>    [junit4]   2> 1455967 WARN  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.CoreContainer Couldn't add files from 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/lib
>>  to classpath: 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/lib
>>    [junit4]   2> 1455968 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
>> 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
>> 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
>> 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
>> false,useRetries : false,
>>    [junit4]   2> 1455971 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
>> params: socketTimeout=340000&connTimeout=45000&retry=true
>>    [junit4]   2> 1455971 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>    [junit4]   2> 1455971 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.l.LogWatcher Registering Log Listener [Log4j 
>> (org.slf4j.impl.Log4jLoggerFactory)]
>>    [junit4]   2> 1455972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51017/solr
>>    [junit4]   2> 1455972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.ZkController zkHost includes chroot
>>    [junit4]   2> 1455972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>>    [junit4]   2> 1455972 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>>    [junit4]   2> 1455974 INFO  (zkCallback-1548-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@481162a6 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017 got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1455977 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>>    [junit4]   2> 1455977 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) [    ] 
>> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>>    [junit4]   2> 1455981 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
>> connect to ZooKeeper
>>    [junit4]   2> 1455984 INFO  
>> (zkCallback-1549-thread-1-processing-n:127.0.0.1:51026_) [n:127.0.0.1:51026_ 
>>    ] o.a.s.c.c.ConnectionManager Watcher 
>> org.apache.solr.common.cloud.ConnectionManager@3ba13230 
>> name:ZooKeeperConnection Watcher:127.0.0.1:51017/solr got event WatchedEvent 
>> state:SyncConnected type:None path:null path:null type:None
>>    [junit4]   2> 1455984 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.c.ConnectionManager Client is connected to 
>> ZooKeeper
>>    [junit4]   2> 1456000 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
>> ZooKeeper...
>>    [junit4]   2> 1457012 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.ZkController Register node as live in 
>> ZooKeeper:/live_nodes/127.0.0.1:51026_
>>    [junit4]   2> 1457013 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.c.SolrZkClient makePath: 
>> /live_nodes/127.0.0.1:51026_
>>    [junit4]   2> 1457014 INFO  
>> (zkCallback-1549-thread-1-processing-n:127.0.0.1:51026_) [n:127.0.0.1:51026_ 
>>    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent 
>> state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred 
>> - updating... (live nodes size: [1])
>>    [junit4]   2> 1457015 INFO  (zkCallback-1544-thread-1) [    ] 
>> o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent 
>> state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred 
>> - updating... (live nodes size: [1])
>>    [junit4]   2> 1457015 INFO  
>> (zkCallback-1542-thread-1-processing-n:127.0.0.1:51020_) [n:127.0.0.1:51020_ 
>>    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent 
>> state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred 
>> - updating... (live nodes size: [1])
>>    [junit4]   2> 1457023 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.Overseer Overseer (id=null) closing
>>    [junit4]   2> 1457024 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.LeaderElector Joined leadership election 
>> with path: 
>> /overseer_elect/election/95350858267951112-127.0.0.1:51026_-n_0000000001
>>    [junit4]   2> 1457024 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.LeaderElector Watching path 
>> /overseer_elect/election/95350858267951108-127.0.0.1:51020_-n_0000000000 to 
>> know if I could be the leader
>>    [junit4]   2> 1457031 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
>> Skipping setup for authorization module.
>>    [junit4]   2> 1457031 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.CoreContainer No authentication plugin used.
>>    [junit4]   2> 1457032 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.CorePropertiesLocator Looking for core 
>> definitions underneath 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores
>>    [junit4]   2> 1457033 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: 
>> {name=collection1, config=solrconfig.xml, transient=false, 
>> schema=schema.xml, loadOnStartup=true, 
>> configSetProperties=configsetprops.json, collection=collection1, 
>> coreNodeName=, dataDir=data/, shard=}
>>    [junit4]   2> 1457033 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.CorePropertiesLocator Found core 
>> collection1 in 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1
>>    [junit4]   2> 1457033 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.c.CorePropertiesLocator Found 1 core 
>> definitions
>>    [junit4]   2> 1457035 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.s.SolrDispatchFilter 
>> user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
>>    [junit4]   2> 1457035 INFO  
>> (TEST-BasicDistributedZkTest.test-seed#[BA21C32BCEFAAE7F]) 
>> [n:127.0.0.1:51026_    ] o.a.s.s.SolrDispatchFilter 
>> SolrDispatchFilter.init() done
>>    [junit4]   2> 1457036 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.ZkController 
>> publishing state=down
>>    [junit4]   2> 1457036 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.ZkController 
>> numShards not found on descriptor - reading it from system property
>>    [junit4]   2> 1457038 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
>> message = {
>>    [junit4]   2>   "core":"collection1",
>>    [junit4]   2>   "roles":null,
>>    [junit4]   2>   "base_url":"http://127.0.0.1:51026";,
>>    [junit4]   2>   "node_name":"127.0.0.1:51026_",
>>    [junit4]   2>   "numShards":"2",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "shard":null,
>>    [junit4]   2>   "collection":"collection1",
>>    [junit4]   2>   "operation":"state"} current state version: 3
>>    [junit4]   2> 1457039 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 
>> message={
>>    [junit4]   2>   "core":"collection1",
>>    [junit4]   2>   "roles":null,
>>    [junit4]   2>   "base_url":"http://127.0.0.1:51026";,
>>    [junit4]   2>   "node_name":"127.0.0.1:51026_",
>>    [junit4]   2>   "numShards":"2",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "shard":null,
>>    [junit4]   2>   "collection":"collection1",
>>    [junit4]   2>   "operation":"state"}
>>    [junit4]   2> 1457039 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
>> collection1
>>    [junit4]   2> 1457039 INFO  
>> (OverseerStateUpdate-95350858267951108-127.0.0.1:51020_-n_0000000000) 
>> [n:127.0.0.1:51020_    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
>> shard shard=shard2
>>    [junit4]   2> 1457039 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.ZkController look 
>> for our core node name
>>    [junit4]   2> 1457143 INFO  (zkCallback-1544-thread-1) [    ] 
>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
>> state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has 
>> occurred - updating... (live nodes size: [2])
>>    [junit4]   2> 1457143 INFO  
>> (zkCallback-1542-thread-1-processing-n:127.0.0.1:51020_) [n:127.0.0.1:51020_ 
>>    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
>> state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has 
>> occurred - updating... (live nodes size: [2])
>>    [junit4]   2> 1457143 INFO  
>> (zkCallback-1549-thread-1-processing-n:127.0.0.1:51026_) [n:127.0.0.1:51026_ 
>>    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
>> state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has 
>> occurred - updating... (live nodes size: [2])
>>    [junit4]   2> 1458041 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.ZkController 
>> waiting to find shard id in clusterstate for collection1
>>    [junit4]   2> 1458042 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.ZkController 
>> Check for collection zkNode:collection1
>>    [junit4]   2> 1458042 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.ZkController 
>> Collection zkNode exists
>>    [junit4]   2> 1458042 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
>> Load collection config from: [/collections/collection1]
>>    [junit4]   2> 1458044 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
>> path=[/collections/collection1] [configName]=[conf1] specified config exists 
>> in ZooKeeper
>>    [junit4]   2> 1458044 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
>> '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1'
>>    [junit4]   2> 1458044 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>>    [junit4]   2> 1458044 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
>> system property or JNDI)
>>    [junit4]   2> 1458046 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.Config loaded 
>> config solrconfig.xml with version 0
>>    [junit4]   2> 1458055 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.SolrConfig 
>> current version of requestparams : -1
>>    [junit4]   2> 1458078 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using 
>> Lucene MatchVersion: 6.0.0
>>    [junit4]   2> 1458089 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.SolrConfig Loaded 
>> SolrConfig: solrconfig.xml
>>    [junit4]   2> 1458095 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
>> [collection1] Schema name=test
>>    [junit4]   2> 1458189 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
>> rates=open-exchange-rates.json, refreshInterval=1440.
>>    [junit4]   2> 1458195 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
>> default search field in schema is text
>>    [junit4]   2> 1458196 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
>> unique key field: id
>>    [junit4]   2> 1458210 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file 
>> currency.xml
>>    [junit4]   2> 1458214 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file 
>> currency.xml
>>    [junit4]   2> 1458216 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
>> open-exchange-rates.json
>>    [junit4]   2> 1458218 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
>> open-exchange-rates.json
>>    [junit4]   2> 1458222 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] 
>> o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming 
>> default properties: Can't find resource 'configsetprops.json' in classpath 
>> or '/configs/conf1', 
>> cwd=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
>>    [junit4]   2> 1458222 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1   x:collection1] o.a.s.c.CoreContainer 
>> Creating SolrCore 'collection1' using configuration from collection 
>> collection1
>>    [junit4]   2> 1458222 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
>>    [junit4]   2> 1458222 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
>> [/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1],
>>  
>> dataDir=[/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1/data/]
>>    [junit4]   2> 1458222 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
>> Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76cd5d4e
>>    [junit4]   2> 1458223 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.CachingDirectoryFactory return new directory for 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1/data
>>    [junit4]   2> 1458223 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.SolrCore New index directory detected: old=null 
>> new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1/data/index/
>>    [junit4]   2> 1458223 WARN  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.SolrCore [collection1] Solr index directory 
>> '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1/data/index'
>>  doesn't exist. Creating new index...
>>    [junit4]   2> 1458223 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.CachingDirectoryFactory return new directory for 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cores/collection1/data/index
>>    [junit4]   2> 1458223 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
>> org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
>> minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, 
>> maxMergeSizeForForcedMerge=9223372036854775807, 
>> calibrateSizeByDeletes=false, maxMergeDocs=2147483647, 
>> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6594206971437206]
>>    [junit4]   2> 1458224 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
>>    [junit4]   2>        
>> commit{dir=MockDirectoryWrapper(RAMDirectory@4905dca1 
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1d06076b),segFN=segments_1,generation=1}
>>    [junit4]   2> 1458224 INFO  
>> (coreLoadExecutor-4192-thread-1-processing-n:127.0.0.1:51026_) 
>> [n:127.0.0.1:51026_ c:collection1 s:shard2 r:core_node1 x:collection1] 
>> o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>>    [junit4]   2> 1458224 INFO  
>> (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:51026_ 
>> c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking 
>> for old index directories to cleanup for core collection1 in 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_BA21C32BCEFAAE7F-001/shard-1-001/cor
>>
>> [...truncated too long message...]
>>
>> DirectUpdateHandler2{commits=0,autocommits=0,soft 
>> autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
>>    [junit4]   2> 1724373 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
>>    [junit4]   2> 1724373 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has 
>> reached 0 - closing IndexWriter
>>    [junit4]   2> 1724373 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with 
>> IndexWriterCloser
>>    [junit4]   2> 1724374 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on 
>> request.
>>    [junit4]   2> 1724374 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory 
>> - 2 directories currently being tracked
>>    [junit4]   2> 1724374 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.c.CachingDirectoryFactory looking to close 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_BA21C32BCEFAAE7F-001/solr-instance-027/./collection1/data/index
>>  
>> [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_BA21C32BCEFAAE7F-001/solr-instance-027/./collection1/data/index;done=false>>]
>>    [junit4]   2> 1724374 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_BA21C32BCEFAAE7F-001/solr-instance-027/./collection1/data/index
>>    [junit4]   2> 1724377 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.c.CachingDirectoryFactory looking to close 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_BA21C32BCEFAAE7F-001/solr-instance-027/./collection1/data
>>  
>> [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_BA21C32BCEFAAE7F-001/solr-instance-027/./collection1/data;done=false>>]
>>    [junit4]   2> 1724377 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: 
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_BA21C32BCEFAAE7F-001/solr-instance-027/./collection1/data
>>    [junit4]   2> 1724378 INFO  
>> (TEST-TestReplicationHandler.doTestDetails-seed#[BA21C32BCEFAAE7F]) [    
>> x:collection1] o.e.j.s.h.ContextHandler Stopped 
>> o.e.j.s.ServletContextHandler@74ea798b{/solr,null,UNAVAILABLE}
>>    [junit4]   2> 1724380 INFO  
>> (SUITE-TestReplicationHandler-seed#[BA21C32BCEFAAE7F]-worker) [    ] 
>> o.a.s.SolrTestCaseJ4 ###deleteCore
>>    [junit4]   2> ObjectTracker found 1 object(s) that were not released!!! 
>> [NRTCachingDirectory]
>>    [junit4]   2> 
>> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
>>    [junit4]   2>        at 
>> org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
>>    [junit4]   2>        at 
>> org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
>>    [junit4]   2>        at 
>> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
>>    [junit4]   2>        at 
>> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
>>    [junit4]   2>        at 
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
>>    [junit4]   2>        at 
>> org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
>>    [junit4]   2>
>>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): 
>> {name=FSTOrd50, id=FSTOrd50, newname=FSTOrd50}, docValues:{}, 
>> sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=de, timezone=BET
>>    [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_72 
>> (64-bit)/cpus=3,threads=1,free=172333824,total=469237760
>>    [junit4]   2> NOTE: All tests run in this JVM: [DistribCursorPagingTest, 
>> RuleEngineTest, DistribJoinFromCollectionTest, TestReloadAndDeleteDocs, 
>> TestSearcherReuse, CSVRequestHandlerTest, TestAtomicUpdateErrorCases, 
>> TestComplexPhraseQParserPlugin, SignatureUpdateProcessorFactoryTest, 
>> TestRestoreCore, TestLeaderElectionZkExpiry, 
>> TestPerFieldSimilarityWithDefaultOverride, LeaderElectionIntegrationTest, 
>> TestCrossCoreJoin, HdfsLockFactoryTest, TestCoreDiscovery, 
>> DistributedVersionInfoTest, TestIndexingPerformance, CloudMLTQParserTest, 
>> TestBlobHandler, TestTrieFacet, TestRequestStatusCollectionAPI, 
>> TermVectorComponentDistributedTest, BlockCacheTest, 
>> TestFoldingMultitermQuery, TestPKIAuthenticationPlugin, LoggingHandlerTest, 
>> TestRecovery, HdfsUnloadDistributedZkTest, DateRangeFieldTest, 
>> StatsComponentTest, ResponseLogComponentTest, TestReload, PluginInfoTest, 
>> TestCloudManagedSchemaConcurrent, TestHashQParserPlugin, 
>> ReplicationFactorTest, TestCharFilters, 
>> OverseerCollectionConfigSetProcessorTest, BasicZkTest, 
>> TestManagedStopFilterFactory, CdcrVersionReplicationTest, 
>> BasicDistributedZk2Test, TestQueryTypes, OpenCloseCoreStressTest, 
>> DateFieldTest, TestScoreJoinQPScore, TestCollectionAPI, 
>> TestSchemaSimilarityResource, CleanupOldIndexTest, 
>> DistributedDebugComponentTest, QueryResultKeyTest, SuggesterWFSTTest, 
>> MultiTermTest, ShowFileRequestHandlerTest, TestFastOutputStream, 
>> HdfsThreadLeakTest, TestManagedSchemaFieldTypeResource, VersionInfoTest, 
>> BadIndexSchemaTest, TestConfigSetImmutable, FullSolrCloudDistribCmdsTest, 
>> FacetPivotSmallTest, TestManagedResourceStorage, 
>> ConcurrentDeleteAndCreateCollectionTest, SearchHandlerTest, TestFastWriter, 
>> TestRandomFaceting, TriLevelCompositeIdRoutingTest, ConvertedLegacyTest, 
>> TestSolrJ, TestFuzzyAnalyzedSuggestions, TestZkChroot, 
>> FileBasedSpellCheckerTest, OutOfBoxZkACLAndCredentialsProvidersTest, 
>> TestImplicitCoreProperties, TestRawTransformer, RecoveryAfterSoftCommitTest, 
>> SharedFSAutoReplicaFailoverTest, DistributedFacetPivotLargeTest, 
>> TestRealTimeGet, SuggesterTest, BufferStoreTest, 
>> TestFieldTypeCollectionResource, TestFieldSortValues, TestQueryUtils, 
>> TestStandardQParsers, LeaderInitiatedRecoveryOnCommitTest, OutputWriterTest, 
>> TestSystemIdResolver, TestSchemaNameResource, GraphQueryTest, 
>> TestNoOpRegenerator, TestReversedWildcardFilterFactory, AtomicUpdatesTest, 
>> CacheHeaderTest, TestInitParams, NoCacheHeaderTest, 
>> CoreMergeIndexesAdminHandlerTest, DistanceFunctionTest, TestBulkSchemaAPI, 
>> BJQParserTest, MergeStrategyTest, ConnectionManagerTest, 
>> TestNonDefinedSimilarityFactory, TestNamedUpdateProcessors, 
>> TestSolrCloudWithKerberosAlt, EchoParamsTest, 
>> ConjunctionSolrSpellCheckerTest, TestExactSharedStatsCache, 
>> CurrencyFieldOpenExchangeTest, SimpleCollectionCreateDeleteTest, TestJoin, 
>> TestPivotHelperCode, OpenExchangeRatesOrgProviderTest, TestJettySolrRunner, 
>> TestConfigSetsAPIExclusivity, MigrateRouteKeyTest, 
>> ExitableDirectoryReaderTest, TestCopyFieldCollectionResource, TestMacros, 
>> TestQueryWrapperFilter, BasicFunctionalityTest, TestSchemaManager, 
>> TestRTimerTree, SolrIndexSplitterTest, NumericFieldsTest, 
>> NotRequiredUniqueKeyTest, TestRecoveryHdfs, AliasIntegrationTest, 
>> CollectionsAPIDistributedZkTest, AutoCommitTest, TestConfigSets, 
>> DistributedIntervalFacetingTest, DocExpirationUpdateProcessorFactoryTest, 
>> HdfsChaosMonkeyNothingIsSafeTest, TermVectorComponentTest, 
>> TestStressRecovery, TestSmileRequest, TestReplicationHandler]
>>    [junit4]   2> NOTE: reproduce with: ant test  
>> -Dtestcase=TestReplicationHandler -Dtests.seed=BA21C32BCEFAAE7F 
>> -Dtests.slow=true -Dtests.locale=de -Dtests.timezone=BET 
>> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>>    [junit4] ERROR   0.00s J1 | TestReplicationHandler (suite) <<<
>>    [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 
>> 1 object(s) that were not released!!! [NRTCachingDirectory]
>>    [junit4]    >        at 
>> __randomizedtesting.SeedInfo.seed([BA21C32BCEFAAE7F]:0)
>>    [junit4]    >        at 
>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:228)
>>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>>    [junit4] Completed [297/577 (2!)] on J1 in 81.67s, 13 tests, 1 failure 
>> <<< FAILURES!
>>
>> [...truncated 900 lines...]
>> BUILD FAILED
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:740: The 
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:684: The 
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:59: The 
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:233: The 
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:524: 
>> The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1457:
>>  The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1014:
>>  There were test failures: 577 suites (8 ignored), 2318 tests, 1 suite-level 
>> error, 1 failure, 121 ignored (71 assumptions) [seed: BA21C32BCEFAAE7F]
>>
>> Total time: 71 minutes 14 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> [WARNINGS] Skipping publisher since build result is FAILURE
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to