Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/344/

7 tests failed.
FAILED:  
org.apache.lucene.search.TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager

Error Message:
Captured an uncaught exception in thread: Thread[id=7461, name=Thread-6579, 
state=RUNNABLE, group=TGRP-TestSearcherManager]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=7461, name=Thread-6579, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm:
 Too many open files
        at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
        at 
org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:661)
Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm:
 Too many open files
        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:335)
        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
        at 
org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
        at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
        at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
        at 
org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108)
        at 
org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76)
        at 
org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:649)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager

Error Message:
Captured an uncaught exception in thread: Thread[id=7486, name=Lucene Merge 
Thread #2, state=RUNNABLE, group=TGRP-TestSearcherManager]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=7486, name=Lucene Merge Thread #2, 
state=RUNNABLE, group=TGRP-TestSearcherManager]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot 
merge
        at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable 
error; cannot merge
        at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4088)
        at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4068)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3922)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:335)
        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
        ... 2 more


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager

Error Message:
Captured an uncaught exception in thread: Thread[id=7484, name=Lucene Merge 
Thread #0, state=RUNNABLE, group=TGRP-TestSearcherManager]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=7484, name=Lucene Merge Thread #0, 
state=RUNNABLE, group=TGRP-TestSearcherManager]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: 
java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
        at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:335)
        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager

Error Message:
Captured an uncaught exception in thread: Thread[id=7458, name=Thread-6576, 
state=RUNNABLE, group=TGRP-TestSearcherManager]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=7458, name=Thread-6576, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is 
closed
        at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
        at 
org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:726)
        at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4582)
        at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4572)
        at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4559)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:517)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083)
        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005)
        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316)
        at 
org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:559)
Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:335)
        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)


FAILED:  
org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at 
__randomizedtesting.SeedInfo.seed([3A47B4CBBFB61A9B:4EB7558EFB925D14]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest(TestCloudRecovery.java:105)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 1884 lines...]
   [junit4] Suite: org.apache.lucene.search.TestSearcherManager
   [junit4]   2> Beal 08, 2017 9:18:38 A.M. 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: 
Thread[Thread-6579,5,TGRP-TestSearcherManager]
   [junit4]   2> java.lang.RuntimeException: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm:
 Too many open files
   [junit4]   2>        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]   2>        at 
org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:661)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm:
 Too many open files
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]   2>        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]   2>        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   2>        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]   2>        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
   [junit4]   2>        at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
   [junit4]   2>        at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
   [junit4]   2>        at 
org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108)
   [junit4]   2>        at 
org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76)
   [junit4]   2>        at 
org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:649)
   [junit4]   2> 
   [junit4]   2> Beal 08, 2017 9:18:38 A.M. 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge 
Thread #2,5,TGRP-TestSearcherManager]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot 
merge
   [junit4]   2>        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
   [junit4]   2> Caused by: java.lang.IllegalStateException: this writer hit an 
unrecoverable error; cannot merge
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4088)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4068)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3922)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]   2>        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]   2>        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   2>        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]   2>        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]   2>        ... 2 more
   [junit4]   2> 
   [junit4]   2> Beal 08, 2017 9:18:38 A.M. 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge 
Thread #0,5,TGRP-TestSearcherManager]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: 
java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]   2>        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]   2>        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]   2>        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   2>        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]   2>        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
   [junit4]   2> 
   [junit4]   2> Beal 08, 2017 9:18:38 A.M. 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: 
Thread[Thread-6576,5,TGRP-TestSearcherManager]
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: this 
IndexWriter is closed
   [junit4]   2>        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:726)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4582)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4572)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4559)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:517)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005)
   [junit4]   2>        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316)
   [junit4]   2>        at 
org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:559)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]   2>        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]   2>        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]   2>        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]   2>        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   2>        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]   2>        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]   2>        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]   2>        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]   2>        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]   2>        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
   [junit4]   2>        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
   [junit4]   2> 
   [junit4]   2> Beal 08, 2017 11:12:26 A.M. 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: 
org.apache.lucene.search.TestSearcherManager
   [junit4]   2>    1) Thread[id=6426, 
name=TEST-TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh-seed#[F70A69FB1A02E070],
 state=WAITING, group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh(TestSearcherManager.java:672)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    2) Thread[id=7460, name=Thread-6578, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Throwable.fillInStackTrace(Native Method)
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
   [junit4]   2>         at java.lang.Throwable.<init>(Throwable.java:265)
   [junit4]   2>         at java.lang.Exception.<init>(Exception.java:66)
   [junit4]   2>         at 
java.lang.RuntimeException.<init>(RuntimeException.java:62)
   [junit4]   2>         at 
java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
   [junit4]   2>         at 
org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.ensureOpen(ReferenceManager.java:54)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:248)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:619)
   [junit4]   2>    3) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    4) Thread[id=9, name=JUnit4-serializer-daemon, 
state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    5) Thread[id=7459, name=Thread-6577, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Throwable.fillInStackTrace(Native Method)
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
   [junit4]   2>         at java.lang.Throwable.<init>(Throwable.java:265)
   [junit4]   2>         at java.lang.Exception.<init>(Exception.java:66)
   [junit4]   2>         at 
java.lang.RuntimeException.<init>(RuntimeException.java:62)
   [junit4]   2>         at 
java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
   [junit4]   2>         at 
org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager$9.run(TestSearcherManager.java:590)
   [junit4]   2>    6) Thread[id=6425, 
name=SUITE-TestSearcherManager-seed#[F70A69FB1A02E070], state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1556)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSearcherManager 
-Dtests.method=testConcurrentIndexCloseSearchAndRefresh 
-Dtests.seed=F70A69FB1A02E070 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=ga-IE -Dtests.timezone=Asia/Muscat -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   6866s J2 | 
TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because 
suite timeout was reached.
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]   2> Beal 08, 2017 11:12:26 A.M. 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked 
thread(s).
   [junit4]   2> Beal 08, 2017 11:12:46 A.M. 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 3 threads leaked from SUITE scope at 
org.apache.lucene.search.TestSearcherManager: 
   [junit4]   2>    1) Thread[id=6426, 
name=TEST-TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh-seed#[F70A69FB1A02E070],
 state=WAITING, group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh(TestSearcherManager.java:672)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    2) Thread[id=7460, name=Thread-6578, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Throwable.fillInStackTrace(Native Method)
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
   [junit4]   2>         at java.lang.Throwable.<init>(Throwable.java:265)
   [junit4]   2>         at java.lang.Exception.<init>(Exception.java:66)
   [junit4]   2>         at 
java.lang.RuntimeException.<init>(RuntimeException.java:62)
   [junit4]   2>         at 
java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
   [junit4]   2>         at 
org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.ensureOpen(ReferenceManager.java:54)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:248)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:619)
   [junit4]   2>    3) Thread[id=7459, name=Thread-6577, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Throwable.fillInStackTrace(Native Method)
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
   [junit4]   2>         at java.lang.Throwable.<init>(Throwable.java:265)
   [junit4]   2>         at java.lang.Exception.<init>(Exception.java:66)
   [junit4]   2>         at 
java.lang.RuntimeException.<init>(RuntimeException.java:62)
   [junit4]   2>         at 
java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
   [junit4]   2>         at 
org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager$9.run(TestSearcherManager.java:590)
   [junit4]   2> Beal 08, 2017 11:12:46 A.M. 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=6426, 
name=TEST-TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh-seed#[F70A69FB1A02E070],
 state=WAITING, group=TGRP-TestSearcherManager]
   [junit4]   2>    2) Thread[id=7460, name=Thread-6578, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>    3) Thread[id=7459, name=Thread-6577, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001
   [junit4]   2> May 08, 2017 7:12:49 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be 
terminated:
   [junit4]   2>    1) Thread[id=7460, name=Thread-6578, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Throwable.fillInStackTrace(Native Method)
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
   [junit4]   2>         at java.lang.Throwable.<init>(Throwable.java:265)
   [junit4]   2>         at java.lang.Exception.<init>(Exception.java:66)
   [junit4]   2>         at 
java.lang.RuntimeException.<init>(RuntimeException.java:62)
   [junit4]   2>         at 
java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
   [junit4]   2>         at 
org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.ensureOpen(ReferenceManager.java:54)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:248)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:619)
   [junit4]   2>    2) Thread[id=7459, name=Thread-6577, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]   2>         at java.lang.Throwable.fillInStackTrace(Native Method)
   [junit4]   2>         at 
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
   [junit4]   2>         at java.lang.Throwable.<init>(Throwable.java:265)
   [junit4]   2>         at java.lang.Exception.<init>(Exception.java:66)
   [junit4]   2>         at 
java.lang.RuntimeException.<init>(RuntimeException.java:62)
   [junit4]   2>         at 
java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
   [junit4]   2>         at 
org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26)
   [junit4]   2>         at 
org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98)
   [junit4]   2>         at 
org.apache.lucene.search.TestSearcherManager$9.run(TestSearcherManager.java:590)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{date=BlockTreeOrds(blocksize=128), extra35=BlockTreeOrds(blocksize=128), 
extra13=BlockTreeOrds(blocksize=128), 
extra36=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra14=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra33=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra11=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra8=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra12=PostingsFormat(name=LuceneFixedGap), 
extra9=PostingsFormat(name=LuceneFixedGap), 
extra34=PostingsFormat(name=LuceneFixedGap), 
extra39=BlockTreeOrds(blocksize=128), extra17=BlockTreeOrds(blocksize=128), 
extra18=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra15=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra37=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra38=PostingsFormat(name=LuceneFixedGap), 
extra16=PostingsFormat(name=LuceneFixedGap), 
extra2=BlockTreeOrds(blocksize=128), 
extra3=PostingsFormat(name=LuceneVarGapFixedInterval), 
titleTokenized=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra0=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra19=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra1=PostingsFormat(name=LuceneFixedGap), body=BlockTreeOrds(blocksize=128), 
title=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra6=BlockTreeOrds(blocksize=128), 
extra7=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra4=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra5=PostingsFormat(name=LuceneFixedGap), 
packID=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra31=BlockTreeOrds(blocksize=128), 
extra32=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra10=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra30=PostingsFormat(name=LuceneFixedGap), 
extra24=BlockTreeOrds(blocksize=128), 
extra25=PostingsFormat(name=LuceneVarGapFixedInterval), 
extra22=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra23=PostingsFormat(name=LuceneFixedGap), 
extra28=BlockTreeOrds(blocksize=128), 
extra29=PostingsFormat(name=LuceneVarGapFixedInterval), 
docid=PostingsFormat(name=LuceneFixedGap), 
extra26=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
extra27=PostingsFormat(name=LuceneFixedGap), 
extra20=BlockTreeOrds(blocksize=128), 
extra21=PostingsFormat(name=LuceneVarGapFixedInterval)}, 
docValues:{date=DocValuesFormat(name=Lucene54), 
extra13=DocValuesFormat(name=Lucene54), extra14=DocValuesFormat(name=Lucene54), 
extra11=DocValuesFormat(name=Lucene54), extra8=DocValuesFormat(name=Lucene54), 
extra12=DocValuesFormat(name=Asserting), extra9=DocValuesFormat(name=Lucene54), 
extra17=DocValuesFormat(name=Asserting), 
extra18=DocValuesFormat(name=Lucene54), extra15=DocValuesFormat(name=Lucene54), 
extra16=DocValuesFormat(name=Lucene54), extra2=DocValuesFormat(name=Lucene54), 
extra3=DocValuesFormat(name=Asserting), 
titleTokenized=DocValuesFormat(name=Lucene54), 
extra0=DocValuesFormat(name=Asserting), 
extra19=DocValuesFormat(name=Asserting), extra1=DocValuesFormat(name=Lucene54), 
body=DocValuesFormat(name=Asserting), extra6=DocValuesFormat(name=Lucene54), 
extra7=DocValuesFormat(name=Lucene54), extra4=DocValuesFormat(name=Lucene54), 
extra5=DocValuesFormat(name=Asserting), extra10=DocValuesFormat(name=Lucene54), 
extra35=DocValuesFormat(name=Asserting), 
extra36=DocValuesFormat(name=Lucene54), 
docid_int=DocValuesFormat(name=Lucene54), 
extra33=DocValuesFormat(name=Lucene54), extra34=DocValuesFormat(name=Lucene54), 
extra39=DocValuesFormat(name=Lucene54), 
extra37=DocValuesFormat(name=Asserting), 
extra38=DocValuesFormat(name=Lucene54), title=DocValuesFormat(name=Lucene54), 
packID=DocValuesFormat(name=Lucene54), 
docid_intDV=DocValuesFormat(name=Lucene54), 
extra31=DocValuesFormat(name=Lucene54), 
extra32=DocValuesFormat(name=Asserting), 
extra30=DocValuesFormat(name=Lucene54), extra24=DocValuesFormat(name=Lucene54), 
extra25=DocValuesFormat(name=Lucene54), extra22=DocValuesFormat(name=Lucene54), 
extra23=DocValuesFormat(name=Asserting), 
extra28=DocValuesFormat(name=Asserting), 
extra29=DocValuesFormat(name=Lucene54), docid=DocValuesFormat(name=Asserting), 
extra26=DocValuesFormat(name=Lucene54), extra27=DocValuesFormat(name=Lucene54), 
extra20=DocValuesFormat(name=Asserting), 
extra21=DocValuesFormat(name=Lucene54), 
titleDV=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1762, 
maxMBSortInHeap=7.4180958464928235, 
sim=RandomSimilarity(queryNorm=true,coord=no): {extra35=DFR I(n)L1, extra13=DFR 
I(F)1, extra36=DFR I(ne)BZ(0.3), extra14=IB LL-LZ(0.3), extra33=DFR G3(800.0), 
extra11=DFI(Saturated), extra8=DFR I(ne)2, extra12=IB SPL-L3(800.0), extra9=DFR 
I(ne)L3(800.0), extra34=IB SPL-D2, extra39=DFR I(ne)1, extra17=DFR I(ne)B1, 
extra18=DFR I(F)L3(800.0), extra15=DFR I(F)L2, extra37=IB SPL-L1, extra38=DFR 
I(F)1, extra16=IB SPL-D3(800.0), extra2=IB SPL-DZ(0.3), extra3=DFR I(ne)Z(0.3), 
titleTokenized=DFR G2, extra0=IB LL-DZ(0.3), extra19=DFR I(ne)BZ(0.3), 
extra1=LM Jelinek-Mercer(0.700000), body=IB SPL-D3(800.0), 
extra6=DFI(Standardized), extra7=DFR I(n)B2, extra4=IB SPL-LZ(0.3), extra5=DFR 
I(n)1, extra31=DFR I(F)L3(800.0), extra32=DFR I(F)L2, extra10=IB LL-L3(800.0), 
extra30=DFR I(F)Z(0.3), extra24=DFR I(F)2, extra25=IB LL-L2, extra22=DFR 
I(n)B3(800.0), extra23=DFR I(n)3(800.0), extra28=DFR I(n)B1, extra29=DFR 
GBZ(0.3), extra26=IB LL-D1, extra27=DFR I(ne)B3(800.0), extra20=DFR I(n)Z(0.3), 
extra21=DFI(ChiSquared)}, locale=ga-IE, timezone=Asia/Muscat
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_121 (64-bit)/cpus=4,threads=3,free=116099416,total=354942976
   [junit4]   2> NOTE: All tests run in this JVM: [Test2BPoints, 
TestStringMSBRadixSorter, TestFrequencyTrackingRingBuffer, 
TestLevenshteinAutomata, MultiCollectorTest, TestDocInverterPerFieldErrorInfo, 
TestStressIndexing, TestSameTokenSamePosition, TestSpanMultiTermQueryWrapper, 
TestForUtil, TestRamUsageEstimator, TestSpansEnum, TestBoolean2, 
TestRoaringDocIdSet, TestTrackingDirectoryWrapper, TestTermVectorsWriter, 
TestSimilarity, TestFieldsReader, TestWildcardRandom, TestLucene53NormsFormat, 
TestBlendedTermQuery, TestGrowableByteArrayDataOutput, TestReqExclBulkScorer, 
TestAllFilesCheckIndexHeader, TestUTF32ToUTF8, TestSpanSearchEquivalence, 
TestSegmentTermEnum, TestBooleanQuery, TestControlledRealTimeReopenThread, 
TestVersion, TestTopFieldCollector, TestMergedIterator, TestSwappedIndexFiles, 
TestGeoUtils, TestBooleanMinShouldMatch, TestClassicSimilarity, 
TestMultiCollector, TestIndexWriterOnJRECrash, TestIndexWriterFromReader, 
TestSpanNotQuery, TestBytesStore, TestSnapshotDeletionPolicy, 
TestByteBlockPool, TestIndexWriterReader, TestIndexWriterMerging, 
TestDeterminizeLexicon, TestCodecs, TestIndexWriterDelete, 
TestPrefixInBooleanQuery, TestRecyclingIntBlockAllocator, 
TestPayloadsOnVectors, TestOperations, TestSloppyPhraseQuery2, 
TestNumericRangeQuery32, TestNativeFSLockFactory, TestBytesRefHash, 
TestBooleanScorer, TestCachingTokenFilter, TestSegmentReader, TestSetOnce, 
TestTermdocPerf, TestTragicIndexWriterDeadlock, TestPackedTokenAttributeImpl, 
TestFileSwitchDirectory, TestCompiledAutomaton, TestSmallFloat, 
TestSimpleExplanationsOfNonMatches, TestCollectionUtil, TestNeedsScores, 
FiniteStringsIteratorTest, TestFilterCodecReader, TestComplexExplanations, 
TestSegmentInfos, TestExceedMaxTermLength, TestAllFilesHaveCodecHeader, 
TestSearchWithThreads, TestIndexWriterWithThreads, TestGraphTokenizers, 
TestAddIndexes, TestShardSearching, TestMultiMMap, TestSearcherManager]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSearcherManager 
-Dtests.seed=F70A69FB1A02E070 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=ga-IE -Dtests.timezone=Asia/Muscat -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestSearcherManager (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=7461, name=Thread-6579, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]    > Caused by: java.lang.RuntimeException: 
java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm:
 Too many open files
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]    >        at 
org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:661)
   [junit4]    > Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm:
 Too many open files
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]    >        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]    >        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]    >        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]    >        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
   [junit4]    >        at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
   [junit4]    >        at 
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
   [junit4]    >        at 
org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108)
   [junit4]    >        at 
org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76)
   [junit4]    >        at 
org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:649)Throwable
 #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an 
uncaught exception in thread: Thread[id=7486, name=Lucene Merge Thread #2, 
state=RUNNABLE, group=TGRP-TestSearcherManager]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot 
merge
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
   [junit4]    > Caused by: java.lang.IllegalStateException: this writer hit an 
unrecoverable error; cannot merge
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4088)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4068)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3922)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
   [junit4]    > Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]    >        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]    >        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]    >        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]    >        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]    >        ... 2 moreThrowable #4: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=7484, name=Lucene Merge Thread #0, 
state=RUNNABLE, group=TGRP-TestSearcherManager]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: 
java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
   [junit4]    > Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]    >        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]    >        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]    >        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]    >        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)Throwable
 #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an 
uncaught exception in thread: Thread[id=7458, name=Thread-6576, state=RUNNABLE, 
group=TGRP-TestSearcherManager]
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: 
this IndexWriter is closed
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:726)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4582)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4572)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4559)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:517)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005)
   [junit4]    >        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316)
   [junit4]    >        at 
org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:559)
   [junit4]    > Caused by: java.nio.file.FileSystemException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm:
 Too many open files
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
   [junit4]    >        at 
org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
   [junit4]    >        at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]    >        at 
java.nio.channels.FileChannel.open(FileChannel.java:335)
   [junit4]    >        at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]    >        at 
org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749)
   [junit4]    >        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
   [junit4]    >        at 
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
   [junit4]    >        at 
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164)
   [junit4]    >        at 
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    >        at 
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
   [junit4]    >        at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
   [junit4] Completed [456/456 (1!)] on J2 in 7223.23s, 3 tests, 6 errors <<< 
FAILURES!

[...truncated 10408 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/init-core-data-001
   [junit4]   2> 771062 WARN  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 771062 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 771064 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 771065 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/tempDir-001
   [junit4]   2> 771065 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 771066 INFO  (Thread-2960) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 771066 INFO  (Thread-2960) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 771075 ERROR (Thread-2960) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 771166 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:47460
   [junit4]   2> 771183 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 771189 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 771192 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@670aecae{/solr,null,AVAILABLE}
   [junit4]   2> 771201 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@1959aef3{SSL,[ssl, 
http/1.1]}{127.0.0.1:47682}
   [junit4]   2> 771201 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.e.j.s.Server Started @774558ms
   [junit4]   2> 771201 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=47682}
   [junit4]   2> 771201 ERROR (jetty-launcher-31445-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 771201 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.6.0
   [junit4]   2> 771201 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 771201 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 771202 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-05-08T09:45:39.298Z
   [junit4]   2> 771203 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4120162e{/solr,null,AVAILABLE}
   [junit4]   2> 771205 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@3729b5bc{SSL,[ssl, 
http/1.1]}{127.0.0.1:41494}
   [junit4]   2> 771205 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.e.j.s.Server Started @774562ms
   [junit4]   2> 771205 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=41494}
   [junit4]   2> 771205 ERROR (jetty-launcher-31445-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 771205 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.6.0
   [junit4]   2> 771205 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 771205 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 771205 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-05-08T09:45:39.301Z
   [junit4]   2> 771215 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 771225 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 771227 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 771228 INFO  (jetty-launcher-31445-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47460/solr
   [junit4]   2> 771232 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 771236 INFO  (jetty-launcher-31445-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47460/solr
   [junit4]   2> 771254 INFO  (jetty-launcher-31445-thread-1) 
[n:127.0.0.1:47682_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 771258 INFO  (jetty-launcher-31445-thread-1) 
[n:127.0.0.1:47682_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:47682_solr
   [junit4]   2> 771259 INFO  (jetty-launcher-31445-thread-1) 
[n:127.0.0.1:47682_solr    ] o.a.s.c.Overseer Overseer 
(id=97926298937786372-127.0.0.1:47682_solr-n_0000000000) starting
   [junit4]   2> 771262 INFO  (jetty-launcher-31445-thread-2) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 771263 INFO  (jetty-launcher-31445-thread-2) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:41494_solr
   [junit4]   2> 771267 INFO  
(zkCallback-31453-thread-1-processing-n:127.0.0.1:47682_solr) 
[n:127.0.0.1:47682_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 771279 INFO  (jetty-launcher-31445-thread-1) 
[n:127.0.0.1:47682_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:47682_solr
   [junit4]   2> 771280 INFO  
(zkCallback-31458-thread-1-processing-n:127.0.0.1:41494_solr) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 771284 INFO  
(zkCallback-31453-thread-1-processing-n:127.0.0.1:47682_solr) 
[n:127.0.0.1:47682_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 771291 INFO  
(zkCallback-31458-thread-1-processing-n:127.0.0.1:41494_solr) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 771343 INFO  (jetty-launcher-31445-thread-2) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/tempDir-001/node2/.
   [junit4]   2> 771352 INFO  (jetty-launcher-31445-thread-1) 
[n:127.0.0.1:47682_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/tempDir-001/node1/.
   [junit4]   2> 771642 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]  

[...truncated too long message...]

41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.u.SolrIndexWriter Calling setCommitData 
with IW:org.apache.solr.update.SolrIndexWriter@56b698b1
   [junit4]   2> 902947 INFO  
(recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr 
x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@27f46dae[collection1_shard2_replica2] main]
   [junit4]   2> 902948 INFO  
(recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr 
x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 902948 INFO  
(searcherExecutor-6981-thread-1-processing-n:127.0.0.1:41494_solr 
x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.c.SolrCore [collection1_shard2_replica2] 
Registered new searcher Searcher@27f46dae[collection1_shard2_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(6.6.0):C2)))}
   [junit4]   2> 902948 INFO  
(recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr 
x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.u.p.LogUpdateProcessorFactory 
[collection1_shard2_replica2] {add=[2 (1566820847685992448), 3 
(1566820847693332480)]} 0 56
   [junit4]   2> 902948 WARN  
(recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr 
x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.u.UpdateLog Log replay finished. 
recoveryInfo=RecoveryInfo{adds=2 deletes=0 deleteByQuery=0 errors=0 
positionOfStart=0}
   [junit4]   2> 902948 INFO  
(coreZkRegister-6973-thread-2-processing-n:127.0.0.1:41494_solr 
x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.c.ZkController I am the leader, no 
recovery necessary
   [junit4]   2> 903053 INFO  
(zkCallback-31492-thread-1-processing-n:127.0.0.1:41494_solr) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [1])
   [junit4]   2> 1012812 INFO  
(TEST-TestCloudRecovery.leaderRecoverFromLogOnStartupTest-seed#[3A47B4CBBFB61A9B])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending leaderRecoverFromLogOnStartupTest
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCloudRecovery 
-Dtests.method=leaderRecoverFromLogOnStartupTest -Dtests.seed=3A47B4CBBFB61A9B 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=ro-RO -Dtests.timezone=Asia/Tel_Aviv -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE  180s J0 | 
TestCloudRecovery.leaderRecoverFromLogOnStartupTest <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Timeout waiting for 
all live and active
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([3A47B4CBBFB61A9B:4EB7558EFB925D14]:0)
   [junit4]    >        at 
org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest(TestCloudRecovery.java:105)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1012817 INFO  (jetty-closer-31446-thread-2) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@6c9e80b9{SSL,[ssl, 
http/1.1]}{127.0.0.1:41494}
   [junit4]   2> 1012818 INFO  (jetty-closer-31446-thread-2) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1598054702
   [junit4]   2> 1012821 INFO  (coreCloseExecutor-6991-thread-1) 
[n:127.0.0.1:41494_solr c:collection1 s:shard1 r:core_node1 
x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2]  
CLOSING SolrCore org.apache.solr.core.SolrCore@2c9f576b
   [junit4]   2> 1012821 INFO  (coreCloseExecutor-6991-thread-2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.c.SolrCore [collection1_shard2_replica2]  
CLOSING SolrCore org.apache.solr.core.SolrCore@7b3a43b2
   [junit4]   2> 1012824 INFO  
(zkCallback-31492-thread-2-processing-n:127.0.0.1:41494_solr) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [1])
   [junit4]   2> 1012825 INFO  (coreCloseExecutor-6991-thread-1) 
[n:127.0.0.1:41494_solr c:collection1 s:shard1 r:core_node1 
x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric 
reporters for: solr.core.collection1.shard1.replica2
   [junit4]   2> 1012826 INFO  (coreCloseExecutor-6991-thread-2) 
[n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 
x:collection1_shard2_replica2] o.a.s.m.SolrMetricManager Closing metric 
reporters for: solr.core.collection1.shard2.replica2
   [junit4]   2> 1012831 INFO  (jetty-closer-31446-thread-2) [    ] 
o.a.s.c.Overseer Overseer 
(id=97926298937786385-127.0.0.1:41494_solr-n_0000000004) closing
   [junit4]   2> 1012832 INFO  
(OverseerStateUpdate-97926298937786385-127.0.0.1:41494_solr-n_0000000004) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:41494_solr
   [junit4]   2> 1014337 WARN  
(zkCallback-31492-thread-2-processing-n:127.0.0.1:41494_solr) 
[n:127.0.0.1:41494_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1014337 INFO  (jetty-closer-31446-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1014337 INFO  (jetty-closer-31446-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@726f8116{/solr,null,UNAVAILABLE}
   [junit4]   2> 1014338 ERROR 
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper 
server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1014338 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:47460 47460
   [junit4]   2> 1014524 INFO  (Thread-2960) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:47460 47460
   [junit4]   2> 1014525 WARN  (Thread-2960) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        6       /solr/aliases.json
   [junit4]   2>        5       /solr/security.json
   [junit4]   2>        5       /solr/configs/config
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        6       /solr/clusterstate.json
   [junit4]   2>        6       /solr/clusterprops.json
   [junit4]   2>        5       /solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        6       /solr/collections
   [junit4]   2>        4       /solr/overseer/queue
   [junit4]   2>        4       /solr/overseer/collection-queue-work
   [junit4]   2>        3       /solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 1014525 INFO  
(SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001
   [junit4]   2> May 08, 2017 9:49:42 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{_version_=BlockTreeOrds(blocksize=128), 
id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, maxPointsInLeafNode=747, 
maxMBSortInHeap=5.582613842313201, 
sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=ro-RO, 
timezone=Asia/Tel_Aviv
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_121 (64-bit)/cpus=4,threads=1,free=305770280,total=517996544
   [junit4]   2> NOTE: All tests run in this JVM: [HighlighterMaxOffsetTest, 
DistributedFacetPivotSmallAdvancedTest, HdfsSyncSliceTest, 
LeaderElectionContextKeyTest, TestInfoStreamLogging, TestBadConfig, 
SmileWriterTest, DistributedQueryComponentCustomSortTest, SSLMigrationTest, 
TestSolrCoreProperties, TestDFISimilarityFactory, TestStressRecovery, 
TestQuerySenderNoQuery, SyncSliceTest, TestStressLiveNodes, 
TestAuthenticationFramework, TestSolrCloudWithSecureImpersonation, 
HdfsUnloadDistributedZkTest, SolrCLIZkUtilsTest, JvmMetricsTest, 
TestSizeLimitedDistributedMap, TestFieldCollectionResource, UUIDFieldTest, 
TestRandomFaceting, TestHighlightDedupGrouping, CoreAdminCreateDiscoverTest, 
TestSolrCloudSnapshots, TestRealTimeGet, SolrIndexMetricsTest, 
TestSolr4Spatial, ConfigureRecoveryStrategyTest, ZkNodePropsTest, 
TestRemoteStreaming, TestRangeQuery, UtilsToolTest, PeerSyncTest, 
TestRestManager, AtomicUpdateProcessorFactoryTest, TestCloudRecovery]
   [junit4] Completed [170/711 (1!)] on J0 in 243.98s, 2 tests, 1 failure <<< 
FAILURES!

[...truncated 45150 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to