Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/344/
7 tests failed. FAILED: org.apache.lucene.search.TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh Error Message: Test abandoned because suite timeout was reached. Stack Trace: java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager Error Message: Suite timeout exceeded (>= 7200000 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager Error Message: Captured an uncaught exception in thread: Thread[id=7461, name=Thread-6579, state=RUNNABLE, group=TGRP-TestSearcherManager] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7461, name=Thread-6579, state=RUNNABLE, group=TGRP-TestSearcherManager] Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm: Too many open files at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:661) Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm: Too many open files at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) at org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108) at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76) at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:649) FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager Error Message: Captured an uncaught exception in thread: Thread[id=7486, name=Lucene Merge Thread #2, state=RUNNABLE, group=TGRP-TestSearcherManager] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7486, name=Lucene Merge Thread #2, state=RUNNABLE, group=TGRP-TestSearcherManager] Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4088) at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4068) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3922) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) ... 2 more FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager Error Message: Captured an uncaught exception in thread: Thread[id=7484, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestSearcherManager] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7484, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestSearcherManager] Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) FAILED: junit.framework.TestSuite.org.apache.lucene.search.TestSearcherManager Error Message: Captured an uncaught exception in thread: Thread[id=7458, name=Thread-6576, state=RUNNABLE, group=TGRP-TestSearcherManager] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7458, name=Thread-6576, state=RUNNABLE, group=TGRP-TestSearcherManager] Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749) at org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:726) at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4582) at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4572) at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4559) at org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:517) at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083) at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005) at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043) at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316) at org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:559) Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) FAILED: org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest Error Message: Timeout waiting for all live and active Stack Trace: java.lang.AssertionError: Timeout waiting for all live and active at __randomizedtesting.SeedInfo.seed([3A47B4CBBFB61A9B:4EB7558EFB925D14]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest(TestCloudRecovery.java:105) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 1884 lines...] [junit4] Suite: org.apache.lucene.search.TestSearcherManager [junit4] 2> Beal 08, 2017 9:18:38 A.M. com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-6579,5,TGRP-TestSearcherManager] [junit4] 2> java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm: Too many open files [junit4] 2> at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:661) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] 2> at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] 2> at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] 2> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] 2> at org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] 2> at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) [junit4] 2> at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) [junit4] 2> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467) [junit4] 2> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) [junit4] 2> at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108) [junit4] 2> at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:649) [junit4] 2> [junit4] 2> Beal 08, 2017 9:18:38 A.M. com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #2,5,TGRP-TestSearcherManager] [junit4] 2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge [junit4] 2> at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) [junit4] 2> Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge [junit4] 2> at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4088) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4068) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3922) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] 2> at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] 2> at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] 2> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] 2> at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) [junit4] 2> ... 2 more [junit4] 2> [junit4] 2> Beal 08, 2017 9:18:38 A.M. com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #0,5,TGRP-TestSearcherManager] [junit4] 2> org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] 2> at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] 2> at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] 2> at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] 2> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] 2> at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) [junit4] 2> [junit4] 2> Beal 08, 2017 9:18:38 A.M. com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-6576,5,TGRP-TestSearcherManager] [junit4] 2> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed [junit4] 2> at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] 2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749) [junit4] 2> at org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:726) [junit4] 2> at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4582) [junit4] 2> at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4572) [junit4] 2> at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4559) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:517) [junit4] 2> at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083) [junit4] 2> at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005) [junit4] 2> at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) [junit4] 2> at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043) [junit4] 2> at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034) [junit4] 2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574) [junit4] 2> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:559) [junit4] 2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] 2> at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] 2> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] 2> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] 2> at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] 2> at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] 2> at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] 2> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] 2> at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] 2> at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] 2> at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] 2> at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] 2> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] 2> at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) [junit4] 2> [junit4] 2> Beal 08, 2017 11:12:26 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate [junit4] 2> WARNING: Suite execution timed out: org.apache.lucene.search.TestSearcherManager [junit4] 2> 1) Thread[id=6426, name=TEST-TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh-seed#[F70A69FB1A02E070], state=WAITING, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.lang.Thread.join(Thread.java:1249) [junit4] 2> at java.lang.Thread.join(Thread.java:1323) [junit4] 2> at org.apache.lucene.search.TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh(TestSearcherManager.java:672) [junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] 2> at java.lang.reflect.Method.invoke(Method.java:498) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> 2) Thread[id=7460, name=Thread-6578, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Throwable.fillInStackTrace(Native Method) [junit4] 2> at java.lang.Throwable.fillInStackTrace(Throwable.java:783) [junit4] 2> at java.lang.Throwable.<init>(Throwable.java:265) [junit4] 2> at java.lang.Exception.<init>(Exception.java:66) [junit4] 2> at java.lang.RuntimeException.<init>(RuntimeException.java:62) [junit4] 2> at java.lang.IllegalStateException.<init>(IllegalStateException.java:55) [junit4] 2> at org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26) [junit4] 2> at org.apache.lucene.search.ReferenceManager.ensureOpen(ReferenceManager.java:54) [junit4] 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:248) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:619) [junit4] 2> 3) Thread[id=1, name=main, state=WAITING, group=main] [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.lang.Thread.join(Thread.java:1249) [junit4] 2> at java.lang.Thread.join(Thread.java:1323) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457) [junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244) [junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355) [junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13) [junit4] 2> 4) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main] [junit4] 2> at java.lang.Thread.sleep(Native Method) [junit4] 2> at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50) [junit4] 2> 5) Thread[id=7459, name=Thread-6577, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Throwable.fillInStackTrace(Native Method) [junit4] 2> at java.lang.Throwable.fillInStackTrace(Throwable.java:783) [junit4] 2> at java.lang.Throwable.<init>(Throwable.java:265) [junit4] 2> at java.lang.Exception.<init>(Exception.java:66) [junit4] 2> at java.lang.RuntimeException.<init>(RuntimeException.java:62) [junit4] 2> at java.lang.IllegalStateException.<init>(IllegalStateException.java:55) [junit4] 2> at org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26) [junit4] 2> at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$9.run(TestSearcherManager.java:590) [junit4] 2> 6) Thread[id=6425, name=SUITE-TestSearcherManager-seed#[F70A69FB1A02E070], state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Thread.getStackTrace(Thread.java:1556) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693) [junit4] 2> at java.security.AccessController.doPrivileged(Native Method) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598) [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearcherManager -Dtests.method=testConcurrentIndexCloseSearchAndRefresh -Dtests.seed=F70A69FB1A02E070 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt -Dtests.locale=ga-IE -Dtests.timezone=Asia/Muscat -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 6866s J2 | TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh <<< [junit4] > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached. [junit4] > at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] 2> Beal 08, 2017 11:12:26 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 3 leaked thread(s). [junit4] 2> Beal 08, 2017 11:12:46 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> SEVERE: 3 threads leaked from SUITE scope at org.apache.lucene.search.TestSearcherManager: [junit4] 2> 1) Thread[id=6426, name=TEST-TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh-seed#[F70A69FB1A02E070], state=WAITING, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.lang.Thread.join(Thread.java:1249) [junit4] 2> at java.lang.Thread.join(Thread.java:1323) [junit4] 2> at org.apache.lucene.search.TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh(TestSearcherManager.java:672) [junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] 2> at java.lang.reflect.Method.invoke(Method.java:498) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> 2) Thread[id=7460, name=Thread-6578, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Throwable.fillInStackTrace(Native Method) [junit4] 2> at java.lang.Throwable.fillInStackTrace(Throwable.java:783) [junit4] 2> at java.lang.Throwable.<init>(Throwable.java:265) [junit4] 2> at java.lang.Exception.<init>(Exception.java:66) [junit4] 2> at java.lang.RuntimeException.<init>(RuntimeException.java:62) [junit4] 2> at java.lang.IllegalStateException.<init>(IllegalStateException.java:55) [junit4] 2> at org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26) [junit4] 2> at org.apache.lucene.search.ReferenceManager.ensureOpen(ReferenceManager.java:54) [junit4] 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:248) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:619) [junit4] 2> 3) Thread[id=7459, name=Thread-6577, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Throwable.fillInStackTrace(Native Method) [junit4] 2> at java.lang.Throwable.fillInStackTrace(Throwable.java:783) [junit4] 2> at java.lang.Throwable.<init>(Throwable.java:265) [junit4] 2> at java.lang.Exception.<init>(Exception.java:66) [junit4] 2> at java.lang.RuntimeException.<init>(RuntimeException.java:62) [junit4] 2> at java.lang.IllegalStateException.<init>(IllegalStateException.java:55) [junit4] 2> at org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26) [junit4] 2> at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$9.run(TestSearcherManager.java:590) [junit4] 2> Beal 08, 2017 11:12:46 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> INFO: Starting to interrupt leaked threads: [junit4] 2> 1) Thread[id=6426, name=TEST-TestSearcherManager.testConcurrentIndexCloseSearchAndRefresh-seed#[F70A69FB1A02E070], state=WAITING, group=TGRP-TestSearcherManager] [junit4] 2> 2) Thread[id=7460, name=Thread-6578, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> 3) Thread[id=7459, name=Thread-6577, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001 [junit4] 2> May 08, 2017 7:12:49 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated: [junit4] 2> 1) Thread[id=7460, name=Thread-6578, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Throwable.fillInStackTrace(Native Method) [junit4] 2> at java.lang.Throwable.fillInStackTrace(Throwable.java:783) [junit4] 2> at java.lang.Throwable.<init>(Throwable.java:265) [junit4] 2> at java.lang.Exception.<init>(Exception.java:66) [junit4] 2> at java.lang.RuntimeException.<init>(RuntimeException.java:62) [junit4] 2> at java.lang.IllegalStateException.<init>(IllegalStateException.java:55) [junit4] 2> at org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26) [junit4] 2> at org.apache.lucene.search.ReferenceManager.ensureOpen(ReferenceManager.java:54) [junit4] 2> at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:248) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$10.run(TestSearcherManager.java:619) [junit4] 2> 2) Thread[id=7459, name=Thread-6577, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] 2> at java.lang.Throwable.fillInStackTrace(Native Method) [junit4] 2> at java.lang.Throwable.fillInStackTrace(Throwable.java:783) [junit4] 2> at java.lang.Throwable.<init>(Throwable.java:265) [junit4] 2> at java.lang.Exception.<init>(Exception.java:66) [junit4] 2> at java.lang.RuntimeException.<init>(RuntimeException.java:62) [junit4] 2> at java.lang.IllegalStateException.<init>(IllegalStateException.java:55) [junit4] 2> at org.apache.lucene.store.AlreadyClosedException.<init>(AlreadyClosedException.java:26) [junit4] 2> at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:98) [junit4] 2> at org.apache.lucene.search.TestSearcherManager$9.run(TestSearcherManager.java:590) [junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {date=BlockTreeOrds(blocksize=128), extra35=BlockTreeOrds(blocksize=128), extra13=BlockTreeOrds(blocksize=128), extra36=PostingsFormat(name=LuceneVarGapFixedInterval), extra14=PostingsFormat(name=LuceneVarGapFixedInterval), extra33=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra11=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra8=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra12=PostingsFormat(name=LuceneFixedGap), extra9=PostingsFormat(name=LuceneFixedGap), extra34=PostingsFormat(name=LuceneFixedGap), extra39=BlockTreeOrds(blocksize=128), extra17=BlockTreeOrds(blocksize=128), extra18=PostingsFormat(name=LuceneVarGapFixedInterval), extra15=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra37=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra38=PostingsFormat(name=LuceneFixedGap), extra16=PostingsFormat(name=LuceneFixedGap), extra2=BlockTreeOrds(blocksize=128), extra3=PostingsFormat(name=LuceneVarGapFixedInterval), titleTokenized=PostingsFormat(name=LuceneVarGapFixedInterval), extra0=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra19=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra1=PostingsFormat(name=LuceneFixedGap), body=BlockTreeOrds(blocksize=128), title=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra6=BlockTreeOrds(blocksize=128), extra7=PostingsFormat(name=LuceneVarGapFixedInterval), extra4=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra5=PostingsFormat(name=LuceneFixedGap), packID=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra31=BlockTreeOrds(blocksize=128), extra32=PostingsFormat(name=LuceneVarGapFixedInterval), extra10=PostingsFormat(name=LuceneVarGapFixedInterval), extra30=PostingsFormat(name=LuceneFixedGap), extra24=BlockTreeOrds(blocksize=128), extra25=PostingsFormat(name=LuceneVarGapFixedInterval), extra22=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra23=PostingsFormat(name=LuceneFixedGap), extra28=BlockTreeOrds(blocksize=128), extra29=PostingsFormat(name=LuceneVarGapFixedInterval), docid=PostingsFormat(name=LuceneFixedGap), extra26=PostingsFormat(name=LuceneVarGapDocFreqInterval), extra27=PostingsFormat(name=LuceneFixedGap), extra20=BlockTreeOrds(blocksize=128), extra21=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{date=DocValuesFormat(name=Lucene54), extra13=DocValuesFormat(name=Lucene54), extra14=DocValuesFormat(name=Lucene54), extra11=DocValuesFormat(name=Lucene54), extra8=DocValuesFormat(name=Lucene54), extra12=DocValuesFormat(name=Asserting), extra9=DocValuesFormat(name=Lucene54), extra17=DocValuesFormat(name=Asserting), extra18=DocValuesFormat(name=Lucene54), extra15=DocValuesFormat(name=Lucene54), extra16=DocValuesFormat(name=Lucene54), extra2=DocValuesFormat(name=Lucene54), extra3=DocValuesFormat(name=Asserting), titleTokenized=DocValuesFormat(name=Lucene54), extra0=DocValuesFormat(name=Asserting), extra19=DocValuesFormat(name=Asserting), extra1=DocValuesFormat(name=Lucene54), body=DocValuesFormat(name=Asserting), extra6=DocValuesFormat(name=Lucene54), extra7=DocValuesFormat(name=Lucene54), extra4=DocValuesFormat(name=Lucene54), extra5=DocValuesFormat(name=Asserting), extra10=DocValuesFormat(name=Lucene54), extra35=DocValuesFormat(name=Asserting), extra36=DocValuesFormat(name=Lucene54), docid_int=DocValuesFormat(name=Lucene54), extra33=DocValuesFormat(name=Lucene54), extra34=DocValuesFormat(name=Lucene54), extra39=DocValuesFormat(name=Lucene54), extra37=DocValuesFormat(name=Asserting), extra38=DocValuesFormat(name=Lucene54), title=DocValuesFormat(name=Lucene54), packID=DocValuesFormat(name=Lucene54), docid_intDV=DocValuesFormat(name=Lucene54), extra31=DocValuesFormat(name=Lucene54), extra32=DocValuesFormat(name=Asserting), extra30=DocValuesFormat(name=Lucene54), extra24=DocValuesFormat(name=Lucene54), extra25=DocValuesFormat(name=Lucene54), extra22=DocValuesFormat(name=Lucene54), extra23=DocValuesFormat(name=Asserting), extra28=DocValuesFormat(name=Asserting), extra29=DocValuesFormat(name=Lucene54), docid=DocValuesFormat(name=Asserting), extra26=DocValuesFormat(name=Lucene54), extra27=DocValuesFormat(name=Lucene54), extra20=DocValuesFormat(name=Asserting), extra21=DocValuesFormat(name=Lucene54), titleDV=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1762, maxMBSortInHeap=7.4180958464928235, sim=RandomSimilarity(queryNorm=true,coord=no): {extra35=DFR I(n)L1, extra13=DFR I(F)1, extra36=DFR I(ne)BZ(0.3), extra14=IB LL-LZ(0.3), extra33=DFR G3(800.0), extra11=DFI(Saturated), extra8=DFR I(ne)2, extra12=IB SPL-L3(800.0), extra9=DFR I(ne)L3(800.0), extra34=IB SPL-D2, extra39=DFR I(ne)1, extra17=DFR I(ne)B1, extra18=DFR I(F)L3(800.0), extra15=DFR I(F)L2, extra37=IB SPL-L1, extra38=DFR I(F)1, extra16=IB SPL-D3(800.0), extra2=IB SPL-DZ(0.3), extra3=DFR I(ne)Z(0.3), titleTokenized=DFR G2, extra0=IB LL-DZ(0.3), extra19=DFR I(ne)BZ(0.3), extra1=LM Jelinek-Mercer(0.700000), body=IB SPL-D3(800.0), extra6=DFI(Standardized), extra7=DFR I(n)B2, extra4=IB SPL-LZ(0.3), extra5=DFR I(n)1, extra31=DFR I(F)L3(800.0), extra32=DFR I(F)L2, extra10=IB LL-L3(800.0), extra30=DFR I(F)Z(0.3), extra24=DFR I(F)2, extra25=IB LL-L2, extra22=DFR I(n)B3(800.0), extra23=DFR I(n)3(800.0), extra28=DFR I(n)B1, extra29=DFR GBZ(0.3), extra26=IB LL-D1, extra27=DFR I(ne)B3(800.0), extra20=DFR I(n)Z(0.3), extra21=DFI(ChiSquared)}, locale=ga-IE, timezone=Asia/Muscat [junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=3,free=116099416,total=354942976 [junit4] 2> NOTE: All tests run in this JVM: [Test2BPoints, TestStringMSBRadixSorter, TestFrequencyTrackingRingBuffer, TestLevenshteinAutomata, MultiCollectorTest, TestDocInverterPerFieldErrorInfo, TestStressIndexing, TestSameTokenSamePosition, TestSpanMultiTermQueryWrapper, TestForUtil, TestRamUsageEstimator, TestSpansEnum, TestBoolean2, TestRoaringDocIdSet, TestTrackingDirectoryWrapper, TestTermVectorsWriter, TestSimilarity, TestFieldsReader, TestWildcardRandom, TestLucene53NormsFormat, TestBlendedTermQuery, TestGrowableByteArrayDataOutput, TestReqExclBulkScorer, TestAllFilesCheckIndexHeader, TestUTF32ToUTF8, TestSpanSearchEquivalence, TestSegmentTermEnum, TestBooleanQuery, TestControlledRealTimeReopenThread, TestVersion, TestTopFieldCollector, TestMergedIterator, TestSwappedIndexFiles, TestGeoUtils, TestBooleanMinShouldMatch, TestClassicSimilarity, TestMultiCollector, TestIndexWriterOnJRECrash, TestIndexWriterFromReader, TestSpanNotQuery, TestBytesStore, TestSnapshotDeletionPolicy, TestByteBlockPool, TestIndexWriterReader, TestIndexWriterMerging, TestDeterminizeLexicon, TestCodecs, TestIndexWriterDelete, TestPrefixInBooleanQuery, TestRecyclingIntBlockAllocator, TestPayloadsOnVectors, TestOperations, TestSloppyPhraseQuery2, TestNumericRangeQuery32, TestNativeFSLockFactory, TestBytesRefHash, TestBooleanScorer, TestCachingTokenFilter, TestSegmentReader, TestSetOnce, TestTermdocPerf, TestTragicIndexWriterDeadlock, TestPackedTokenAttributeImpl, TestFileSwitchDirectory, TestCompiledAutomaton, TestSmallFloat, TestSimpleExplanationsOfNonMatches, TestCollectionUtil, TestNeedsScores, FiniteStringsIteratorTest, TestFilterCodecReader, TestComplexExplanations, TestSegmentInfos, TestExceedMaxTermLength, TestAllFilesHaveCodecHeader, TestSearchWithThreads, TestIndexWriterWithThreads, TestGraphTokenizers, TestAddIndexes, TestShardSearching, TestMultiMMap, TestSearcherManager] [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSearcherManager -Dtests.seed=F70A69FB1A02E070 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt -Dtests.locale=ga-IE -Dtests.timezone=Asia/Muscat -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 0.00s J2 | TestSearcherManager (suite) <<< [junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). [junit4] > at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7461, name=Thread-6579, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm: Too many open files [junit4] > at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] > at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:661) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c0_Asserting_0.dvm: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] > at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] > at org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] > at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] > at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] > at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) [junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467) [junit4] > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) [junit4] > at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:108) [junit4] > at org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:76) [junit4] > at org.apache.lucene.search.TestSearcherManager$11.run(TestSearcherManager.java:649)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7486, name=Lucene Merge Thread #2, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge [junit4] > at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) [junit4] > Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge [junit4] > at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4088) [junit4] > at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4068) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3922) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] > at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] > at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] > at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] > at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) [junit4] > ... 2 moreThrowable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7484, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] > at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] > at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] > at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] > at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] > at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7458, name=Thread-6576, state=RUNNABLE, group=TGRP-TestSearcherManager] [junit4] > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed [junit4] > at __randomizedtesting.SeedInfo.seed([F70A69FB1A02E070]:0) [junit4] > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749) [junit4] > at org.apache.lucene.index.IndexWriter.numDeletedDocs(IndexWriter.java:726) [junit4] > at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4582) [junit4] > at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4572) [junit4] > at org.apache.lucene.index.IndexWriter.segString(IndexWriter.java:4559) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:517) [junit4] > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083) [junit4] > at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005) [junit4] > at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731) [junit4] > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043) [junit4] > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034) [junit4] > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574) [junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316) [junit4] > at org.apache.lucene.search.TestSearcherManager$8.run(TestSearcherManager.java:559) [junit4] > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J2/temp/lucene.search.TestSearcherManager_F70A69FB1A02E070-001/tempDir-001/_c9_Lucene54_0.dvm: Too many open files [junit4] > at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:48) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81) [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197) [junit4] > at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:287) [junit4] > at java.nio.channels.FileChannel.open(FileChannel.java:335) [junit4] > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81) [junit4] > at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2741) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:749) [junit4] > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:113) [junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:294) [junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:383) [junit4] > at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) [junit4] > at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) [junit4] > at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:164) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:90) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4286) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) [junit4] Completed [456/456 (1!)] on J2 in 7223.23s, 3 tests, 6 errors <<< FAILURES! [...truncated 10408 lines...] [junit4] Suite: org.apache.solr.cloud.TestCloudRecovery [junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/init-core-data-001 [junit4] 2> 771062 WARN (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5 [junit4] 2> 771062 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields [junit4] 2> 771064 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN) [junit4] 2> 771065 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/tempDir-001 [junit4] 2> 771065 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 771066 INFO (Thread-2960) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 771066 INFO (Thread-2960) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 771075 ERROR (Thread-2960) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> 771166 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:47460 [junit4] 2> 771183 INFO (jetty-launcher-31445-thread-1) [ ] o.e.j.s.Server jetty-9.3.14.v20161028 [junit4] 2> 771189 INFO (jetty-launcher-31445-thread-2) [ ] o.e.j.s.Server jetty-9.3.14.v20161028 [junit4] 2> 771192 INFO (jetty-launcher-31445-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@670aecae{/solr,null,AVAILABLE} [junit4] 2> 771201 INFO (jetty-launcher-31445-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1959aef3{SSL,[ssl, http/1.1]}{127.0.0.1:47682} [junit4] 2> 771201 INFO (jetty-launcher-31445-thread-1) [ ] o.e.j.s.Server Started @774558ms [junit4] 2> 771201 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=47682} [junit4] 2> 771201 ERROR (jetty-launcher-31445-thread-1) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 771201 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.0 [junit4] 2> 771201 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 771201 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 771202 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-05-08T09:45:39.298Z [junit4] 2> 771203 INFO (jetty-launcher-31445-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4120162e{/solr,null,AVAILABLE} [junit4] 2> 771205 INFO (jetty-launcher-31445-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3729b5bc{SSL,[ssl, http/1.1]}{127.0.0.1:41494} [junit4] 2> 771205 INFO (jetty-launcher-31445-thread-2) [ ] o.e.j.s.Server Started @774562ms [junit4] 2> 771205 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41494} [junit4] 2> 771205 ERROR (jetty-launcher-31445-thread-2) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 771205 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.6.0 [junit4] 2> 771205 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 771205 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 771205 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-05-08T09:45:39.301Z [junit4] 2> 771215 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 771225 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 771227 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 771228 INFO (jetty-launcher-31445-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47460/solr [junit4] 2> 771232 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 771236 INFO (jetty-launcher-31445-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47460/solr [junit4] 2> 771254 INFO (jetty-launcher-31445-thread-1) [n:127.0.0.1:47682_solr ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 771258 INFO (jetty-launcher-31445-thread-1) [n:127.0.0.1:47682_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:47682_solr [junit4] 2> 771259 INFO (jetty-launcher-31445-thread-1) [n:127.0.0.1:47682_solr ] o.a.s.c.Overseer Overseer (id=97926298937786372-127.0.0.1:47682_solr-n_0000000000) starting [junit4] 2> 771262 INFO (jetty-launcher-31445-thread-2) [n:127.0.0.1:41494_solr ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 771263 INFO (jetty-launcher-31445-thread-2) [n:127.0.0.1:41494_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41494_solr [junit4] 2> 771267 INFO (zkCallback-31453-thread-1-processing-n:127.0.0.1:47682_solr) [n:127.0.0.1:47682_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 771279 INFO (jetty-launcher-31445-thread-1) [n:127.0.0.1:47682_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47682_solr [junit4] 2> 771280 INFO (zkCallback-31458-thread-1-processing-n:127.0.0.1:41494_solr) [n:127.0.0.1:41494_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 771284 INFO (zkCallback-31453-thread-1-processing-n:127.0.0.1:47682_solr) [n:127.0.0.1:47682_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 771291 INFO (zkCallback-31458-thread-1-processing-n:127.0.0.1:41494_solr) [n:127.0.0.1:41494_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 771343 INFO (jetty-launcher-31445-thread-2) [n:127.0.0.1:41494_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/tempDir-001/node2/. [junit4] 2> 771352 INFO (jetty-launcher-31445-thread-1) [n:127.0.0.1:47682_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001/tempDir-001/node1/. [junit4] 2> 771642 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) [junit4] [...truncated too long message...] 41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@56b698b1 [junit4] 2> 902947 INFO (recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@27f46dae[collection1_shard2_replica2] main] [junit4] 2> 902948 INFO (recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.u.DirectUpdateHandler2 end_commit_flush [junit4] 2> 902948 INFO (searcherExecutor-6981-thread-1-processing-n:127.0.0.1:41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.c.SolrCore [collection1_shard2_replica2] Registered new searcher Searcher@27f46dae[collection1_shard2_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(6.6.0):C2)))} [junit4] 2> 902948 INFO (recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica2] {add=[2 (1566820847685992448), 3 (1566820847693332480)]} 0 56 [junit4] 2> 902948 WARN (recoveryExecutor-6988-thread-1-processing-n:127.0.0.1:41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=2 deletes=0 deleteByQuery=0 errors=0 positionOfStart=0} [junit4] 2> 902948 INFO (coreZkRegister-6973-thread-2-processing-n:127.0.0.1:41494_solr x:collection1_shard2_replica2 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 903053 INFO (zkCallback-31492-thread-1-processing-n:127.0.0.1:41494_solr) [n:127.0.0.1:41494_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [1]) [junit4] 2> 1012812 INFO (TEST-TestCloudRecovery.leaderRecoverFromLogOnStartupTest-seed#[3A47B4CBBFB61A9B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending leaderRecoverFromLogOnStartupTest [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestCloudRecovery -Dtests.method=leaderRecoverFromLogOnStartupTest -Dtests.seed=3A47B4CBBFB61A9B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt -Dtests.locale=ro-RO -Dtests.timezone=Asia/Tel_Aviv -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] FAILURE 180s J0 | TestCloudRecovery.leaderRecoverFromLogOnStartupTest <<< [junit4] > Throwable #1: java.lang.AssertionError: Timeout waiting for all live and active [junit4] > at __randomizedtesting.SeedInfo.seed([3A47B4CBBFB61A9B:4EB7558EFB925D14]:0) [junit4] > at org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest(TestCloudRecovery.java:105) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> 1012817 INFO (jetty-closer-31446-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@6c9e80b9{SSL,[ssl, http/1.1]}{127.0.0.1:41494} [junit4] 2> 1012818 INFO (jetty-closer-31446-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1598054702 [junit4] 2> 1012821 INFO (coreCloseExecutor-6991-thread-1) [n:127.0.0.1:41494_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore@2c9f576b [junit4] 2> 1012821 INFO (coreCloseExecutor-6991-thread-2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.c.SolrCore [collection1_shard2_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore@7b3a43b2 [junit4] 2> 1012824 INFO (zkCallback-31492-thread-2-processing-n:127.0.0.1:41494_solr) [n:127.0.0.1:41494_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [1]) [junit4] 2> 1012825 INFO (coreCloseExecutor-6991-thread-1) [n:127.0.0.1:41494_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica2 [junit4] 2> 1012826 INFO (coreCloseExecutor-6991-thread-2) [n:127.0.0.1:41494_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard2.replica2 [junit4] 2> 1012831 INFO (jetty-closer-31446-thread-2) [ ] o.a.s.c.Overseer Overseer (id=97926298937786385-127.0.0.1:41494_solr-n_0000000004) closing [junit4] 2> 1012832 INFO (OverseerStateUpdate-97926298937786385-127.0.0.1:41494_solr-n_0000000004) [n:127.0.0.1:41494_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41494_solr [junit4] 2> 1014337 WARN (zkCallback-31492-thread-2-processing-n:127.0.0.1:41494_solr) [n:127.0.0.1:41494_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 1014337 INFO (jetty-closer-31446-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node [junit4] 2> 1014337 INFO (jetty-closer-31446-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@726f8116{/solr,null,UNAVAILABLE} [junit4] 2> 1014338 ERROR (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> 1014338 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47460 47460 [junit4] 2> 1014524 INFO (Thread-2960) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47460 47460 [junit4] 2> 1014525 WARN (Thread-2960) [ ] o.a.s.c.ZkTestServer Watch limit violations: [junit4] 2> Maximum concurrent create/delete watches above limit: [junit4] 2> [junit4] 2> 6 /solr/aliases.json [junit4] 2> 5 /solr/security.json [junit4] 2> 5 /solr/configs/config [junit4] 2> [junit4] 2> Maximum concurrent data watches above limit: [junit4] 2> [junit4] 2> 6 /solr/clusterstate.json [junit4] 2> 6 /solr/clusterprops.json [junit4] 2> 5 /solr/collections/collection1/state.json [junit4] 2> [junit4] 2> Maximum concurrent children watches above limit: [junit4] 2> [junit4] 2> 6 /solr/collections [junit4] 2> 4 /solr/overseer/queue [junit4] 2> 4 /solr/overseer/collection-queue-work [junit4] 2> 3 /solr/live_nodes [junit4] 2> [junit4] 2> 1014525 INFO (SUITE-TestCloudRecovery-seed#[3A47B4CBBFB61A9B]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_3A47B4CBBFB61A9B-001 [junit4] 2> May 08, 2017 9:49:42 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s). [junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {_version_=BlockTreeOrds(blocksize=128), id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, maxPointsInLeafNode=747, maxMBSortInHeap=5.582613842313201, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=ro-RO, timezone=Asia/Tel_Aviv [junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=305770280,total=517996544 [junit4] 2> NOTE: All tests run in this JVM: [HighlighterMaxOffsetTest, DistributedFacetPivotSmallAdvancedTest, HdfsSyncSliceTest, LeaderElectionContextKeyTest, TestInfoStreamLogging, TestBadConfig, SmileWriterTest, DistributedQueryComponentCustomSortTest, SSLMigrationTest, TestSolrCoreProperties, TestDFISimilarityFactory, TestStressRecovery, TestQuerySenderNoQuery, SyncSliceTest, TestStressLiveNodes, TestAuthenticationFramework, TestSolrCloudWithSecureImpersonation, HdfsUnloadDistributedZkTest, SolrCLIZkUtilsTest, JvmMetricsTest, TestSizeLimitedDistributedMap, TestFieldCollectionResource, UUIDFieldTest, TestRandomFaceting, TestHighlightDedupGrouping, CoreAdminCreateDiscoverTest, TestSolrCloudSnapshots, TestRealTimeGet, SolrIndexMetricsTest, TestSolr4Spatial, ConfigureRecoveryStrategyTest, ZkNodePropsTest, TestRemoteStreaming, TestRangeQuery, UtilsToolTest, PeerSyncTest, TestRestManager, AtomicUpdateProcessorFactoryTest, TestCloudRecovery] [junit4] Completed [170/711 (1!)] on J0 in 243.98s, 2 tests, 1 failure <<< FAILURES! [...truncated 45150 lines...]
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org