I don't hit an OOM with the reported parameters, but I do see the test takes extremely long time to finish (I killed it after >10 minutes!).
With the reported parameters, I see that the test uses SerialMergeScheduler, spawns 18 threads and indexes 13K docs. Also, each thread calls IW.updateNDV 197 times. The test also hard-sets maxBufDocs to 2, which means a lot of segment merging. With these numbers, it just takes too long to complete (I see it makes progress, slowly). Maybe that, together with other tests running on Jenkins, can result in OOM. Anyway, I'd like to remove the hard-coded maxBufDocs=2. With numDocs=atLeast(2000), we should get few segments. Even with that, the test takes 4m to complete (w/ those settings). Maybe it's too stressing? Shai On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server < [email protected]> wrote: > Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/ > > 1 tests failed. > REGRESSION: > org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading > > Error Message: > Captured an uncaught exception in thread: Thread[id=3130, > name=UpdateThread-1, state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > > Stack Trace: > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, > state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > Caused by: java.lang.OutOfMemoryError: Java heap space > at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > at java.util.Arrays.copyOf(Arrays.java:2367) > at > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) > at java.lang.StringBuilder.append(StringBuilder.java:132) > at java.lang.StringBuilder.append(StringBuilder.java:128) > at > java.util.AbstractCollection.toString(AbstractCollection.java:450) > at java.lang.String.valueOf(String.java:2854) > at java.lang.StringBuilder.append(StringBuilder.java:128) > at > org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > at > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > > > > > Build Log: > [...truncated 1672 lines...] > [junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates > [junit4] 2> 9 20, 0025 3:45:48 ?? > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > uncaughtException > [junit4] 2> WARNING: Uncaught exception in thread: > Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates] > [junit4] 2> java.lang.OutOfMemoryError: Java heap space > [junit4] 2> at > __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > [junit4] 2> at java.util.Arrays.copyOf(Arrays.java:2367) > [junit4] 2> at > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) > [junit4] 2> at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) > [junit4] 2> at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) > [junit4] 2> at > java.lang.StringBuilder.append(StringBuilder.java:132) > [junit4] 2> at > java.lang.StringBuilder.append(StringBuilder.java:128) > [junit4] 2> at > java.util.AbstractCollection.toString(AbstractCollection.java:450) > [junit4] 2> at java.lang.String.valueOf(String.java:2854) > [junit4] 2> at > java.lang.StringBuilder.append(StringBuilder.java:128) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > [junit4] 2> at > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > [junit4] 2> > [junit4] 2> 9 20, 0025 3:47:09 ?? > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > uncaughtException > [junit4] 2> WARNING: Uncaught exception in thread: > Thread[UpdateThread-3,5,TGRP-TestNumericDocValuesUpdates] > [junit4] 2> java.lang.IllegalStateException: this writer hit an > OutOfMemoryError; cannot commit > [junit4] 2> at > __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > [junit4] 2> at > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > [junit4] 2> > [junit4] 2> 9 20, 0025 3:47:32 ?? > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > uncaughtException > [junit4] 2> WARNING: Uncaught exception in thread: > Thread[UpdateThread-8,5,TGRP-TestNumericDocValuesUpdates] > [junit4] 2> java.lang.IllegalStateException: this writer hit an > OutOfMemoryError; cannot commit > [junit4] 2> at > __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > [junit4] 2> at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > [junit4] 2> at > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > [junit4] 2> > [junit4] 2> NOTE: download the large Jenkins line-docs file by > running 'ant get-jenkins-line-docs' in the lucene directory. > [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=TestNumericDocValuesUpdates > -Dtests.method=testStressMultiThreading -Dtests.seed=EA6A02F6820CB4B7 > -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true > -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt > -Dtests.locale=ja_JP_JP_#u-ca-japanese -Dtests.timezone=Asia/Damascus > -Dtests.file.encoding=US-ASCII > [junit4] ERROR 1336s J1 | > TestNumericDocValuesUpdates.testStressMultiThreading <<< > [junit4] > Throwable #1: > com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, > state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space > [junit4] > at > __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > [junit4] > at java.util.Arrays.copyOf(Arrays.java:2367) > [junit4] > at > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) > [junit4] > at > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) > [junit4] > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) > [junit4] > at > java.lang.StringBuilder.append(StringBuilder.java:132) > [junit4] > at > java.lang.StringBuilder.append(StringBuilder.java:128) > [junit4] > at > java.util.AbstractCollection.toString(AbstractCollection.java:450) > [junit4] > at java.lang.String.valueOf(String.java:2854) > [junit4] > at > java.lang.StringBuilder.append(StringBuilder.java:128) > [junit4] > at > org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) > [junit4] > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) > [junit4] > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > [junit4] > at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > [junit4] > at > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable > #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > uncaught exception in thread: Thread[id=3132, name=UpdateThread-3, > state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > [junit4] > Caused by: java.lang.IllegalStateException: this writer > hit an OutOfMemoryError; cannot commit > [junit4] > at > __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > [junit4] > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > [junit4] > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > [junit4] > at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > [junit4] > at > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable > #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > uncaught exception in thread: Thread[id=3137, name=UpdateThread-8, > state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > [junit4] > Caused by: java.lang.IllegalStateException: this writer > hit an OutOfMemoryError; cannot commit > [junit4] > at > __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > [junit4] > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > [junit4] > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > [junit4] > at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > [junit4] > at > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > [junit4] 2> NOTE: test params are: codec=Lucene45: > {ndv=PostingsFormat(name=Memory doPackFST= false), > f=PostingsFormat(name=TestBloomFilteredLucene41Postings), > id=PostingsFormat(name=SimpleText), > dvUpdateKey=PostingsFormat(name=TestBloomFilteredLucene41Postings), > foo=PostingsFormat(name=TestBloomFilteredLucene41Postings), > updKey=PostingsFormat(name=Memory doPackFST= false), > k1=PostingsFormat(name=TestBloomFilteredLucene41Postings), > key=PostingsFormat(name=SimpleText), k2=PostingsFormat(name=SimpleText)}, > docValues:{val=DocValuesFormat(name=Memory), > f6=DocValuesFormat(name=Lucene45), f7=DocValuesFormat(name=Memory), > f8=DocValuesFormat(name=SimpleText), f9=DocValuesFormat(name=Disk), > f12=DocValuesFormat(name=Disk), f11=DocValuesFormat(name=SimpleText), > f10=DocValuesFormat(name=Memory), f16=DocValuesFormat(name=Disk), > f15=DocValuesFormat(name=SimpleText), f14=DocValuesFormat(name=Memory), > f13=DocValuesFormat(name=Lucene45), ssdv=DocValuesFormat(name=SimpleText), > f17=DocValuesFormat(name=Lucene45), cf14=DocValuesFormat(name=SimpleText), > ndv=DocValuesFormat(name=Lucene45), cf15=DocValuesFormat(name=Disk), > cf12=DocValuesFormat(name=Lucene45), cf13=DocValuesFormat(name=Memory), > cf10=DocValuesFormat(name=SimpleText), cf11=DocValuesFormat(name=Disk), > ndv1=DocValuesFormat(name=Memory), f1=DocValuesFormat(name=Disk), > ndv2=DocValuesFormat(name=SimpleText), f0=DocValuesFormat(name=SimpleText), > f3=DocValuesFormat(name=Memory), f2=DocValuesFormat(name=Lucene45), > bdv=DocValuesFormat(name=Lucene45), cf16=DocValuesFormat(name=Lucene45), > f5=DocValuesFormat(name=Disk), cf17=DocValuesFormat(name=Memory), > f4=DocValuesFormat(name=SimpleText), f=DocValuesFormat(name=SimpleText), > sdv=DocValuesFormat(name=Memory), cf5=DocValuesFormat(name=SimpleText), > cf6=DocValuesFormat(name=Disk), cf3=DocValuesFormat(name=Lucene45), > cf4=DocValuesFormat(name=Memory), cf9=DocValuesFormat(name=SimpleText), > cf7=DocValuesFormat(name=Lucene45), cf8=DocValuesFormat(name=Memory), > cf2=DocValuesFormat(name=Disk), cf1=DocValuesFormat(name=SimpleText), > cf0=DocValuesFormat(name=Memory), cf=DocValuesFormat(name=Disk)}, > sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, > locale=ja_JP_JP_#u-ca-japanese, timezone=Asia/Damascus > [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation > 1.7.0_25 (64-bit)/cpus=16,threads=1,free=203415056,total=515244032 > [junit4] 2> NOTE: All tests run in this JVM: [TestIndexSearcher, > TestNearSpansOrdered, TestAttributeSource, Nested1, TestRegexpRandom2, > TestQueryWrapperFilter, TestToken, TestByteBlockPool, > TestWorstCaseTestBehavior, TestLongPostings, TestSimpleSearchEquivalence, > TestCollectionUtil, TestTerm, TestBoolean2, TestDirectory, > TestLazyProxSkipping, NestedSetupChain, NestedTeardownChain, > TestMultiThreadTermVectors, TestCharFilter, TestSpansAdvanced, > TestIndexWriterMergePolicy, TestVersion, TestMinimize, TestUnicodeUtil, > TestPerFieldPostingsFormat, TestPerFieldDocValuesFormat, > TestPerSegmentDeletes, TestDoc, TestSortDocValues, TestDuelingCodecs, > TestSegmentMerger, TestControlledRealTimeReopenThread, TestCharsRef, > TestGraphTokenizers, TestIntBlockPool, TestIndexInput, > TestRamUsageEstimatorOnWildAnimals, TestNeverDelete, > TestIndexWriterNRTIsCurrent, TestSpanExplanations, TestRAMDirectory, > TestPositiveScoresOnlyCollector, Nested, TestSizeBoundedForceMerge, > TestDateSort, TestPayloadExplanations, TestNewestSegment, > TestTotalHitCountCollector, Test4GBStoredFields, TestDocCount, > TestPackedInts, TestFieldValueFilter, TestCachingWrapperFilter, > TestDocsAndPositions, TestNumericUtils, TestElevationComparator, > TestIndexFileDeleter, TestLucene40StoredFieldsFormat, TestDocumentWriter, > TestOpenBitSet, TestMultiDocValues, Test2BDocs, TestTermVectors, > TestDocValuesScoring, TestWindowsMMap, TestFlex, TestBlockPostingsFormat2, > TestIndexWriterUnicode, TestMultiMMap, TestPayloadNearQuery, > TestSnapshotDeletionPolicy, TestFuzzyQuery, TestTermRangeQuery, Nested, > TestTieredMergePolicy, TestAtomicUpdate, TestDateFilter, > TestIdentityHashSet, TestIntsRef, TestIndexWriterOutOfFileDescriptors, > TestParallelCompositeReader, TestFieldCache, TestBufferedIndexInput, > TestIOUtils, TestBooleanQuery, TestPForDeltaDocIdSet, > TestMultiLevelSkipList, TestCodecs, TestDocument, TestInPlaceMergeSorter, > TestDocValuesFormat, TestNumericRangeQuery64, TestBytesStore, > TestLucene42DocValuesFormat, TestShardSearching, TestSearcherManager, > TestOmitNorms, TestFixedBitSet, TestBitVector, TestCustomSearcherSort, > TestDocValuesIndexing, TestSimilarityBase, TestHugeRamFile, > TestBooleanMinShouldMatch, TestDeterminizeLexicon, > TestIndexWriterOnDiskFull, TestSameTokenSamePosition, TestBinaryDocument, > TestAutomatonQueryUnicode, TestFieldCacheTermsFilter, Nested1, > TestBooleanScorer, TestMockDirectoryWrapper, TestRamUsageEstimator, > Nested1, Nested, TestNoMergePolicy, TestNamedSPILoader, > TestPhrasePrefixQuery, TestDemo, TestAssertions, TestRollback, > TestFieldInfos, TestTwoPhaseCommitTool, TestNot, TestExplanations, > TestVersionComparator, TestIndexWriterOnJRECrash, TestPrefixFilter, > TestCodecHoldsOpenFiles, TestReaderClosed, MultiCollectorTest, > TestDocBoost, TestNGramPhraseQuery, TestTermdocPerf, Test2BPositions, > TestIndexCommit, Test2BPostingsBytes, TestTopScoreDocCollector, > TestByteArrayDataInput, TestNumericDocValuesUpdates] > [junit4] Completed on J1 in 1346.04s, 24 tests, 1 error <<< FAILURES! > > [...truncated 16 lines...] > BUILD FAILED > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:428: > The following error occurred while executing this line: > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:401: > The following error occurred while executing this line: > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: > The following error occurred while executing this line: > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:49: > The following error occurred while executing this line: > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1230: > The following error occurred while executing this line: > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:873: > There were test failures: 368 suites, 2606 tests, 1 error, 59 ignored (46 > assumptions) > > Total time: 53 minutes 6 seconds > Build step 'Invoke Ant' marked build as failure > Archiving artifacts > Recording test results > Email was triggered for: Failure > Sending email for trigger: Failure > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
