Maybe we can make maxBufDocs greater than 2 but still smallish? Just to make sure we are stressing merging + updating ...
Mike McCandless http://blog.mikemccandless.com On Sat, Sep 21, 2013 at 9:05 AM, Shai Erera <[email protected]> wrote: > I don't hit an OOM with the reported parameters, but I do see the test takes > extremely long time to finish (I killed it after >10 minutes!). > > With the reported parameters, I see that the test uses SerialMergeScheduler, > spawns 18 threads and indexes 13K docs. Also, each thread calls IW.updateNDV > 197 times. The test also hard-sets maxBufDocs to 2, which means a lot of > segment merging. With these numbers, it just takes too long to complete (I > see it makes progress, slowly). Maybe that, together with other tests > running on Jenkins, can result in OOM. > > Anyway, I'd like to remove the hard-coded maxBufDocs=2. With > numDocs=atLeast(2000), we should get few segments. Even with that, the test > takes 4m to complete (w/ those settings). Maybe it's too stressing? > > Shai > > > On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server > <[email protected]> wrote: >> >> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/ >> >> 1 tests failed. >> REGRESSION: >> org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading >> >> Error Message: >> Captured an uncaught exception in thread: Thread[id=3130, >> name=UpdateThread-1, state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> >> Stack Trace: >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> Caused by: java.lang.OutOfMemoryError: Java heap space >> at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> at java.util.Arrays.copyOf(Arrays.java:2367) >> at >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) >> at >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) >> at >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >> at java.lang.StringBuilder.append(StringBuilder.java:132) >> at java.lang.StringBuilder.append(StringBuilder.java:128) >> at >> java.util.AbstractCollection.toString(AbstractCollection.java:450) >> at java.lang.String.valueOf(String.java:2854) >> at java.lang.StringBuilder.append(StringBuilder.java:128) >> at >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) >> at >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) >> at >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> at >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> >> >> >> >> Build Log: >> [...truncated 1672 lines...] >> [junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates >> [junit4] 2> 9 20, 0025 3:45:48 ?? >> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >> uncaughtException >> [junit4] 2> WARNING: Uncaught exception in thread: >> Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates] >> [junit4] 2> java.lang.OutOfMemoryError: Java heap space >> [junit4] 2> at >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> [junit4] 2> at java.util.Arrays.copyOf(Arrays.java:2367) >> [junit4] 2> at >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) >> [junit4] 2> at >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) >> [junit4] 2> at >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >> [junit4] 2> at >> java.lang.StringBuilder.append(StringBuilder.java:132) >> [junit4] 2> at >> java.lang.StringBuilder.append(StringBuilder.java:128) >> [junit4] 2> at >> java.util.AbstractCollection.toString(AbstractCollection.java:450) >> [junit4] 2> at java.lang.String.valueOf(String.java:2854) >> [junit4] 2> at >> java.lang.StringBuilder.append(StringBuilder.java:128) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> [junit4] 2> at >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> [junit4] 2> >> [junit4] 2> 9 20, 0025 3:47:09 ?? >> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >> uncaughtException >> [junit4] 2> WARNING: Uncaught exception in thread: >> Thread[UpdateThread-3,5,TGRP-TestNumericDocValuesUpdates] >> [junit4] 2> java.lang.IllegalStateException: this writer hit an >> OutOfMemoryError; cannot commit >> [junit4] 2> at >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> [junit4] 2> at >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> [junit4] 2> >> [junit4] 2> 9 20, 0025 3:47:32 ?? >> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >> uncaughtException >> [junit4] 2> WARNING: Uncaught exception in thread: >> Thread[UpdateThread-8,5,TGRP-TestNumericDocValuesUpdates] >> [junit4] 2> java.lang.IllegalStateException: this writer hit an >> OutOfMemoryError; cannot commit >> [junit4] 2> at >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> [junit4] 2> at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> [junit4] 2> at >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> [junit4] 2> >> [junit4] 2> NOTE: download the large Jenkins line-docs file by >> running 'ant get-jenkins-line-docs' in the lucene directory. >> [junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestNumericDocValuesUpdates >> -Dtests.method=testStressMultiThreading -Dtests.seed=EA6A02F6820CB4B7 >> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true >> -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt >> -Dtests.locale=ja_JP_JP_#u-ca-japanese -Dtests.timezone=Asia/Damascus >> -Dtests.file.encoding=US-ASCII >> [junit4] ERROR 1336s J1 | >> TestNumericDocValuesUpdates.testStressMultiThreading <<< >> [junit4] > Throwable #1: >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space >> [junit4] > at >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> [junit4] > at java.util.Arrays.copyOf(Arrays.java:2367) >> [junit4] > at >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) >> [junit4] > at >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) >> [junit4] > at >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >> [junit4] > at >> java.lang.StringBuilder.append(StringBuilder.java:132) >> [junit4] > at >> java.lang.StringBuilder.append(StringBuilder.java:128) >> [junit4] > at >> java.util.AbstractCollection.toString(AbstractCollection.java:450) >> [junit4] > at java.lang.String.valueOf(String.java:2854) >> [junit4] > at >> java.lang.StringBuilder.append(StringBuilder.java:128) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> [junit4] > at >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable >> #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> uncaught exception in thread: Thread[id=3132, name=UpdateThread-3, >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> [junit4] > Caused by: java.lang.IllegalStateException: this writer >> hit an OutOfMemoryError; cannot commit >> [junit4] > at >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> [junit4] > at >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable >> #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> uncaught exception in thread: Thread[id=3137, name=UpdateThread-8, >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> [junit4] > Caused by: java.lang.IllegalStateException: this writer >> hit an OutOfMemoryError; cannot commit >> [junit4] > at >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> [junit4] > at >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> [junit4] > at >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> [junit4] 2> NOTE: test params are: codec=Lucene45: >> {ndv=PostingsFormat(name=Memory doPackFST= false), >> f=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> id=PostingsFormat(name=SimpleText), >> dvUpdateKey=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> foo=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> updKey=PostingsFormat(name=Memory doPackFST= false), >> k1=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> key=PostingsFormat(name=SimpleText), k2=PostingsFormat(name=SimpleText)}, >> docValues:{val=DocValuesFormat(name=Memory), >> f6=DocValuesFormat(name=Lucene45), f7=DocValuesFormat(name=Memory), >> f8=DocValuesFormat(name=SimpleText), f9=DocValuesFormat(name=Disk), >> f12=DocValuesFormat(name=Disk), f11=DocValuesFormat(name=SimpleText), >> f10=DocValuesFormat(name=Memory), f16=DocValuesFormat(name=Disk), >> f15=DocValuesFormat(name=SimpleText), f14=DocValuesFormat(name=Memory), >> f13=DocValuesFormat(name=Lucene45), ssdv=DocValuesFormat(name=SimpleText), >> f17=DocValuesFormat(name=Lucene45), cf14=DocValuesFormat(name=SimpleText), >> ndv=DocValuesFormat(name=Lucene45), cf15=DocValuesFormat(name=Disk), >> cf12=DocValuesFormat(name=Lucene45), cf13=DocValuesFormat(name=Memory), >> cf10=DocValuesFormat(name=SimpleText), cf11=DocValuesFormat(name=Disk), >> ndv1=DocValuesFormat(name=Memory), f1=DocValuesFormat(name=Disk), >> ndv2=DocValuesFormat(name=SimpleText), f0=DocValuesFormat(name=SimpleText), >> f3=DocValuesFormat(name=Memory), f2=DocValuesFormat(name=Lucene45), >> bdv=DocValuesFormat(name=Lucene45), cf16=DocValuesFormat(name=Lucene45), >> f5=DocValuesFormat(name=Disk), cf17=DocValuesFormat(name=Memory), >> f4=DocValuesFormat(name=SimpleText), f=DocValuesFormat(name=SimpleText), >> sdv=DocValuesFormat(name=Memory), cf5=DocValuesFormat(name=SimpleText), >> cf6=DocValuesFormat(name=Disk), cf3=DocValuesFormat(name=Lucene45), >> cf4=DocValuesFormat(name=Memory), cf9=DocValuesFormat(name=SimpleText), >> cf7=DocValuesFormat(name=Lucene45), cf8=DocValuesFormat(name=Memory), >> cf2=DocValuesFormat(name=Disk), cf1=DocValuesFormat(name=SimpleText), >> cf0=DocValuesFormat(name=Memory), cf=DocValuesFormat(name=Disk)}, >> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, >> locale=ja_JP_JP_#u-ca-japanese, timezone=Asia/Damascus >> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation >> 1.7.0_25 (64-bit)/cpus=16,threads=1,free=203415056,total=515244032 >> [junit4] 2> NOTE: All tests run in this JVM: [TestIndexSearcher, >> TestNearSpansOrdered, TestAttributeSource, Nested1, TestRegexpRandom2, >> TestQueryWrapperFilter, TestToken, TestByteBlockPool, >> TestWorstCaseTestBehavior, TestLongPostings, TestSimpleSearchEquivalence, >> TestCollectionUtil, TestTerm, TestBoolean2, TestDirectory, >> TestLazyProxSkipping, NestedSetupChain, NestedTeardownChain, >> TestMultiThreadTermVectors, TestCharFilter, TestSpansAdvanced, >> TestIndexWriterMergePolicy, TestVersion, TestMinimize, TestUnicodeUtil, >> TestPerFieldPostingsFormat, TestPerFieldDocValuesFormat, >> TestPerSegmentDeletes, TestDoc, TestSortDocValues, TestDuelingCodecs, >> TestSegmentMerger, TestControlledRealTimeReopenThread, TestCharsRef, >> TestGraphTokenizers, TestIntBlockPool, TestIndexInput, >> TestRamUsageEstimatorOnWildAnimals, TestNeverDelete, >> TestIndexWriterNRTIsCurrent, TestSpanExplanations, TestRAMDirectory, >> TestPositiveScoresOnlyCollector, Nested, TestSizeBoundedForceMerge, >> TestDateSort, TestPayloadExplanations, TestNewestSegment, >> TestTotalHitCountCollector, Test4GBStoredFields, TestDocCount, >> TestPackedInts, TestFieldValueFilter, TestCachingWrapperFilter, >> TestDocsAndPositions, TestNumericUtils, TestElevationComparator, >> TestIndexFileDeleter, TestLucene40StoredFieldsFormat, TestDocumentWriter, >> TestOpenBitSet, TestMultiDocValues, Test2BDocs, TestTermVectors, >> TestDocValuesScoring, TestWindowsMMap, TestFlex, TestBlockPostingsFormat2, >> TestIndexWriterUnicode, TestMultiMMap, TestPayloadNearQuery, >> TestSnapshotDeletionPolicy, TestFuzzyQuery, TestTermRangeQuery, Nested, >> TestTieredMergePolicy, TestAtomicUpdate, TestDateFilter, >> TestIdentityHashSet, TestIntsRef, TestIndexWriterOutOfFileDescriptors, >> TestParallelCompositeReader, TestFieldCache, TestBufferedIndexInput, >> TestIOUtils, TestBooleanQuery, TestPForDeltaDocIdSet, >> TestMultiLevelSkipList, TestCodecs, TestDocument, TestInPlaceMergeSorter, >> TestDocValuesFormat, TestNumericRangeQuery64, TestBytesStore, >> TestLucene42DocValuesFormat, TestShardSearching, TestSearcherManager, >> TestOmitNorms, TestFixedBitSet, TestBitVector, TestCustomSearcherSort, >> TestDocValuesIndexing, TestSimilarityBase, TestHugeRamFile, >> TestBooleanMinShouldMatch, TestDeterminizeLexicon, >> TestIndexWriterOnDiskFull, TestSameTokenSamePosition, TestBinaryDocument, >> TestAutomatonQueryUnicode, TestFieldCacheTermsFilter, Nested1, >> TestBooleanScorer, TestMockDirectoryWrapper, TestRamUsageEstimator, Nested1, >> Nested, TestNoMergePolicy, TestNamedSPILoader, TestPhrasePrefixQuery, >> TestDemo, TestAssertions, TestRollback, TestFieldInfos, >> TestTwoPhaseCommitTool, TestNot, TestExplanations, TestVersionComparator, >> TestIndexWriterOnJRECrash, TestPrefixFilter, TestCodecHoldsOpenFiles, >> TestReaderClosed, MultiCollectorTest, TestDocBoost, TestNGramPhraseQuery, >> TestTermdocPerf, Test2BPositions, TestIndexCommit, Test2BPostingsBytes, >> TestTopScoreDocCollector, TestByteArrayDataInput, >> TestNumericDocValuesUpdates] >> [junit4] Completed on J1 in 1346.04s, 24 tests, 1 error <<< FAILURES! >> >> [...truncated 16 lines...] >> BUILD FAILED >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:428: >> The following error occurred while executing this line: >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:401: >> The following error occurred while executing this line: >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: >> The following error occurred while executing this line: >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:49: >> The following error occurred while executing this line: >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1230: >> The following error occurred while executing this line: >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:873: >> There were test failures: 368 suites, 2606 tests, 1 error, 59 ignored (46 >> assumptions) >> >> Total time: 53 minutes 6 seconds >> Build step 'Invoke Ant' marked build as failure >> Archiving artifacts >> Recording test results >> Email was triggered for: Failure >> Sending email for trigger: Failure >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
