I thought about that, but then realized that the test does all the merging before the update threads start. This test only verifies that many concurrent updates (to same docs, fields etc.) works well. testSegmentMerging tests that concurrent updates w/ segment merging works well. So there really isn't a good reason for force many segments, and w/ maxBufDocs=2 and numDocs=atLeast(2000), that means lots of merging, all happen before the update threads start anyway.
Shai On Sat, Sep 21, 2013 at 8:01 PM, Michael McCandless < [email protected]> wrote: > Maybe we can make maxBufDocs greater than 2 but still smallish? Just > to make sure we are stressing merging + updating ... > > Mike McCandless > > http://blog.mikemccandless.com > > > On Sat, Sep 21, 2013 at 9:05 AM, Shai Erera <[email protected]> wrote: > > I don't hit an OOM with the reported parameters, but I do see the test > takes > > extremely long time to finish (I killed it after >10 minutes!). > > > > With the reported parameters, I see that the test uses > SerialMergeScheduler, > > spawns 18 threads and indexes 13K docs. Also, each thread calls > IW.updateNDV > > 197 times. The test also hard-sets maxBufDocs to 2, which means a lot of > > segment merging. With these numbers, it just takes too long to complete > (I > > see it makes progress, slowly). Maybe that, together with other tests > > running on Jenkins, can result in OOM. > > > > Anyway, I'd like to remove the hard-coded maxBufDocs=2. With > > numDocs=atLeast(2000), we should get few segments. Even with that, the > test > > takes 4m to complete (w/ those settings). Maybe it's too stressing? > > > > Shai > > > > > > On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server > > <[email protected]> wrote: > >> > >> Build: > https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/ > >> > >> 1 tests failed. > >> REGRESSION: > >> > org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading > >> > >> Error Message: > >> Captured an uncaught exception in thread: Thread[id=3130, > >> name=UpdateThread-1, state=RUNNABLE, > group=TGRP-TestNumericDocValuesUpdates] > >> > >> Stack Trace: > >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, > >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > >> Caused by: java.lang.OutOfMemoryError: Java heap space > >> at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > >> at java.util.Arrays.copyOf(Arrays.java:2367) > >> at > >> > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) > >> at > >> > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) > >> at > >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) > >> at java.lang.StringBuilder.append(StringBuilder.java:132) > >> at java.lang.StringBuilder.append(StringBuilder.java:128) > >> at > >> java.util.AbstractCollection.toString(AbstractCollection.java:450) > >> at java.lang.String.valueOf(String.java:2854) > >> at java.lang.StringBuilder.append(StringBuilder.java:128) > >> at > >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) > >> at > >> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) > >> at > >> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > >> at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > >> at > >> > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > >> > >> > >> > >> > >> Build Log: > >> [...truncated 1672 lines...] > >> [junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates > >> [junit4] 2> 9 20, 0025 3:45:48 ?? > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > >> uncaughtException > >> [junit4] 2> WARNING: Uncaught exception in thread: > >> Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates] > >> [junit4] 2> java.lang.OutOfMemoryError: Java heap space > >> [junit4] 2> at > >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > >> [junit4] 2> at java.util.Arrays.copyOf(Arrays.java:2367) > >> [junit4] 2> at > >> > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) > >> [junit4] 2> at > >> > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) > >> [junit4] 2> at > >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) > >> [junit4] 2> at > >> java.lang.StringBuilder.append(StringBuilder.java:132) > >> [junit4] 2> at > >> java.lang.StringBuilder.append(StringBuilder.java:128) > >> [junit4] 2> at > >> java.util.AbstractCollection.toString(AbstractCollection.java:450) > >> [junit4] 2> at java.lang.String.valueOf(String.java:2854) > >> [junit4] 2> at > >> java.lang.StringBuilder.append(StringBuilder.java:128) > >> [junit4] 2> at > >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) > >> [junit4] 2> at > >> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) > >> [junit4] 2> at > >> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > >> [junit4] 2> at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > >> [junit4] 2> at > >> > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > >> [junit4] 2> > >> [junit4] 2> 9 20, 0025 3:47:09 ?? > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > >> uncaughtException > >> [junit4] 2> WARNING: Uncaught exception in thread: > >> Thread[UpdateThread-3,5,TGRP-TestNumericDocValuesUpdates] > >> [junit4] 2> java.lang.IllegalStateException: this writer hit an > >> OutOfMemoryError; cannot commit > >> [junit4] 2> at > >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > >> [junit4] 2> at > >> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > >> [junit4] 2> at > >> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > >> [junit4] 2> at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > >> [junit4] 2> at > >> > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > >> [junit4] 2> > >> [junit4] 2> 9 20, 0025 3:47:32 ?? > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler > >> uncaughtException > >> [junit4] 2> WARNING: Uncaught exception in thread: > >> Thread[UpdateThread-8,5,TGRP-TestNumericDocValuesUpdates] > >> [junit4] 2> java.lang.IllegalStateException: this writer hit an > >> OutOfMemoryError; cannot commit > >> [junit4] 2> at > >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > >> [junit4] 2> at > >> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > >> [junit4] 2> at > >> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > >> [junit4] 2> at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > >> [junit4] 2> at > >> > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > >> [junit4] 2> > >> [junit4] 2> NOTE: download the large Jenkins line-docs file by > >> running 'ant get-jenkins-line-docs' in the lucene directory. > >> [junit4] 2> NOTE: reproduce with: ant test > >> -Dtestcase=TestNumericDocValuesUpdates > >> -Dtests.method=testStressMultiThreading -Dtests.seed=EA6A02F6820CB4B7 > >> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true > >> -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt > >> -Dtests.locale=ja_JP_JP_#u-ca-japanese -Dtests.timezone=Asia/Damascus > >> -Dtests.file.encoding=US-ASCII > >> [junit4] ERROR 1336s J1 | > >> TestNumericDocValuesUpdates.testStressMultiThreading <<< > >> [junit4] > Throwable #1: > >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an > >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, > >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > >> [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space > >> [junit4] > at > >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > >> [junit4] > at java.util.Arrays.copyOf(Arrays.java:2367) > >> [junit4] > at > >> > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) > >> [junit4] > at > >> > java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) > >> [junit4] > at > >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) > >> [junit4] > at > >> java.lang.StringBuilder.append(StringBuilder.java:132) > >> [junit4] > at > >> java.lang.StringBuilder.append(StringBuilder.java:128) > >> [junit4] > at > >> java.util.AbstractCollection.toString(AbstractCollection.java:450) > >> [junit4] > at java.lang.String.valueOf(String.java:2854) > >> [junit4] > at > >> java.lang.StringBuilder.append(StringBuilder.java:128) > >> [junit4] > at > >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) > >> [junit4] > at > >> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) > >> [junit4] > at > >> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > >> [junit4] > at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > >> [junit4] > at > >> > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable > >> #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured > an > >> uncaught exception in thread: Thread[id=3132, name=UpdateThread-3, > >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > >> [junit4] > Caused by: java.lang.IllegalStateException: this writer > >> hit an OutOfMemoryError; cannot commit > >> [junit4] > at > >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > >> [junit4] > at > >> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > >> [junit4] > at > >> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > >> [junit4] > at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > >> [junit4] > at > >> > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable > >> #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured > an > >> uncaught exception in thread: Thread[id=3137, name=UpdateThread-8, > >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] > >> [junit4] > Caused by: java.lang.IllegalStateException: this writer > >> hit an OutOfMemoryError; cannot commit > >> [junit4] > at > >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) > >> [junit4] > at > >> > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) > >> [junit4] > at > >> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) > >> [junit4] > at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) > >> [junit4] > at > >> > org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) > >> [junit4] 2> NOTE: test params are: codec=Lucene45: > >> {ndv=PostingsFormat(name=Memory doPackFST= false), > >> f=PostingsFormat(name=TestBloomFilteredLucene41Postings), > >> id=PostingsFormat(name=SimpleText), > >> dvUpdateKey=PostingsFormat(name=TestBloomFilteredLucene41Postings), > >> foo=PostingsFormat(name=TestBloomFilteredLucene41Postings), > >> updKey=PostingsFormat(name=Memory doPackFST= false), > >> k1=PostingsFormat(name=TestBloomFilteredLucene41Postings), > >> key=PostingsFormat(name=SimpleText), > k2=PostingsFormat(name=SimpleText)}, > >> docValues:{val=DocValuesFormat(name=Memory), > >> f6=DocValuesFormat(name=Lucene45), f7=DocValuesFormat(name=Memory), > >> f8=DocValuesFormat(name=SimpleText), f9=DocValuesFormat(name=Disk), > >> f12=DocValuesFormat(name=Disk), f11=DocValuesFormat(name=SimpleText), > >> f10=DocValuesFormat(name=Memory), f16=DocValuesFormat(name=Disk), > >> f15=DocValuesFormat(name=SimpleText), f14=DocValuesFormat(name=Memory), > >> f13=DocValuesFormat(name=Lucene45), > ssdv=DocValuesFormat(name=SimpleText), > >> f17=DocValuesFormat(name=Lucene45), > cf14=DocValuesFormat(name=SimpleText), > >> ndv=DocValuesFormat(name=Lucene45), cf15=DocValuesFormat(name=Disk), > >> cf12=DocValuesFormat(name=Lucene45), cf13=DocValuesFormat(name=Memory), > >> cf10=DocValuesFormat(name=SimpleText), cf11=DocValuesFormat(name=Disk), > >> ndv1=DocValuesFormat(name=Memory), f1=DocValuesFormat(name=Disk), > >> ndv2=DocValuesFormat(name=SimpleText), > f0=DocValuesFormat(name=SimpleText), > >> f3=DocValuesFormat(name=Memory), f2=DocValuesFormat(name=Lucene45), > >> bdv=DocValuesFormat(name=Lucene45), cf16=DocValuesFormat(name=Lucene45), > >> f5=DocValuesFormat(name=Disk), cf17=DocValuesFormat(name=Memory), > >> f4=DocValuesFormat(name=SimpleText), f=DocValuesFormat(name=SimpleText), > >> sdv=DocValuesFormat(name=Memory), cf5=DocValuesFormat(name=SimpleText), > >> cf6=DocValuesFormat(name=Disk), cf3=DocValuesFormat(name=Lucene45), > >> cf4=DocValuesFormat(name=Memory), cf9=DocValuesFormat(name=SimpleText), > >> cf7=DocValuesFormat(name=Lucene45), cf8=DocValuesFormat(name=Memory), > >> cf2=DocValuesFormat(name=Disk), cf1=DocValuesFormat(name=SimpleText), > >> cf0=DocValuesFormat(name=Memory), cf=DocValuesFormat(name=Disk)}, > >> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, > >> locale=ja_JP_JP_#u-ca-japanese, timezone=Asia/Damascus > >> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation > >> 1.7.0_25 (64-bit)/cpus=16,threads=1,free=203415056,total=515244032 > >> [junit4] 2> NOTE: All tests run in this JVM: [TestIndexSearcher, > >> TestNearSpansOrdered, TestAttributeSource, Nested1, TestRegexpRandom2, > >> TestQueryWrapperFilter, TestToken, TestByteBlockPool, > >> TestWorstCaseTestBehavior, TestLongPostings, > TestSimpleSearchEquivalence, > >> TestCollectionUtil, TestTerm, TestBoolean2, TestDirectory, > >> TestLazyProxSkipping, NestedSetupChain, NestedTeardownChain, > >> TestMultiThreadTermVectors, TestCharFilter, TestSpansAdvanced, > >> TestIndexWriterMergePolicy, TestVersion, TestMinimize, TestUnicodeUtil, > >> TestPerFieldPostingsFormat, TestPerFieldDocValuesFormat, > >> TestPerSegmentDeletes, TestDoc, TestSortDocValues, TestDuelingCodecs, > >> TestSegmentMerger, TestControlledRealTimeReopenThread, TestCharsRef, > >> TestGraphTokenizers, TestIntBlockPool, TestIndexInput, > >> TestRamUsageEstimatorOnWildAnimals, TestNeverDelete, > >> TestIndexWriterNRTIsCurrent, TestSpanExplanations, TestRAMDirectory, > >> TestPositiveScoresOnlyCollector, Nested, TestSizeBoundedForceMerge, > >> TestDateSort, TestPayloadExplanations, TestNewestSegment, > >> TestTotalHitCountCollector, Test4GBStoredFields, TestDocCount, > >> TestPackedInts, TestFieldValueFilter, TestCachingWrapperFilter, > >> TestDocsAndPositions, TestNumericUtils, TestElevationComparator, > >> TestIndexFileDeleter, TestLucene40StoredFieldsFormat, > TestDocumentWriter, > >> TestOpenBitSet, TestMultiDocValues, Test2BDocs, TestTermVectors, > >> TestDocValuesScoring, TestWindowsMMap, TestFlex, > TestBlockPostingsFormat2, > >> TestIndexWriterUnicode, TestMultiMMap, TestPayloadNearQuery, > >> TestSnapshotDeletionPolicy, TestFuzzyQuery, TestTermRangeQuery, Nested, > >> TestTieredMergePolicy, TestAtomicUpdate, TestDateFilter, > >> TestIdentityHashSet, TestIntsRef, TestIndexWriterOutOfFileDescriptors, > >> TestParallelCompositeReader, TestFieldCache, TestBufferedIndexInput, > >> TestIOUtils, TestBooleanQuery, TestPForDeltaDocIdSet, > >> TestMultiLevelSkipList, TestCodecs, TestDocument, > TestInPlaceMergeSorter, > >> TestDocValuesFormat, TestNumericRangeQuery64, TestBytesStore, > >> TestLucene42DocValuesFormat, TestShardSearching, TestSearcherManager, > >> TestOmitNorms, TestFixedBitSet, TestBitVector, TestCustomSearcherSort, > >> TestDocValuesIndexing, TestSimilarityBase, TestHugeRamFile, > >> TestBooleanMinShouldMatch, TestDeterminizeLexicon, > >> TestIndexWriterOnDiskFull, TestSameTokenSamePosition, > TestBinaryDocument, > >> TestAutomatonQueryUnicode, TestFieldCacheTermsFilter, Nested1, > >> TestBooleanScorer, TestMockDirectoryWrapper, TestRamUsageEstimator, > Nested1, > >> Nested, TestNoMergePolicy, TestNamedSPILoader, TestPhrasePrefixQuery, > >> TestDemo, TestAssertions, TestRollback, TestFieldInfos, > >> TestTwoPhaseCommitTool, TestNot, TestExplanations, > TestVersionComparator, > >> TestIndexWriterOnJRECrash, TestPrefixFilter, TestCodecHoldsOpenFiles, > >> TestReaderClosed, MultiCollectorTest, TestDocBoost, > TestNGramPhraseQuery, > >> TestTermdocPerf, Test2BPositions, TestIndexCommit, Test2BPostingsBytes, > >> TestTopScoreDocCollector, TestByteArrayDataInput, > >> TestNumericDocValuesUpdates] > >> [junit4] Completed on J1 in 1346.04s, 24 tests, 1 error <<< FAILURES! > >> > >> [...truncated 16 lines...] > >> BUILD FAILED > >> > >> > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:428: > >> The following error occurred while executing this line: > >> > >> > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:401: > >> The following error occurred while executing this line: > >> > >> > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: > >> The following error occurred while executing this line: > >> > >> > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:49: > >> The following error occurred while executing this line: > >> > >> > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1230: > >> The following error occurred while executing this line: > >> > >> > /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:873: > >> There were test failures: 368 suites, 2606 tests, 1 error, 59 ignored > (46 > >> assumptions) > >> > >> Total time: 53 minutes 6 seconds > >> Build step 'Invoke Ant' marked build as failure > >> Archiving artifacts > >> Recording test results > >> Email was triggered for: Failure > >> Sending email for trigger: Failure > >> > >> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
