Oh, I see ... OK +1 to remove the maxBufDocs=2. Mike McCandless
http://blog.mikemccandless.com On Sat, Sep 21, 2013 at 3:28 PM, Shai Erera <[email protected]> wrote: > I thought about that, but then realized that the test does all the merging > before the update threads start. This test only verifies that many > concurrent updates (to same docs, fields etc.) works well. > testSegmentMerging tests that concurrent updates w/ segment merging works > well. So there really isn't a good reason for force many segments, and w/ > maxBufDocs=2 and numDocs=atLeast(2000), that means lots of merging, all > happen before the update threads start anyway. > > Shai > > > On Sat, Sep 21, 2013 at 8:01 PM, Michael McCandless > <[email protected]> wrote: >> >> Maybe we can make maxBufDocs greater than 2 but still smallish? Just >> to make sure we are stressing merging + updating ... >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> >> On Sat, Sep 21, 2013 at 9:05 AM, Shai Erera <[email protected]> wrote: >> > I don't hit an OOM with the reported parameters, but I do see the test >> > takes >> > extremely long time to finish (I killed it after >10 minutes!). >> > >> > With the reported parameters, I see that the test uses >> > SerialMergeScheduler, >> > spawns 18 threads and indexes 13K docs. Also, each thread calls >> > IW.updateNDV >> > 197 times. The test also hard-sets maxBufDocs to 2, which means a lot of >> > segment merging. With these numbers, it just takes too long to complete >> > (I >> > see it makes progress, slowly). Maybe that, together with other tests >> > running on Jenkins, can result in OOM. >> > >> > Anyway, I'd like to remove the hard-coded maxBufDocs=2. With >> > numDocs=atLeast(2000), we should get few segments. Even with that, the >> > test >> > takes 4m to complete (w/ those settings). Maybe it's too stressing? >> > >> > Shai >> > >> > >> > On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server >> > <[email protected]> wrote: >> >> >> >> Build: >> >> https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/ >> >> >> >> 1 tests failed. >> >> REGRESSION: >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading >> >> >> >> Error Message: >> >> Captured an uncaught exception in thread: Thread[id=3130, >> >> name=UpdateThread-1, state=RUNNABLE, >> >> group=TGRP-TestNumericDocValuesUpdates] >> >> >> >> Stack Trace: >> >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, >> >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> >> Caused by: java.lang.OutOfMemoryError: Java heap space >> >> at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> >> at java.util.Arrays.copyOf(Arrays.java:2367) >> >> at >> >> >> >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) >> >> at >> >> >> >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) >> >> at >> >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >> >> at java.lang.StringBuilder.append(StringBuilder.java:132) >> >> at java.lang.StringBuilder.append(StringBuilder.java:128) >> >> at >> >> java.util.AbstractCollection.toString(AbstractCollection.java:450) >> >> at java.lang.String.valueOf(String.java:2854) >> >> at java.lang.StringBuilder.append(StringBuilder.java:128) >> >> at >> >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) >> >> at >> >> >> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) >> >> at >> >> >> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> >> at >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> >> at >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> >> >> >> >> >> >> >> >> >> Build Log: >> >> [...truncated 1672 lines...] >> >> [junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates >> >> [junit4] 2> 9 20, 0025 3:45:48 ?? >> >> >> >> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >> >> uncaughtException >> >> [junit4] 2> WARNING: Uncaught exception in thread: >> >> Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates] >> >> [junit4] 2> java.lang.OutOfMemoryError: Java heap space >> >> [junit4] 2> at >> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> >> [junit4] 2> at java.util.Arrays.copyOf(Arrays.java:2367) >> >> [junit4] 2> at >> >> >> >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) >> >> [junit4] 2> at >> >> >> >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) >> >> [junit4] 2> at >> >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >> >> [junit4] 2> at >> >> java.lang.StringBuilder.append(StringBuilder.java:132) >> >> [junit4] 2> at >> >> java.lang.StringBuilder.append(StringBuilder.java:128) >> >> [junit4] 2> at >> >> java.util.AbstractCollection.toString(AbstractCollection.java:450) >> >> [junit4] 2> at java.lang.String.valueOf(String.java:2854) >> >> [junit4] 2> at >> >> java.lang.StringBuilder.append(StringBuilder.java:128) >> >> [junit4] 2> at >> >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> >> [junit4] 2> at >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> >> [junit4] 2> >> >> [junit4] 2> 9 20, 0025 3:47:09 ?? >> >> >> >> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >> >> uncaughtException >> >> [junit4] 2> WARNING: Uncaught exception in thread: >> >> Thread[UpdateThread-3,5,TGRP-TestNumericDocValuesUpdates] >> >> [junit4] 2> java.lang.IllegalStateException: this writer hit an >> >> OutOfMemoryError; cannot commit >> >> [junit4] 2> at >> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> >> [junit4] 2> at >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> >> [junit4] 2> >> >> [junit4] 2> 9 20, 0025 3:47:32 ?? >> >> >> >> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler >> >> uncaughtException >> >> [junit4] 2> WARNING: Uncaught exception in thread: >> >> Thread[UpdateThread-8,5,TGRP-TestNumericDocValuesUpdates] >> >> [junit4] 2> java.lang.IllegalStateException: this writer hit an >> >> OutOfMemoryError; cannot commit >> >> [junit4] 2> at >> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> >> [junit4] 2> at >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> >> [junit4] 2> at >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> >> [junit4] 2> >> >> [junit4] 2> NOTE: download the large Jenkins line-docs file by >> >> running 'ant get-jenkins-line-docs' in the lucene directory. >> >> [junit4] 2> NOTE: reproduce with: ant test >> >> -Dtestcase=TestNumericDocValuesUpdates >> >> -Dtests.method=testStressMultiThreading -Dtests.seed=EA6A02F6820CB4B7 >> >> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true >> >> -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt >> >> -Dtests.locale=ja_JP_JP_#u-ca-japanese -Dtests.timezone=Asia/Damascus >> >> -Dtests.file.encoding=US-ASCII >> >> [junit4] ERROR 1336s J1 | >> >> TestNumericDocValuesUpdates.testStressMultiThreading <<< >> >> [junit4] > Throwable #1: >> >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an >> >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1, >> >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> >> [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space >> >> [junit4] > at >> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> >> [junit4] > at java.util.Arrays.copyOf(Arrays.java:2367) >> >> [junit4] > at >> >> >> >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) >> >> [junit4] > at >> >> >> >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) >> >> [junit4] > at >> >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >> >> [junit4] > at >> >> java.lang.StringBuilder.append(StringBuilder.java:132) >> >> [junit4] > at >> >> java.lang.StringBuilder.append(StringBuilder.java:128) >> >> [junit4] > at >> >> java.util.AbstractCollection.toString(AbstractCollection.java:450) >> >> [junit4] > at java.lang.String.valueOf(String.java:2854) >> >> [junit4] > at >> >> java.lang.StringBuilder.append(StringBuilder.java:128) >> >> [junit4] > at >> >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> >> [junit4] > at >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable >> >> #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured >> >> an >> >> uncaught exception in thread: Thread[id=3132, name=UpdateThread-3, >> >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> >> [junit4] > Caused by: java.lang.IllegalStateException: this >> >> writer >> >> hit an OutOfMemoryError; cannot commit >> >> [junit4] > at >> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> >> [junit4] > at >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)Throwable >> >> #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured >> >> an >> >> uncaught exception in thread: Thread[id=3137, name=UpdateThread-8, >> >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates] >> >> [junit4] > Caused by: java.lang.IllegalStateException: this >> >> writer >> >> hit an OutOfMemoryError; cannot commit >> >> [junit4] > at >> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2750) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922) >> >> [junit4] > at >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897) >> >> [junit4] > at >> >> >> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957) >> >> [junit4] 2> NOTE: test params are: codec=Lucene45: >> >> {ndv=PostingsFormat(name=Memory doPackFST= false), >> >> f=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> >> id=PostingsFormat(name=SimpleText), >> >> dvUpdateKey=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> >> foo=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> >> updKey=PostingsFormat(name=Memory doPackFST= false), >> >> k1=PostingsFormat(name=TestBloomFilteredLucene41Postings), >> >> key=PostingsFormat(name=SimpleText), >> >> k2=PostingsFormat(name=SimpleText)}, >> >> docValues:{val=DocValuesFormat(name=Memory), >> >> f6=DocValuesFormat(name=Lucene45), f7=DocValuesFormat(name=Memory), >> >> f8=DocValuesFormat(name=SimpleText), f9=DocValuesFormat(name=Disk), >> >> f12=DocValuesFormat(name=Disk), f11=DocValuesFormat(name=SimpleText), >> >> f10=DocValuesFormat(name=Memory), f16=DocValuesFormat(name=Disk), >> >> f15=DocValuesFormat(name=SimpleText), f14=DocValuesFormat(name=Memory), >> >> f13=DocValuesFormat(name=Lucene45), >> >> ssdv=DocValuesFormat(name=SimpleText), >> >> f17=DocValuesFormat(name=Lucene45), >> >> cf14=DocValuesFormat(name=SimpleText), >> >> ndv=DocValuesFormat(name=Lucene45), cf15=DocValuesFormat(name=Disk), >> >> cf12=DocValuesFormat(name=Lucene45), cf13=DocValuesFormat(name=Memory), >> >> cf10=DocValuesFormat(name=SimpleText), cf11=DocValuesFormat(name=Disk), >> >> ndv1=DocValuesFormat(name=Memory), f1=DocValuesFormat(name=Disk), >> >> ndv2=DocValuesFormat(name=SimpleText), >> >> f0=DocValuesFormat(name=SimpleText), >> >> f3=DocValuesFormat(name=Memory), f2=DocValuesFormat(name=Lucene45), >> >> bdv=DocValuesFormat(name=Lucene45), >> >> cf16=DocValuesFormat(name=Lucene45), >> >> f5=DocValuesFormat(name=Disk), cf17=DocValuesFormat(name=Memory), >> >> f4=DocValuesFormat(name=SimpleText), >> >> f=DocValuesFormat(name=SimpleText), >> >> sdv=DocValuesFormat(name=Memory), cf5=DocValuesFormat(name=SimpleText), >> >> cf6=DocValuesFormat(name=Disk), cf3=DocValuesFormat(name=Lucene45), >> >> cf4=DocValuesFormat(name=Memory), cf9=DocValuesFormat(name=SimpleText), >> >> cf7=DocValuesFormat(name=Lucene45), cf8=DocValuesFormat(name=Memory), >> >> cf2=DocValuesFormat(name=Disk), cf1=DocValuesFormat(name=SimpleText), >> >> cf0=DocValuesFormat(name=Memory), cf=DocValuesFormat(name=Disk)}, >> >> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, >> >> locale=ja_JP_JP_#u-ca-japanese, timezone=Asia/Damascus >> >> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation >> >> 1.7.0_25 (64-bit)/cpus=16,threads=1,free=203415056,total=515244032 >> >> [junit4] 2> NOTE: All tests run in this JVM: [TestIndexSearcher, >> >> TestNearSpansOrdered, TestAttributeSource, Nested1, TestRegexpRandom2, >> >> TestQueryWrapperFilter, TestToken, TestByteBlockPool, >> >> TestWorstCaseTestBehavior, TestLongPostings, >> >> TestSimpleSearchEquivalence, >> >> TestCollectionUtil, TestTerm, TestBoolean2, TestDirectory, >> >> TestLazyProxSkipping, NestedSetupChain, NestedTeardownChain, >> >> TestMultiThreadTermVectors, TestCharFilter, TestSpansAdvanced, >> >> TestIndexWriterMergePolicy, TestVersion, TestMinimize, TestUnicodeUtil, >> >> TestPerFieldPostingsFormat, TestPerFieldDocValuesFormat, >> >> TestPerSegmentDeletes, TestDoc, TestSortDocValues, TestDuelingCodecs, >> >> TestSegmentMerger, TestControlledRealTimeReopenThread, TestCharsRef, >> >> TestGraphTokenizers, TestIntBlockPool, TestIndexInput, >> >> TestRamUsageEstimatorOnWildAnimals, TestNeverDelete, >> >> TestIndexWriterNRTIsCurrent, TestSpanExplanations, TestRAMDirectory, >> >> TestPositiveScoresOnlyCollector, Nested, TestSizeBoundedForceMerge, >> >> TestDateSort, TestPayloadExplanations, TestNewestSegment, >> >> TestTotalHitCountCollector, Test4GBStoredFields, TestDocCount, >> >> TestPackedInts, TestFieldValueFilter, TestCachingWrapperFilter, >> >> TestDocsAndPositions, TestNumericUtils, TestElevationComparator, >> >> TestIndexFileDeleter, TestLucene40StoredFieldsFormat, >> >> TestDocumentWriter, >> >> TestOpenBitSet, TestMultiDocValues, Test2BDocs, TestTermVectors, >> >> TestDocValuesScoring, TestWindowsMMap, TestFlex, >> >> TestBlockPostingsFormat2, >> >> TestIndexWriterUnicode, TestMultiMMap, TestPayloadNearQuery, >> >> TestSnapshotDeletionPolicy, TestFuzzyQuery, TestTermRangeQuery, Nested, >> >> TestTieredMergePolicy, TestAtomicUpdate, TestDateFilter, >> >> TestIdentityHashSet, TestIntsRef, TestIndexWriterOutOfFileDescriptors, >> >> TestParallelCompositeReader, TestFieldCache, TestBufferedIndexInput, >> >> TestIOUtils, TestBooleanQuery, TestPForDeltaDocIdSet, >> >> TestMultiLevelSkipList, TestCodecs, TestDocument, >> >> TestInPlaceMergeSorter, >> >> TestDocValuesFormat, TestNumericRangeQuery64, TestBytesStore, >> >> TestLucene42DocValuesFormat, TestShardSearching, TestSearcherManager, >> >> TestOmitNorms, TestFixedBitSet, TestBitVector, TestCustomSearcherSort, >> >> TestDocValuesIndexing, TestSimilarityBase, TestHugeRamFile, >> >> TestBooleanMinShouldMatch, TestDeterminizeLexicon, >> >> TestIndexWriterOnDiskFull, TestSameTokenSamePosition, >> >> TestBinaryDocument, >> >> TestAutomatonQueryUnicode, TestFieldCacheTermsFilter, Nested1, >> >> TestBooleanScorer, TestMockDirectoryWrapper, TestRamUsageEstimator, >> >> Nested1, >> >> Nested, TestNoMergePolicy, TestNamedSPILoader, TestPhrasePrefixQuery, >> >> TestDemo, TestAssertions, TestRollback, TestFieldInfos, >> >> TestTwoPhaseCommitTool, TestNot, TestExplanations, >> >> TestVersionComparator, >> >> TestIndexWriterOnJRECrash, TestPrefixFilter, TestCodecHoldsOpenFiles, >> >> TestReaderClosed, MultiCollectorTest, TestDocBoost, >> >> TestNGramPhraseQuery, >> >> TestTermdocPerf, Test2BPositions, TestIndexCommit, Test2BPostingsBytes, >> >> TestTopScoreDocCollector, TestByteArrayDataInput, >> >> TestNumericDocValuesUpdates] >> >> [junit4] Completed on J1 in 1346.04s, 24 tests, 1 error <<< >> >> FAILURES! >> >> >> >> [...truncated 16 lines...] >> >> BUILD FAILED >> >> >> >> >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:428: >> >> The following error occurred while executing this line: >> >> >> >> >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:401: >> >> The following error occurred while executing this line: >> >> >> >> >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: >> >> The following error occurred while executing this line: >> >> >> >> >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:49: >> >> The following error occurred while executing this line: >> >> >> >> >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1230: >> >> The following error occurred while executing this line: >> >> >> >> >> >> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:873: >> >> There were test failures: 368 suites, 2606 tests, 1 error, 59 ignored >> >> (46 >> >> assumptions) >> >> >> >> Total time: 53 minutes 6 seconds >> >> Build step 'Invoke Ant' marked build as failure >> >> Archiving artifacts >> >> Recording test results >> >> Email was triggered for: Failure >> >> Sending email for trigger: Failure >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
