[
https://issues.apache.org/jira/browse/LUCENE-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenczi Jim updated LUCENE-7569:
---------------------------------
Attachment: LUCENE-7569.patch
It turns out that the problem is a just a test bug related to
MockRandomMergePolicy. This merge policy is only used in tests and randomly
wraps the reader to be merged in a SlowCodecReaderWrapper in order to
deactivate the bulk merging. The test pass if I add a MergeReaderWrapper around
the original reader. This makes index sorting happy since the docvalues
instances are now re-created each time.
Bottom line is that merging of multi-valued docvalues works fine even when
index sorting is on except in these tests where the MockRandomMergePolicy
disables the bulk merge.
I've attached a patch that fixes this bug and the other test bug on
testNumericAlreadySorted. The patch is for the master branch (since
testNumericAlreadySorted should also fail on this branch) but the backport
should be straightforward.
> TestIndexSorting failures
> -------------------------
>
> Key: LUCENE-7569
> URL: https://issues.apache.org/jira/browse/LUCENE-7569
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Steve Rowe
> Assignee: Michael McCandless
> Priority: Blocker
> Fix For: master (7.0), 6.4
>
> Attachments: LUCENE-7569.patch
>
>
> My Jenkins found two reproducing seeds on branch_6x - these look different,
> but the failures happened on consecutive nightly runs:
> {noformat}
> Checking out Revision 535bf59a3b239f5c7bcd8c00f3e452c9b5e9b539
> (refs/remotes/origin/branch_6x)
> [...]
> [junit4] Suite: org.apache.lucene.index.TestIndexSorting
> [junit4] 2> ??? 18, 2016 9:50:39 AM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> [junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge
> Thread #0,5,TGRP-TestIndexSorting]
> [junit4] 2> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.AssertionError: nextValue=4594289799775307848 vs
> previous=4606302611760746829
> [junit4] 2> at
> __randomizedtesting.SeedInfo.seed([5F8898DCABBFD056]:0)
> [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
> [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
> [junit4] 2> Caused by: java.lang.AssertionError:
> nextValue=4594289799775307848 vs previous=4606302611760746829
> [junit4] 2> at
> org.apache.lucene.codecs.asserting.AssertingDocValuesFormat$AssertingDocValuesConsumer.addSortedNumericField(AssertingDocValuesFormat.java:152)
> [junit4] 2> at
> org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:470)
> [junit4] 2> at
> org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:243)
> [junit4] 2> at
> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:153)
> [junit4] 2> at
> org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:167)
> [junit4] 2> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:111)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4320)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3897)
> [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
> [junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
> [junit4] 2>
> [junit4] 2> NOTE: download the large Jenkins line-docs file by running
> 'ant get-jenkins-line-docs' in the lucene directory.
> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexSorting
> -Dtests.method=testRandom3 -Dtests.seed=5F8898DCABBFD056 -Dtests.multiplier=2
> -Dtests.nightly=true -Dtests.slow=true
> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt
> -Dtests.locale=he -Dtests.timezone=Canada/Central -Dtests.asserts=true
> -Dtests.file.encoding=ISO-8859-1
> [junit4] ERROR 14.9s J5 | TestIndexSorting.testRandom3 <<<
> [junit4] > Throwable #1:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> [junit4] > at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:748)
> [junit4] > at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:762)
> [junit4] > at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1566)
> [junit4] > at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1315)
> [junit4] > at
> org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2019)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] > Caused by: java.lang.AssertionError:
> nextValue=4594289799775307848 vs previous=4606302611760746829
> [junit4] > at
> org.apache.lucene.codecs.asserting.AssertingDocValuesFormat$AssertingDocValuesConsumer.addSortedNumericField(AssertingDocValuesFormat.java:152)
> [junit4] > at
> org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:470)
> [junit4] > at
> org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:243)
> [junit4] > at
> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:153)
> [junit4] > at
> org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:167)
> [junit4] > at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:111)
> [junit4] > at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4320)
> [junit4] > at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3897)
> [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
> [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)Throwable
> #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=1115, name=Lucene Merge Thread #0,
> state=RUNNABLE, group=TGRP-TestIndexSorting]
> [junit4] > Caused by:
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError:
> nextValue=4594289799775307848 vs previous=4606302611760746829
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([5F8898DCABBFD056]:0)
> [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
> [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
> [junit4] > Caused by: java.lang.AssertionError:
> nextValue=4594289799775307848 vs previous=4606302611760746829
> [junit4] > at
> org.apache.lucene.codecs.asserting.AssertingDocValuesFormat$AssertingDocValuesConsumer.addSortedNumericField(AssertingDocValuesFormat.java:152)
> [junit4] > at
> org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:470)
> [junit4] > at
> org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:243)
> [junit4] > at
> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:153)
> [junit4] > at
> org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:167)
> [junit4] > at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:111)
> [junit4] > at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4320)
> [junit4] > at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3897)
> [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
> [junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
> [junit4] 2> NOTE: leaving temporary files on disk at:
> /var/lib/jenkins/jobs/Lucene-Solr-Nightly-6.x/workspace/lucene/build/core/test/J5/temp/lucene.index.TestIndexSorting_5F8898DCABBFD056-001
> [junit4] 2> NOTE: test params are: codec=Asserting(Lucene62):
> {docs=PostingsFormat(name=Direct), norms=PostingsFormat(name=Direct),
> positions=PostingsFormat(name=Memory doPackFST= false),
> id=PostingsFormat(name=Direct),
> term_vectors=PostingsFormat(name=LuceneFixedGap)},
> docValues:{multi_valued_long=DocValuesFormat(name=Lucene54),
> double=DocValuesFormat(name=Direct), foo=DocValuesFormat(name=Lucene54),
> numeric=DocValuesFormat(name=Direct),
> positions=DocValuesFormat(name=Lucene54),
> multi_valued_numeric=DocValuesFormat(name=Asserting),
> float=DocValuesFormat(name=Memory), int=DocValuesFormat(name=Asserting),
> long=DocValuesFormat(name=Memory), points=DocValuesFormat(name=Asserting),
> sorted=DocValuesFormat(name=Direct),
> multi_valued_double=DocValuesFormat(name=Asserting),
> docs=DocValuesFormat(name=Asserting),
> multi_valued_string=DocValuesFormat(name=Asserting),
> norms=DocValuesFormat(name=Asserting), bytes=DocValuesFormat(name=Asserting),
> binary=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Asserting),
> multi_valued_int=DocValuesFormat(name=Direct),
> multi_valued_bytes=DocValuesFormat(name=Direct),
> multi_valued_float=DocValuesFormat(name=Memory),
> term_vectors=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1312,
> maxMBSortInHeap=5.376962888308618,
> sim=RandomSimilarity(queryNorm=true,coord=yes): {positions=DFR I(ne)3(800.0),
> id=IB LL-L3(800.0), term_vectors=DFR I(ne)B3(800.0)}, locale=he,
> timezone=Canada/Central
> [junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=132261744,total=508559360
> [junit4] 2> NOTE: All tests run in this JVM: [TestFixedBitSet,
> TestSearchAfter, TestMinShouldMatch2, TestStopFilter, TestMmapDirectory,
> TestAutomaton, TestTransactions, TestMultiLevelSkipList, TestForUtil,
> TestFieldType, TestIntBlockPool, TestNumericRangeQuery64, TestBoolean2,
> Test2BPositions, TestDemo, TestSpanBoostQuery, TestSpanMultiTermQueryWrapper,
> TestCharsRefBuilder, TestIOUtils, TestAllFilesHaveChecksumFooter,
> TestMaxTermFrequency, TestConjunctionDISI, TestIndexWriterMerging,
> TestPackedInts, TestLucene50StoredFieldsFormat, TestGeoEncodingUtils,
> TestGeoUtils, Test2BPoints, Test2BSortedDocValuesFixedSorted, TestDocValues,
> TestExceedMaxTermLength, TestExitableDirectoryReader,
> TestFilterDirectoryReader, TestIndexSorting]
> [junit4] Completed [345/441 (1!)] on J5 in 39.46s, 43 tests, 1 error <<<
> FAILURES!
> {noformat}
> {noformat}
> Checking out Revision 500f6c7875be31c34ca68c58f850b7e64fd049a9
> (refs/remotes/origin/branch_6x)
> [...]
> [junit4] Suite: org.apache.lucene.index.TestIndexSorting
> [junit4] 2> NOTE: download the large Jenkins line-docs file by running
> 'ant get-jenkins-line-docs' in the lucene directory.
> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexSorting
> -Dtests.method=testRandom3 -Dtests.seed=5D1E2D97B0777D5C -Dtests.multiplier=2
> -Dtests.nightly=true -Dtests.slow=true
> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt
> -Dtests.locale=fi-FI -Dtests.timezone=America/Tegucigalpa
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> [junit4] ERROR 245s J3 | TestIndexSorting.testRandom3 <<<
> [junit4] > Throwable #1: java.lang.RuntimeException: segment has
> indexSort=<float: "float">! missingValue=0.38226813,<sortedset:
> "multi_valued_bytes">! selector=MIN,<int: "id"> but docID=75 sorts after
> docID=76
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([5D1E2D97B0777D5C:FFC6634DD485545A]:0)
> [junit4] > at
> org.apache.lucene.index.CheckIndex.testSort(CheckIndex.java:876)
> [junit4] > at
> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:756)
> [junit4] > at
> org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
> [junit4] > at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:865)
> [junit4] > at
> org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
> [junit4] > at
> org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
> [junit4] > at
> org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2087)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] 2> NOTE: leaving temporary files on disk at:
> /var/lib/jenkins/jobs/Lucene-Solr-Nightly-6.x/workspace/lucene/build/core/test/J3/temp/lucene.index.TestIndexSorting_5D1E2D97B0777D5C-001
> [junit4] 2> NOTE: test params are: codec=CheapBastard,
> sim=RandomSimilarity(queryNorm=true,coord=no): {positions=DFR I(F)BZ(0.3),
> id=IB SPL-DZ(0.3), term_vectors=DFR I(ne)3(800.0)}, locale=fi-FI,
> timezone=America/Tegucigalpa
> [junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=252779544,total=525336576
> [junit4] 2> NOTE: All tests run in this JVM: [TestBooleanRewrites,
> TestSearchForDuplicates, TestSwappedIndexFiles, TestClassicSimilarity,
> TestNeverDelete, TestAllFilesCheckIndexHeader, TestBooleanScorer,
> TestPayloads, TestDelegatingAnalyzerWrapper, TestWeakIdentityMap,
> TestRollingBuffer, TestParallelCompositeReader, TestExitableDirectoryReader,
> TestCollectionUtil, TestLegacyNumericUtils, TestBagOfPostings, TestNot,
> TestPhrasePrefixQuery, TestSimpleExplanationsWithFillerDocs,
> TestPriorityQueue, TestBlockPostingsFormat2, TestIntroSorter,
> FiniteStringsIteratorTest, TestIndexWriterReader, TestNumericRangeQuery64,
> TestDeletionPolicy, TestBooleanOr, TestSloppyPhraseQuery, TestNRTThreads,
> TestPolygon, TestPolygon2D, TestDocValues, TestFilterDirectoryReader,
> TestIndexSorting]
> [junit4] Completed [431/441 (1!)] on J3 in 263.65s, 43 tests, 1 error <<<
> FAILURES!
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
