I pushed a fix for this case as well as the other recent fail, which was due to an incorrect assertion added in a recent change on the random-access-vectors branch that was adding safety to the iterator used during merging, where we assume that only a single iterator will be used to access values at once. The previous assertion assumed that all vector values would be consumed, but during quantization we sample the vectors, iterating over all of them, but only retrieving the values for some, so the new safety check keeps track of the ords iterated over rather than the ords whose values are retrieved.
On Sat, Sep 28, 2024 at 5:15 PM Michael Sokolov <[email protected]> wrote: > > These failures relate to the way Arrays.binarySearch works when there > are repeated values, in which case the result is undefined (it can be > any of the indexes with the value), but in SlowCompositeReaderWrapper > we are relying on finding the lowest-indexed of the repeats. I'll work > on a fix > > On Sat, Sep 28, 2024 at 3:32 PM Policeman Jenkins Server > <[email protected]> wrote: > > > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/50364/ > > Java: 64bit/hotspot/jdk-24-ea+16 -XX:-UseCompressedOops -XX:+UseParallelGC > > > > 2 tests failed. > > FAILED: > > org.apache.lucene.backward_codecs.lucene90.TestLucene90HnswVectorsFormat.testSparseVectors > > > > Error Message: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > > > Stack Trace: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:916) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:929) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1550) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1838) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1478) > > at > > app/[email protected]/org.apache.lucene.tests.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:224) > > at > > app/[email protected]/org.apache.lucene.tests.index.BaseKnnVectorsFormatTestCase.testSparseVectors(BaseKnnVectorsFormatTestCase.java:873) > > at > > java.base@24-ea/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > > at java.base@24-ea/java.lang.reflect.Method.invoke(Method.java:573) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) > > at > > app/[email protected]/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/[email protected]/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) > > at > > app/[email protected]/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > > at > > app/[email protected]/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) > > at > > app/[email protected]/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) > > at java.base@24-ea/java.lang.Thread.run(Thread.java:1576) > > Caused by: java.lang.NullPointerException: Cannot invoke > > "org.apache.lucene.index.FloatVectorValues.vectorValue(int)" because > > "this.subs[this.lastSubIndex].sub" is null > > at > > [email protected]/org.apache.lucene.index.SlowCompositeCodecReaderWrapper$SlowCompositeKnnVectorsReaderWrapper$MergedFloatVectorValues.vectorValue(SlowCompositeCodecReaderWrapper.java:905) > > at > > [email protected]/org.apache.lucene.index.SortingCodecReader$SortingFloatVectorValues.vectorValue(SortingCodecReader.java:324) > > at > > [email protected]/org.apache.lucene.codecs.KnnVectorsWriter$MergedVectorValues$MergedFloat32VectorValues.vectorValue(KnnVectorsWriter.java:364) > > at > > org.apache.lucene.backward_codecs.lucene90.Lucene90HnswVectorsWriter.writeVectorData(Lucene90HnswVectorsWriter.java:194) > > at > > org.apache.lucene.backward_codecs.lucene90.Lucene90HnswVectorsWriter.writeField(Lucene90HnswVectorsWriter.java:122) > > at > > [email protected]/org.apache.lucene.codecs.BufferingKnnVectorsWriter.mergeOneField(BufferingKnnVectorsWriter.java:203) > > at > > [email protected]/org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:121) > > at > > [email protected]/org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105) > > at > > [email protected]/org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271) > > at > > [email protected]/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) > > at > > [email protected]/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:158) > > at > > [email protected]/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5283) > > at > > [email protected]/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4749) > > at > > [email protected]/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6567) > > at > > [email protected]/org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668) > > at > > [email protected]/org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729) > > > > > > FAILED: > > org.apache.lucene.backward_codecs.lucene92.TestLucene92HnswVectorsFormat.testSparseVectors > > > > Error Message: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > > > Stack Trace: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:916) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:929) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1550) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1838) > > at > > app/[email protected]/org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1478) > > at > > app/[email protected]/org.apache.lucene.tests.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:224) > > at > > app/[email protected]/org.apache.lucene.tests.index.BaseKnnVectorsFormatTestCase.testSparseVectors(BaseKnnVectorsFormatTestCase.java:873) > > at > > java.base@24-ea/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > > at java.base@24-ea/java.lang.reflect.Method.invoke(Method.java:573) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) > > at > > app/[email protected]/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/[email protected]/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) > > at > > app/[email protected]/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > > at > > app/[email protected]/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/[email protected]/org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) > > at > > app/[email protected]/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) > > at java.base@24-ea/java.lang.Thread.run(Thread.java:1576) > > Caused by: java.lang.NullPointerException: Cannot invoke > > "org.apache.lucene.index.FloatVectorValues.vectorValue(int)" because > > "this.subs[this.lastSubIndex].sub" is null > > at > > [email protected]/org.apache.lucene.index.SlowCompositeCodecReaderWrapper$SlowCompositeKnnVectorsReaderWrapper$MergedFloatVectorValues.vectorValue(SlowCompositeCodecReaderWrapper.java:905) > > at > > [email protected]/org.apache.lucene.index.SortingCodecReader$SortingFloatVectorValues.vectorValue(SortingCodecReader.java:324) > > at > > [email protected]/org.apache.lucene.codecs.KnnVectorsWriter$MergedVectorValues$MergedFloat32VectorValues.vectorValue(KnnVectorsWriter.java:364) > > at > > org.apache.lucene.backward_codecs.lucene92.Lucene92HnswVectorsWriter.writeVectorData(Lucene92HnswVectorsWriter.java:197) > > at > > org.apache.lucene.backward_codecs.lucene92.Lucene92HnswVectorsWriter.writeField(Lucene92HnswVectorsWriter.java:129) > > at > > [email protected]/org.apache.lucene.codecs.BufferingKnnVectorsWriter.mergeOneField(BufferingKnnVectorsWriter.java:203) > > at > > [email protected]/org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:121) > > at > > [email protected]/org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105) > > at > > [email protected]/org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271) > > at > > [email protected]/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) > > at > > [email protected]/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:158) > > at > > [email protected]/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5283) > > at > > [email protected]/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4749) > > at > > [email protected]/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6567) > > at > > [email protected]/org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668) > > at > > [email protected]/org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729) > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
