I can't reproduce this, even when looping, but your analysis seems correct, neoremind. I filed this -
https://github.com/apache/lucene/pull/15957 Dawid On Wed, Apr 8, 2026 at 7:17 AM xu <[email protected]> wrote: > I think the merge itself isn't actually getting skipped, it's a > thread-safety issue in the test. > The test collects InfoStream messages into a ArrayList<String>, but with > ConcurrentMergeScheduler, the "merged new segment" message gets written by > the merge background thread. One thread writes to a non-thread-safe list, > the other test thread reads, oops, visibility problem in JMM. To address, > simply change: > > List<String> infoStream = new ArrayList<>(); > > to: > > List<String> infoStream = Collections.synchronizedList(new ArrayList<>()); > > I can reproduce this by putting the test in a for-loop running 10k times, > it shows roughly 0.3% chance of failure. > > To verify the theory, I quickly create a JCStress test (a dummy one > though, but it illustrates the point). It shows the write from one thread > is completely or partially invisible to the reader thread: > > @JCStressTest > @State > @Outcome(id = "1, 42", expect = Expect.ACCEPTABLE, desc = "Fully visible > write.") > @Outcome(id = "0, -1", expect = Expect.ACCEPTABLE, desc = "Write not > visible at all.") > @Outcome( expect = Expect.ACCEPTABLE_INTERESTING, desc = > "Partial / stale visibility.") > public class ArrayListVisibilityTest { > > final List<Integer> list = new ArrayList<>(); > > @Actor > public void writer() { > list.add(42); > } > > @Actor > public void reader(II_Result r) { > int size = list.size(); > r.r1 = size; > try { > r.r2 = size > 0 ? list.get(0) : -1; > } catch (Exception e) { > // ArrayIndexOutOfBoundsException is possible under races > r.r2 = -2; > } > } > } > > ``` > RESULT SAMPLES FREQ EXPECT DESCRIPTION > 0, -1 1,838,722,183 67.78% Acceptable Write not visible at > all. > 1, -2 106,178 <0.01% Interesting Partial / stale > visibility. > 1, 42 873,770,813 32.21% Acceptable Fully visible write. > ``` > In practice, a lock (or a memory barrier) would guarantee visibility, in > essence, the merge thread's CPU flushes writes to store buffer, and the > test thread's CPU is able to snoop the invalid to get the freshly added > message (at least this should be how things work on x86). > > Thanks, > neoremind > > Michael Sokolov <[email protected]> 于2026年4月6日周一 09:52写道： > >> I just added this TestInfoStream.testMergeSmallIndex test so I thought >> I'd try to reproduce, but it didn't repro for me. The failure >> indicates that no segment merge was logged. I'm not sure how it could >> get skipped though. >> >> On Sun, Apr 5, 2026 at 1:57 PM Policeman Jenkins Server via builds >> <[email protected]> wrote: >> > >> > Build: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Linux/5230/ >> > Java: 64bit/hotspot/jdk-27-ea+14 -XX:-UseCompressedOops -XX:+UseZGC >> > >> > 3 tests failed. >> > FAILED: org.apache.lucene.index.TestInfoStream.testMergeSmallIndex >> > >> > Error Message: >> > java.lang.AssertionError: expected:<1> but was:<0> >> > >> > Stack Trace: >> > java.lang.AssertionError: expected:<1> but was:<0> >> > at >> __randomizedtesting.SeedInfo.seed([F77FD74A101B3B0B:45303098296E3A19]:0) >> > at org.junit.Assert.fail(Assert.java:89) >> > at org.junit.Assert.failNotEquals(Assert.java:835) >> > at org.junit.Assert.assertEquals(Assert.java:647) >> > at org.junit.Assert.assertEquals(Assert.java:633) >> > at >> org.apache.lucene.index.TestInfoStream.testMergeSmallIndex(TestInfoStream.java:151) >> > at >> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) >> > at java.base/java.lang.reflect.Method.invoke(Method.java:565) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) >> > at >> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> > at >> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> > at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:52) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) >> > at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) >> > at java.base/java.lang.Thread.run(Thread.java:1527) >> > >> > >> > FAILED: >> org.apache.lucene.tests.store.TestMockDirectoryWrapper.testIsLoaded >> > >> > Error Message: >> > java.lang.AssertionError >> > >> > Stack Trace: >> > java.lang.AssertionError >> > at >> __randomizedtesting.SeedInfo.seed([F77FD74A101B3B0B:D95E8D3E0CAB8C7C]:0) >> > at org.junit.Assert.fail(Assert.java:87) >> > at org.junit.Assert.assertTrue(Assert.java:42) >> > at org.junit.Assert.assertTrue(Assert.java:53) >> > at >> org.apache.lucene.tests.store.BaseDirectoryTestCase.testIsLoaded(BaseDirectoryTestCase.java:1663) >> > at >> org.apache.lucene.tests.store.BaseDirectoryTestCase.testIsLoaded(BaseDirectoryTestCase.java:1623) >> > at >> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) >> > at java.base/java.lang.reflect.Method.invoke(Method.java:565) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) >> > at >> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> > at >> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> > at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:52) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) >> > at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) >> > at java.base/java.lang.Thread.run(Thread.java:1527) >> > >> > >> > FAILED: >> org.apache.lucene.tests.store.TestSerializedIOCountingDirectory.testIsLoaded >> > >> > Error Message: >> > java.lang.AssertionError >> > >> > Stack Trace: >> > java.lang.AssertionError >> > at >> __randomizedtesting.SeedInfo.seed([F77FD74A101B3B0B:D95E8D3E0CAB8C7C]:0) >> > at org.junit.Assert.fail(Assert.java:87) >> > at org.junit.Assert.assertTrue(Assert.java:42) >> > at org.junit.Assert.assertTrue(Assert.java:53) >> > at >> org.apache.lucene.tests.store.BaseDirectoryTestCase.testIsLoaded(BaseDirectoryTestCase.java:1663) >> > at >> org.apache.lucene.tests.store.BaseDirectoryTestCase.testIsLoaded(BaseDirectoryTestCase.java:1623) >> > at >> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) >> > at java.base/java.lang.reflect.Method.invoke(Method.java:565) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) >> > at >> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> > at >> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> > at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:52) >> > at >> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> > at >> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> > at >> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) >> > at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) >> > at java.base/java.lang.Thread.run(Thread.java:1527) >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > -- > > Best regards, > > Xu >
