Re: [JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4814 - Still Failing
This will reproduce if you make the machine busy / beast it. I just tacked on -Dtests.dups=20 and it tripped once. I'll try to investigate... On Sun, Aug 24, 2014 at 9:42 AM, Apache Jenkins Server jenk...@builds.apache.org wrote: Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4814/ 1 tests failed. REGRESSION: org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions Error Message: file _8.fdx was already written to Stack Trace: java.io.IOException: file _8.fdx was already written to at __randomizedtesting.SeedInfo.seed([CD1E08778E5DAA4A:9373618C3018B013]:0) at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:495) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.init(CompressingStoredFieldsWriter.java:113) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:201) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3993) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3588) at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1884) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1700) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1653) at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:165) at org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions(TestBackwardsCompatibility.java:1045) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at
Re: [JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4814 - Still Failing
I think its a bug, and this was just exposed by randomization of commitOnClose in IndexWriterConfig? we start a CMS merge for _8 segment (consumer creates _8.fdt and _8.fdx), but this merge doesn't make it into the commit, and we close() without waiting for merges. this merge is aborted and indexfiledeleter cleans up _8.fdt and _8.fdx we start a new IW on the same dir, and the next segment it tries to create is _8 (how else would it know?) which trips the double-write logic in createOutput. This logic currently only checks files that we ever created, and doesn't check if we ever deleted the file. We can fix mockdirectorywrapper, but I feel like this could be a real problem on e.g. windows if the file was busy for both IFD runs. On Sun, Aug 24, 2014 at 10:00 AM, Robert Muir rcm...@gmail.com wrote: This will reproduce if you make the machine busy / beast it. I just tacked on -Dtests.dups=20 and it tripped once. I'll try to investigate... On Sun, Aug 24, 2014 at 9:42 AM, Apache Jenkins Server jenk...@builds.apache.org wrote: Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4814/ 1 tests failed. REGRESSION: org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions Error Message: file _8.fdx was already written to Stack Trace: java.io.IOException: file _8.fdx was already written to at __randomizedtesting.SeedInfo.seed([CD1E08778E5DAA4A:9373618C3018B013]:0) at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:495) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.init(CompressingStoredFieldsWriter.java:113) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:201) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3993) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3588) at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1884) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1700) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1653) at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:165) at org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions(TestBackwardsCompatibility.java:1045) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at
Re: [JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4814 - Still Failing
I opened an issue: https://issues.apache.org/jira/browse/LUCENE-5903 It should not impact the release, as its an old situation and maybe nothing we can do to improve it, besides allowing MDW leniency for the case. On Sun, Aug 24, 2014 at 10:46 AM, Robert Muir rcm...@gmail.com wrote: I think its a bug, and this was just exposed by randomization of commitOnClose in IndexWriterConfig? we start a CMS merge for _8 segment (consumer creates _8.fdt and _8.fdx), but this merge doesn't make it into the commit, and we close() without waiting for merges. this merge is aborted and indexfiledeleter cleans up _8.fdt and _8.fdx we start a new IW on the same dir, and the next segment it tries to create is _8 (how else would it know?) which trips the double-write logic in createOutput. This logic currently only checks files that we ever created, and doesn't check if we ever deleted the file. We can fix mockdirectorywrapper, but I feel like this could be a real problem on e.g. windows if the file was busy for both IFD runs. On Sun, Aug 24, 2014 at 10:00 AM, Robert Muir rcm...@gmail.com wrote: This will reproduce if you make the machine busy / beast it. I just tacked on -Dtests.dups=20 and it tripped once. I'll try to investigate... On Sun, Aug 24, 2014 at 9:42 AM, Apache Jenkins Server jenk...@builds.apache.org wrote: Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4814/ 1 tests failed. REGRESSION: org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions Error Message: file _8.fdx was already written to Stack Trace: java.io.IOException: file _8.fdx was already written to at __randomizedtesting.SeedInfo.seed([CD1E08778E5DAA4A:9373618C3018B013]:0) at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:495) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.init(CompressingStoredFieldsWriter.java:113) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:201) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3993) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3588) at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1884) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1700) at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1653) at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:165) at org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions(TestBackwardsCompatibility.java:1045) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at
