I opened an issue: https://issues.apache.org/jira/browse/LUCENE-5903
It should not impact the release, as its an old situation and maybe nothing we can do to improve it, besides allowing MDW leniency for the case. On Sun, Aug 24, 2014 at 10:46 AM, Robert Muir <rcm...@gmail.com> wrote: > I think its a bug, and this was just exposed by randomization of > commitOnClose in IndexWriterConfig? > > we start a CMS merge for _8 segment (consumer creates _8.fdt and > _8.fdx), but this merge doesn't make it into the commit, and we > close() without waiting for merges. > this merge is aborted and indexfiledeleter cleans up _8.fdt and _8.fdx > > we start a new IW on the same dir, and the next segment it tries to > create is _8 (how else would it know?) which trips the double-write > logic in createOutput. > This logic currently only checks files that we ever created, and > doesn't check if we ever deleted the file. > We can fix mockdirectorywrapper, but I feel like this could be a real > problem on e.g. windows if the file was busy for both IFD runs. > > On Sun, Aug 24, 2014 at 10:00 AM, Robert Muir <rcm...@gmail.com> wrote: >> This will reproduce if you make the machine busy / beast it. >> >> I just tacked on -Dtests.dups=20 and it tripped once. I'll try to >> investigate... >> >> On Sun, Aug 24, 2014 at 9:42 AM, Apache Jenkins Server >> <jenk...@builds.apache.org> wrote: >>> Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4814/ >>> >>> 1 tests failed. >>> REGRESSION: >>> org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions >>> >>> Error Message: >>> file "_8.fdx" was already written to >>> >>> Stack Trace: >>> java.io.IOException: file "_8.fdx" was already written to >>> at >>> __randomizedtesting.SeedInfo.seed([CD1E08778E5DAA4A:9373618C3018B013]:0) >>> at >>> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:495) >>> at >>> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) >>> at >>> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:113) >>> at >>> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) >>> at >>> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:201) >>> at >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95) >>> at >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3993) >>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3588) >>> at >>> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) >>> at >>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1884) >>> at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1700) >>> at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1653) >>> at >>> org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:165) >>> at >>> org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions(TestBackwardsCompatibility.java:1045) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) >>> at >>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >>> at >>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>> at >>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>> at >>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >>> at >>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >>> at >>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) >>> at >>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) >>> at >>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >>> at >>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >>> at >>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >>> at >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>> at >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >>> at >>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >>> at >>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >>> at >>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >>> at >>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> >>> >>> Build Log: >>> [...truncated 889 lines...] >>> [junit4] Suite: org.apache.lucene.index.TestBackwardsCompatibility >>> [junit4] 2> NOTE: reproduce with: ant test >>> -Dtestcase=TestBackwardsCompatibility >>> -Dtests.method=testUpgradeOldSingleSegmentIndexWithAdditions >>> -Dtests.seed=CD1E08778E5DAA4A -Dtests.multiplier=3 -Dtests.slow=true >>> -Dtests.locale=da -Dtests.timezone=SystemV/MST7MDT >>> -Dtests.file.encoding=US-ASCII >>> [junit4] ERROR 0.08s J1 | >>> TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions <<< >>> [junit4] > Throwable #1: java.io.IOException: file "_8.fdx" was >>> already written to >>> [junit4] > at >>> __randomizedtesting.SeedInfo.seed([CD1E08778E5DAA4A:9373618C3018B013]:0) >>> [junit4] > at >>> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:495) >>> [junit4] > at >>> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) >>> [junit4] > at >>> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:113) >>> [junit4] > at >>> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) >>> [junit4] > at >>> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:201) >>> [junit4] > at >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95) >>> [junit4] > at >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3993) >>> [junit4] > at >>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3588) >>> [junit4] > at >>> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) >>> [junit4] > at >>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1884) >>> [junit4] > at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1700) >>> [junit4] > at >>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1653) >>> [junit4] > at >>> org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:165) >>> [junit4] > at >>> org.apache.lucene.index.TestBackwardsCompatibility.testUpgradeOldSingleSegmentIndexWithAdditions(TestBackwardsCompatibility.java:1045) >>> [junit4] > at java.lang.Thread.run(Thread.java:745) >>> [junit4] 2> NOTE: leaving temporary files on disk at: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/build/core/test/J1/./temp/lucene.index.TestBackwardsCompatibility-CD1E08778E5DAA4A-001 >>> [junit4] 2> NOTE: test params are: codec=Lucene410: {fie?ld=FST41, >>> id=FST41, autf8=PostingsFormat(name=Lucene41VarGapFixedInterval), >>> content5=PostingsFormat(name=MockRandom), >>> content3=PostingsFormat(name=Lucene41VarGapFixedInterval), >>> utf8=PostingsFormat(name=FSTOrdPulsing41), >>> trieLong=PostingsFormat(name=Lucene41VarGapFixedInterval), trieInt=FST41, >>> content6=FST41, content=PostingsFormat(name=FSTOrdPulsing41), >>> content2=FST41}, docValues:{dvByte=DocValuesFormat(name=Memory), >>> dvInt=DocValuesFormat(name=Lucene410), bdv2=DocValuesFormat(name=Memory), >>> dvSortedNumeric=DocValuesFormat(name=Memory), >>> bdv2_c=DocValuesFormat(name=Memory), ndv1=DocValuesFormat(name=Lucene410), >>> dvBytesDerefFixed=DocValuesFormat(name=Lucene410), >>> dvBytesSortedVar=DocValuesFormat(name=Asserting), >>> dvSortedSet=DocValuesFormat(name=Asserting), >>> dvDouble=DocValuesFormat(name=Asserting), >>> dvBytesStraightVar=DocValuesFormat(name=SimpleText), >>> ndv2_c=DocValuesFormat(name=Memory), ndv2=DocValuesFormat(name=Memory), >>> dvFloat=DocValuesFormat(name=Memory), dvLong=DocValuesFormat(name=Memory), >>> dvBytesSortedFixed=DocValuesFormat(name=Memory), >>> dvBytesStraightFixed=DocValuesFormat(name=Asserting), >>> dvPacked=DocValuesFormat(name=Memory), >>> bdv1_c=DocValuesFormat(name=Asserting), >>> bdv1=DocValuesFormat(name=Lucene410), >>> ndv1_c=DocValuesFormat(name=Asserting), >>> dvShort=DocValuesFormat(name=Memory), >>> dvBytesDerefVar=DocValuesFormat(name=Memory)}, >>> sim=RandomSimilarityProvider(queryNorm=true,coord=no): {fie?ld=DFR I(F)L2, >>> autf8=BM25(k1=1.2,b=0.75), content5=DFR G2, utf8=DFR I(ne)BZ(0.3), >>> content6=DFR GB2, content=IB SPL-D3(800.0), content2=DFR I(F)2}, locale=da, >>> timezone=SystemV/MST7MDT >>> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation >>> 1.7.0_60 (64-bit)/cpus=16,threads=1,free=229225600,total=384827392 >>> [junit4] 2> NOTE: All tests run in this JVM: [TestCollectionUtil, >>> TestPrefixCodedTerms, TestTopFieldCollector, TestBufferedChecksum, >>> TestMultiDocValues, TestTransactionRollback, TestNoMergeScheduler, >>> TestSortRandom, TestCharTermAttributeImpl, TestOmitPositions, >>> TestIndexWriterExceptions2, TestSubScorerFreqs, TestPhrasePrefixQuery, >>> TestDateFilter, TestLucene42NormsFormat, TestFieldReuse, >>> TestFilterAtomicReader, TestBinaryDocValuesUpdates, TestUniqueTermCount, >>> TestDirectPacked, TestSegmentMerger, TestTwoPhaseCommitTool, >>> TestRegexpQuery, TestParallelCompositeReader, TestMultiMMap, >>> TestPerFieldPostingsFormat, TestPayloadsOnVectors, Nested, TestLongBitSet, >>> TestIntsRef, TestAutomatonQuery, TestPayloads, NestedSetupChain, >>> NestedTeardownChain, TestRateLimitedDirectoryWrapper, TestFuzzyQuery, >>> TestCachingCollector, TestDocCount, TestDateTools, TestDeterminizeLexicon, >>> TestMergedIterator, TestDemo, TestSearchWithThreads, >>> TestDocTermOrdsRangeFilter, TestIndexWriterConfig, TestSearch, >>> TestAttributeSource, TestDocIdSet, TestDocInverterPerFieldErrorInfo, >>> MultiCollectorTest, TestBasics, TestStressDeletes, TestStressNRT, >>> TestSentinelIntSet, TestMathUtil, TestOpenBitSet, TestFlex, >>> TestLevenshteinAutomata, TestPrefixInBooleanQuery, TestScorerPerf, >>> TestEarlyTermination, TestPrefixFilter, TestRollback, >>> TestCompressingStoredFieldsFormat, TestIndexWriterWithThreads, >>> TestIndexableField, TestMmapDirectory, TestBlockPostingsFormat2, >>> InBeforeClass, InAfterClass, InTestMethod, NonStringProperties, >>> TestIndexWriterLockRelease, TestSnapshotDeletionPolicy, >>> TestQueryWrapperFilter, TestCodecHoldsOpenFiles, TestIntBlockPool, >>> TestFilteredQuery, TestDisjunctionMaxQuery, TestPostingsOffsets, TestTerm, >>> TestEliasFanoDocIdSet, FuzzyTermOnShortTermsTest, TestConstantScoreQuery, >>> TestWorstCaseTestBehavior, TestComplexExplanationsOfNonMatches, >>> TestIndexReaderClose, TestNormsFormat, TestBackwardsCompatibility] >>> [junit4] Completed on J1 in 21.56s, 15 tests, 1 error <<< FAILURES! >>> >>> [...truncated 938 lines...] >>> BUILD FAILED >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:485: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:465: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:48: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/extra-targets.xml:37: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/build.xml:49: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1342: >>> The following error occurred while executing this line: >>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:957: >>> There were test failures: 406 suites, 3522 tests, 1 error, 124 ignored >>> (113 assumptions) >>> >>> Total time: 28 minutes 59 seconds >>> Build step 'Invoke Ant' marked build as failure >>> Archiving artifacts >>> Sending artifact delta relative to Lucene-Solr-Tests-trunk-Java7 #4806 >>> Archived 3 artifacts >>> Archive block size is 32768 >>> Received 0 blocks and 4580211 bytes >>> Compression is 0.0% >>> Took 1.1 sec >>> Recording test results >>> Email was triggered for: Failure >>> Sending email for trigger: Failure >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
