Going with your suggestions, seems like we’d be wiping out all of the backward-codecs folder/package, is that correct ? Also, do we need to put in anything to ensure back-combat between 6x, and 7x?
-Anshum > On Jun 29, 2017, at 7:21 AM, Anshum Gupta <ansh...@apple.com> wrote: > > Thanks Adrien, I’d want to try and do this myself as long as you can validate > the correctness :). > > I’ll be working on this in a few hours and should have an update later today > and hopefully we’d wrap it up soon. > > -Anshum > > > >> On Jun 28, 2017, at 10:39 AM, Adrien Grand <jpou...@gmail.com >> <mailto:jpou...@gmail.com>> wrote: >> >> If you don't want to do it, I can do it tomorrow but if you'd like to give >> it a try I'd be happy to help if you need any guidance. >> >> Le mer. 28 juin 2017 à 19:38, Adrien Grand <jpou...@gmail.com >> <mailto:jpou...@gmail.com>> a écrit : >> Hi Anshum, >> >> This looks like a good start to me. You would also need to remove the 6.x >> version constants so that TestBackwardCompatibility does not think they are >> worth testing, as well as all codecs, postings formats and doc values >> formats that are defined in the lucene/backward-codecs module since they are >> only about 6.x codecs. >> >> Le mer. 28 juin 2017 à 09:57, Anshum Gupta <ansh...@apple.com >> <mailto:ansh...@apple.com>> a écrit : >> Thanks for confirming that Alan, I had similar thoughts but wasn’t sure. >> >> I don’t want to change anything that I’m not confident about so I’m just >> going to create remove those and commit it to my fork. If someone who’s >> confident agrees with what I’m doing, I’ll go ahead and make those changes >> to the upstream :). >> >> -Anshum >> >> >> >>> On Jun 28, 2017, at 12:54 AM, Alan Woodward <a...@flax.co.uk >>> <mailto:a...@flax.co.uk>> wrote: >>> >>> We don’t need to support lucene5x codecs in 7, so you should be able to >>> just remove those tests (and the the relevant packages from >>> backwards-codecs too), I think? >>> >>> >>>> On 28 Jun 2017, at 08:38, Anshum Gupta <ansh...@apple.com >>>> <mailto:ansh...@apple.com>> wrote: >>>> >>>> I tried to move forward to see this work before automatically computing >>>> the versions but I have about 30 odd failing test. I’ve made those changes >>>> and pushed to my local GitHub account in case you have the time to look: >>>> https://github.com/anshumg/lucene-solr >>>> <https://github.com/anshumg/lucene-solr> >>>> >>>> Here’s the build summary if that helps: >>>> >>>> [junit4] Tests with failures [seed: 31C3B60E557C7E14] (first 10 out of >>>> 31): >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testOutliers2 >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testShortRange >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewValues >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFullLongRange >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testRamBytesUsed >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewLargeValues >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testByteRange >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testLongRange >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene50.TestLucene50SegmentInfoFormat.testRandomExceptions >>>> [junit4] - >>>> org.apache.lucene.codecs.lucene62.TestLucene62SegmentInfoFormat.testRandomExceptions >>>> [junit4] >>>> [junit4] >>>> [junit4] JVM J0: 0.56 .. 9.47 = 8.91s >>>> [junit4] JVM J1: 0.56 .. 4.13 = 3.57s >>>> [junit4] JVM J2: 0.56 .. 47.28 = 46.73s >>>> [junit4] JVM J3: 0.56 .. 3.89 = 3.33s >>>> [junit4] Execution time total: 47 seconds >>>> [junit4] Tests summary: 8 suites, 215 tests, 30 errors, 1 failure, 24 >>>> ignored (24 assumptions) >>>> >>>> >>>> -Anshum >>>> >>>> >>>> >>>>> On Jun 27, 2017, at 4:15 AM, Adrien Grand <jpou...@gmail.com >>>>> <mailto:jpou...@gmail.com>> wrote: >>>>> >>>>> The test***BackwardCompatibility cases can be removed since they make >>>>> sure that Lucene 7 can read Lucene 6 norms, while Lucene 8 doesn't have >>>>> to be able to read Lucene 6 norms. >>>>> >>>>> TestSegmentInfos needs to be adapted to the new versions, we need to >>>>> replace 5 with 6 and 8 with 9. Maybe we should compute those numbers >>>>> automatically based on Version.LATEST.major so that it does not require >>>>> manual changes when moving to a new major version. That would give 5 -> >>>>> Version.LATEST.major-2 and 8 -> Version.LATEST.major+1. >>>>> >>>>> I can do those changes on Thursday if you don't feel comfortable doing >>>>> them. >>>>> >>>>> >>>>> >>>>> Le mar. 27 juin 2017 à 08:12, Anshum Gupta <ansh...@apple.com >>>>> <mailto:ansh...@apple.com>> a écrit : >>>>> Without making any changes at all and just bumping up the version, I hit >>>>> these errors when running the tests: >>>>> >>>>> [junit4] 2> NOTE: reproduce with: ant test >>>>> -Dtestcase=TestSegmentInfos -Dtests.method=testIllegalCreatedVersion >>>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>>> -Dtests.file.encoding=US-ASCII >>>>> [junit4] FAILURE 0.01s J0 | TestSegmentInfos.testIllegalCreatedVersion >>>>> <<< >>>>> [junit4] > Throwable #1: junit.framework.AssertionFailedError: >>>>> Expected exception IllegalArgumentException but no exception was thrown >>>>> [junit4] > at >>>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:CE340683BE44C211]:0) >>>>> [junit4] > at >>>>> org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2672) >>>>> [junit4] > at >>>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion(TestSegmentInfos.java:35) >>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>> [junit4] 2> NOTE: reproduce with: ant test >>>>> -Dtestcase=TestSegmentInfos -Dtests.method=testVersionsOneSegment >>>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>>> -Dtests.file.encoding=US-ASCII >>>>> [junit4] ERROR 0.00s J0 | TestSegmentInfos.testVersionsOneSegment <<< >>>>> [junit4] > Throwable #1: >>>>> org.apache.lucene.index.CorruptIndexException: segments file recorded >>>>> indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older >>>>> version=7.0.0 >>>>> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>>> [junit4] > at >>>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:A7477EE8875F2E36]:0) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>>> [junit4] > at >>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment(TestSegmentInfos.java:67) >>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>> [junit4] 2> NOTE: reproduce with: ant test >>>>> -Dtestcase=TestSegmentInfos -Dtests.method=testVersionsTwoSegments >>>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>>> -Dtests.file.encoding=US-ASCII >>>>> [junit4] ERROR 0.00s J0 | TestSegmentInfos.testVersionsTwoSegments >>>>> <<< >>>>> [junit4] > Throwable #1: >>>>> org.apache.lucene.index.CorruptIndexException: segments file recorded >>>>> indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older >>>>> version=7.0.0 >>>>> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>>> [junit4] > at >>>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:4EE9CC4194FBB648]:0) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>>> [junit4] > at >>>>> org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>>> [junit4] > at >>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments(TestSegmentInfos.java:96) >>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>> >>>>> On applying the patch here (https://pastebin.com/tM4Fpy1Q >>>>> <https://pastebin.com/tM4Fpy1Q>), I end up with the following errors: >>>>> >>>>> [junit4] Tests with failures [seed: 5B388AB1E2BEFF87]: >>>>> [junit4] - >>>>> org.apache.lucene.search.similarities.TestSimilarityBase.testLengthEncodingBackwardCompatibility >>>>> [junit4] - >>>>> org.apache.lucene.search.similarities.TestClassicSimilarity.testNormEncodingBackwardCompatibility >>>>> [junit4] - >>>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>>> [junit4] - >>>>> org.apache.lucene.search.similarities.TestBM25Similarity.testLengthEncodingBackwardCompatibility >>>>> >>>>> Any ideas on what I’m missing here? >>>>> >>>>> >>>>> -Anshum >>>>> >>>>> >>>>> >>>>>> On Jun 26, 2017, at 2:55 PM, Ryan Ernst <r...@iernst.net >>>>>> <mailto:r...@iernst.net>> wrote: >>>>>> >>>>>> After changing that constant check, do you get the same test failures? >>>>>> What are the actual failure messages? >>>>>> >>>>>> On Mon, Jun 26, 2017 at 10:38 AM Anshum Gupta <ans...@anshumgupta.net >>>>>> <mailto:ans...@anshumgupta.net>> wrote: >>>>>> Thanks Adrien, sadly, the tests aren't passing. That's the reason why I >>>>>> didn't push the changes. I'll see if someone else can help while you're >>>>>> away. >>>>>> >>>>>> -Anshum >>>>>> >>>>>> On Mon, Jun 26, 2017 at 9:55 AM Adrien Grand <jpou...@gmail.com >>>>>> <mailto:jpou...@gmail.com>> wrote: >>>>>> Your proposed change looks correct to me. As far as whether other >>>>>> changes are required, I'm currently traveling but can look when I'm back >>>>>> on Thursday. Feel free to push the branches if tests are passing, we can >>>>>> fix things later? >>>>>> >>>>>> Le lun. 26 juin 2017 à 07:13, Anshum Gupta <ans...@anshumgupta.net >>>>>> <mailto:ans...@anshumgupta.net>> a écrit : >>>>>> The only throw case that I think needs changing is in SegmentInfos.java >>>>>> L315 >>>>>> Version luceneVersion = Version.fromBits(input.readVInt(), >>>>>> input.readVInt(), input.readVInt()); >>>>>> if (luceneVersion.onOrAfter(Version.LUCENE_7_0_0) == false) { >>>>>> // TODO: should we check indexCreatedVersion instead? >>>>>> throw new IndexFormatTooOldException(input, "this index is too old >>>>>> (version: " + luceneVersion + ")"); >>>>>> } >>>>>> >>>>>> The Lucene version here should be LUCENE_7_0_0, instead of the original >>>>>> LUCENE_6_0_0. >>>>>> >>>>>> Is there anything else that's needed (and is this the correct change?). >>>>>> >>>>>> -Anshum >>>>>> >>>>>> >>>>>> On Sun, Jun 25, 2017 at 9:45 PM Anshum Gupta <ans...@anshumgupta.net >>>>>> <mailto:ans...@anshumgupta.net>> wrote: >>>>>> Hi, >>>>>> >>>>>> I am working on creating the 7x, and 7.0 branches but I have the >>>>>> following failing tests: >>>>>> [junit4] Tests with failures [seed: 4FBDDCD3F96316D3]: >>>>>> [junit4] - >>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments >>>>>> [junit4] - >>>>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>>>> [junit4] - >>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment >>>>>> >>>>>> I assume this is a result of me not doing anything about the following >>>>>> TODO that got printed when I ran addVersion.py. >>>>>> TODO >>>>>> - Update IndexFormatTooOldException throw cases >>>>>> >>>>>> Can someone shed more light on what needs to be done here? >>>>>> >>>>>> -Anshum >>>>> >>>> >>> >> >
