Removing most backward codecs sounds good to me since the only codec that 8.0 needs to be able to read so far is the 7.0 codec which is in core. It looks like you removed the code, but you also need to remove their declaration in META-INF/services or the SPI loaded will try to load them and fail since it cannot find the class.
Backcompat indexes will be added as we perform 7.x releases. However you'd need to rename the 6.x indices from index.6.x.x to unsupported.6.x.x. We have some specific tests like "moreterms" and "dvupdates". I think we need to disable them for now and make sure to reenable them once 7.0 is released. I think the changes you did in SegmentInfos.java <https://github.com/anshumg/lucene-solr/commit/c7f5b9f9fa94fe1a6e6abc7ffc74fc3df16293a4#diff-e3ccf9ee90355b10f2dd22ce2da6c73c> are not necessary. It looks like the version numbers are related to the current major, but it is actually due to the fact that 7.0 is the first version to record the version that was used at creation time. I think you can revert changes in this file entirely. In the testIllegalCreatedVersion test, I'd just replace 8 with 9 or Version.CURRENT.major + 1. We'd need to remove the compatibility layer in similarities but it can be done as a follow-up. Thanks for taking care of this! Le jeu. 29 juin 2017 à 23:12, Anshum Gupta <ansh...@apple.com> a écrit : > Adrien, I’ve pushed some more changes and seems like I’d have to > regenerate some test indexes but I’m not sure how to do that. Do you mind > taking a look at this in it’s current form, and also my commits? It is all > @ my fork here: https://github.com/anshumg/lucene-solr > > P.S: I thought it’d make more sense to do this on a feature-branch but the > upgrade script wasn’t happy about that. > > -Anshum > > > > On Jun 29, 2017, at 9:20 AM, Anshum Gupta <ansh...@apple.com> wrote: > > Going with your suggestions, seems like we’d be wiping out all of the > backward-codecs folder/package, is that correct ? Also, do we need to put > in anything to ensure back-combat between 6x, and 7x? > > -Anshum > > > > On Jun 29, 2017, at 7:21 AM, Anshum Gupta <ansh...@apple.com> wrote: > > Thanks Adrien, I’d want to try and do this myself as long as you can > validate the correctness :). > > I’ll be working on this in a few hours and should have an update later > today and hopefully we’d wrap it up soon. > > -Anshum > > > > On Jun 28, 2017, at 10:39 AM, Adrien Grand <jpou...@gmail.com> wrote: > > If you don't want to do it, I can do it tomorrow but if you'd like to give > it a try I'd be happy to help if you need any guidance. > > Le mer. 28 juin 2017 à 19:38, Adrien Grand <jpou...@gmail.com> a écrit : > >> Hi Anshum, >> >> This looks like a good start to me. You would also need to remove the 6.x >> version constants so that TestBackwardCompatibility does not think they are >> worth testing, as well as all codecs, postings formats and doc values >> formats that are defined in the lucene/backward-codecs module since they >> are only about 6.x codecs. >> >> Le mer. 28 juin 2017 à 09:57, Anshum Gupta <ansh...@apple.com> a écrit : >> >>> Thanks for confirming that Alan, I had similar thoughts but wasn’t sure. >>> >>> I don’t want to change anything that I’m not confident about so I’m just >>> going to create remove those and commit it to my fork. If someone who’s >>> confident agrees with what I’m doing, I’ll go ahead and make those changes >>> to the upstream :). >>> >>> -Anshum >>> >>> >>> >>> On Jun 28, 2017, at 12:54 AM, Alan Woodward <a...@flax.co.uk> wrote: >>> >>> We don’t need to support lucene5x codecs in 7, so you should be able to >>> just remove those tests (and the the relevant packages from >>> backwards-codecs too), I think? >>> >>> >>> On 28 Jun 2017, at 08:38, Anshum Gupta <ansh...@apple.com> wrote: >>> >>> I tried to move forward to see this work before automatically computing >>> the versions but I have about 30 odd failing test. I’ve made those changes >>> and pushed to my local GitHub account in case you have the time to look: >>> https://github.com/anshumg/lucene-solr >>> >>> Here’s the build summary if that helps: >>> >>> [junit4] Tests with failures [seed: 31C3B60E557C7E14] (first 10 out >>> of 31): >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testOutliers2 >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testShortRange >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewValues >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFullLongRange >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testRamBytesUsed >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewLargeValues >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testByteRange >>> [junit4] - >>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testLongRange >>> [junit4] - >>> org.apache.lucene.codecs.lucene50.TestLucene50SegmentInfoFormat.testRandomExceptions >>> [junit4] - >>> org.apache.lucene.codecs.lucene62.TestLucene62SegmentInfoFormat.testRandomExceptions >>> [junit4] >>> [junit4] >>> [junit4] JVM J0: 0.56 .. 9.47 = 8.91s >>> [junit4] JVM J1: 0.56 .. 4.13 = 3.57s >>> [junit4] JVM J2: 0.56 .. 47.28 = 46.73s >>> [junit4] JVM J3: 0.56 .. 3.89 = 3.33s >>> [junit4] Execution time total: 47 seconds >>> [junit4] Tests summary: 8 suites, 215 tests, 30 errors, 1 failure, 24 >>> ignored (24 assumptions) >>> >>> >>> -Anshum >>> >>> >>> >>> On Jun 27, 2017, at 4:15 AM, Adrien Grand <jpou...@gmail.com> wrote: >>> >>> The test***BackwardCompatibility cases can be removed since they make >>> sure that Lucene 7 can read Lucene 6 norms, while Lucene 8 doesn't have to >>> be able to read Lucene 6 norms. >>> >>> TestSegmentInfos needs to be adapted to the new versions, we need to >>> replace 5 with 6 and 8 with 9. Maybe we should compute those numbers >>> automatically based on Version.LATEST.major so that it does not require >>> manual changes when moving to a new major version. That would give 5 -> >>> Version.LATEST.major-2 and 8 -> Version.LATEST.major+1. >>> >>> I can do those changes on Thursday if you don't feel comfortable doing >>> them. >>> >>> >>> >>> Le mar. 27 juin 2017 à 08:12, Anshum Gupta <ansh...@apple.com> a écrit : >>> >>>> Without making any changes at all and just bumping up the version, I >>>> hit these errors when running the tests: >>>> >>>> [junit4] 2> NOTE: reproduce with: ant test >>>> -Dtestcase=TestSegmentInfos -Dtests.method=testIllegalCreatedVersion >>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>> -Dtests.file.encoding=US-ASCII >>>> [junit4] FAILURE 0.01s J0 | >>>> TestSegmentInfos.testIllegalCreatedVersion <<< >>>> [junit4] > Throwable #1: junit.framework.AssertionFailedError: >>>> Expected exception IllegalArgumentException but no exception was thrown >>>> [junit4] > at >>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:CE340683BE44C211]:0) >>>> [junit4] > at >>>> org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2672) >>>> [junit4] > at >>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion(TestSegmentInfos.java:35) >>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>> [junit4] 2> NOTE: reproduce with: ant test >>>> -Dtestcase=TestSegmentInfos -Dtests.method=testVersionsOneSegment >>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>> -Dtests.file.encoding=US-ASCII >>>> [junit4] ERROR 0.00s J0 | TestSegmentInfos.testVersionsOneSegment >>>> <<< >>>> [junit4] > Throwable #1: >>>> org.apache.lucene.index.CorruptIndexException: segments file recorded >>>> indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older version=7.0.0 >>>> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>> [junit4] > at >>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:A7477EE8875F2E36]:0) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>> [junit4] > at >>>> org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment(TestSegmentInfos.java:67) >>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>> [junit4] 2> NOTE: reproduce with: ant test >>>> -Dtestcase=TestSegmentInfos -Dtests.method=testVersionsTwoSegments >>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>> -Dtests.file.encoding=US-ASCII >>>> [junit4] ERROR 0.00s J0 | TestSegmentInfos.testVersionsTwoSegments >>>> <<< >>>> [junit4] > Throwable #1: >>>> org.apache.lucene.index.CorruptIndexException: segments file recorded >>>> indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older version=7.0.0 >>>> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>> [junit4] > at >>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:4EE9CC4194FBB648]:0) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>> [junit4] > at >>>> org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>> [junit4] > at >>>> org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments(TestSegmentInfos.java:96) >>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>> >>>> On applying the patch here (https://pastebin.com/tM4Fpy1Q), I end up >>>> with the following errors: >>>> >>>> [junit4] Tests with failures [seed: 5B388AB1E2BEFF87]: >>>> [junit4] - >>>> org.apache.lucene.search.similarities.TestSimilarityBase.testLengthEncodingBackwardCompatibility >>>> [junit4] - >>>> org.apache.lucene.search.similarities.TestClassicSimilarity.testNormEncodingBackwardCompatibility >>>> [junit4] - >>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>> [junit4] - >>>> org.apache.lucene.search.similarities.TestBM25Similarity.testLengthEncodingBackwardCompatibility >>>> >>>> Any ideas on what I’m missing here? >>>> >>>> >>>> -Anshum >>>> >>>> >>>> >>>> On Jun 26, 2017, at 2:55 PM, Ryan Ernst <r...@iernst.net> wrote: >>>> >>>> After changing that constant check, do you get the same test failures? >>>> What are the actual failure messages? >>>> >>>> On Mon, Jun 26, 2017 at 10:38 AM Anshum Gupta <ans...@anshumgupta.net> >>>> wrote: >>>> >>>>> Thanks Adrien, sadly, the tests aren't passing. That's the reason why >>>>> I didn't push the changes. I'll see if someone else can help while you're >>>>> away. >>>>> >>>>> -Anshum >>>>> >>>>> On Mon, Jun 26, 2017 at 9:55 AM Adrien Grand <jpou...@gmail.com> >>>>> wrote: >>>>> >>>>>> Your proposed change looks correct to me. As far as whether other >>>>>> changes are required, I'm currently traveling but can look when I'm back >>>>>> on >>>>>> Thursday. Feel free to push the branches if tests are passing, we can fix >>>>>> things later? >>>>>> >>>>>> Le lun. 26 juin 2017 à 07:13, Anshum Gupta <ans...@anshumgupta.net> >>>>>> a écrit : >>>>>> >>>>>>> The only throw case that I think needs changing is in >>>>>>> SegmentInfos.java L315 >>>>>>> >>>>>>> Version luceneVersion = Version.fromBits(input.readVInt(), >>>>>>> input.readVInt(), input.readVInt()); >>>>>>> if (luceneVersion.onOrAfter(Version.LUCENE_7_0_0) == false) { >>>>>>> // TODO: should we check indexCreatedVersion instead? >>>>>>> throw new IndexFormatTooOldException(input, "this index is too old >>>>>>> (version: " + luceneVersion + ")"); >>>>>>> } >>>>>>> >>>>>>> >>>>>>> The Lucene version here should be LUCENE_7_0_0, instead of the >>>>>>> original LUCENE_6_0_0. >>>>>>> >>>>>>> Is there anything else that's needed (and is this the correct >>>>>>> change?). >>>>>>> >>>>>>> -Anshum >>>>>>> >>>>>>> >>>>>>> On Sun, Jun 25, 2017 at 9:45 PM Anshum Gupta <ans...@anshumgupta.net> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am working on creating the 7x, and 7.0 branches but I have the >>>>>>>> following failing tests: >>>>>>>> [junit4] Tests with failures [seed: 4FBDDCD3F96316D3]: >>>>>>>> [junit4] - >>>>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments >>>>>>>> [junit4] - >>>>>>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>>>>>> [junit4] - >>>>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment >>>>>>>> >>>>>>>> I assume this is a result of me not doing anything about the >>>>>>>> following TODO that got printed when I ran addVersion.py. >>>>>>>> *TODO* >>>>>>>> * - Update IndexFormatTooOldException throw cases* >>>>>>>> >>>>>>>> Can someone shed more light on what needs to be done here? >>>>>>>> >>>>>>>> -Anshum >>>>>>>> >>>>>>> >>>> >>> >>> >>> > > >
