The problem is that old 7.x indexes still use some codecs named by version 6. They were never updated!
So backwards codec must keep all stuff in metainf and as classes that the 7.0 original index format requires. Maybe create a dummy 7.0 index in branch-7x to have a list of codecs to test. Uwe Am 30. Juni 2017 00:43:06 MESZ schrieb Anshum Gupta <ansh...@apple.com>: >I’ve pushed more changes there, and we have a new set of errors. This >is one of them: > >[junit4] 2> NOTE: reproduce with: ant test >-Dtestcase=TestBackwardsCompatibility >-Dtests.method=testUnsupportedOldIndexes -Dtests.seed=8FDA7D3598A2FB46 >-Dtests.slow=true -Dtests.locale=ar-LB >-Dtests.timezone=America/Indiana/Marengo -Dtests.asserts=true >-Dtests.file.encoding=UTF-8 >[junit4] ERROR 3.07s | >TestBackwardsCompatibility.testUnsupportedOldIndexes <<< >[junit4] > Throwable #1: java.lang.IllegalArgumentException: Could >not load codec 'Lucene60'. Did you forget to add >lucene-backward-codecs.jar? >[junit4] > at >__randomizedtesting.SeedInfo.seed([8FDA7D3598A2FB46:74214F1628395C1A]:0) >[junit4] > at >org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:433) >[junit4] > at >org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:360) >[junit4] > at >org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:295) >[junit4] > at >org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:59) >[junit4] > at >org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:56) >[junit4] > at >org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:694) >[junit4] > at >org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:79) >[junit4] > at >org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) >[junit4] > at >org.apache.lucene.index.TestBackwardsCompatibility.testUnsupportedOldIndexes(TestBackwardsCompatibility.java:613) > [junit4] > at java.lang.Thread.run(Thread.java:748) >[junit4] > Caused by: java.lang.IllegalArgumentException: An SPI >class of type org.apache.lucene.codecs.Codec with name 'Lucene60' does >not exist. You need to add the corresponding JAR file supporting this >SPI to your classpath. The current classpath supports the following >names: [Asserting, CheapBastard, FastCompressingStoredFields, >FastDecompressionCompressingStoredFields, >HighCompressionCompressingStoredFields, DummyCompressingStoredFields, >SimpleText, Lucene70] > > >Do you intend to Ignore this for now? Also, in the last commit, I’ve >Ignored a bunch of tests that use the old indexes. > > >-Anshum > > > >> On Jun 29, 2017, at 3:10 PM, Anshum Gupta <ansh...@apple.com> wrote: >> >> I did remove the declaration in META-INT/services, atleast everything >that had a version in it’s name i.e. 5x, or 6x. >> >> I’ve also renamed the indexes for 6x, but here are a few that I >wasn’t sure about what to do with these: >> sorted.6.3.0.zip >> sorted.6.2.1.zip >> sorted.6.2.0.zip >> moreterms.6.0.0.zip >> maxposindex.zip >> manypointsindex.zip >> empty.6.0.0.zip >> dvupdates.6.0.0.zip >> >> Considering you suggested disabling the tests, should we be removing >these indexes and regenerating these post release when re re-enable >tests or should we keep them here and just disable the tests? >> >> I’ve reverted the changes in SegmentInfos.java, and also changed >testIllegalCreatedVersion as per your suggestion. >> >> I’m running the tests now, and will commit to my fork right after. >> >> Thanks for helping out with this. >> >> -Anshum >> >> >> >>> On Jun 29, 2017, at 2:33 PM, Adrien Grand <jpou...@gmail.com ><mailto:jpou...@gmail.com>> wrote: >>> >>> Removing most backward codecs sounds good to me since the only codec >that 8.0 needs to be able to read so far is the 7.0 codec which is in >core. It looks like you removed the code, but you also need to remove >their declaration in META-INF/services or the SPI loaded will try to >load them and fail since it cannot find the class. >>> >>> Backcompat indexes will be added as we perform 7.x releases. However >you'd need to rename the 6.x indices from index.6.x.x to >unsupported.6.x.x. >>> >>> We have some specific tests like "moreterms" and "dvupdates". I >think we need to disable them for now and make sure to reenable them >once 7.0 is released. >>> >>> I think the changes you did in SegmentInfos.java ><https://github.com/anshumg/lucene-solr/commit/c7f5b9f9fa94fe1a6e6abc7ffc74fc3df16293a4#diff-e3ccf9ee90355b10f2dd22ce2da6c73c> >are not necessary. It looks like the version numbers are related to the >current major, but it is actually due to the fact that 7.0 is the first >version to record the version that was used at creation time. I think >you can revert changes in this file entirely. In the >testIllegalCreatedVersion test, I'd just replace 8 with 9 or >Version.CURRENT.major + 1. >>> >>> We'd need to remove the compatibility layer in similarities but it >can be done as a follow-up. >>> >>> Thanks for taking care of this! >>> >>> Le jeu. 29 juin 2017 à 23:12, Anshum Gupta <ansh...@apple.com ><mailto:ansh...@apple.com>> a écrit : >>> Adrien, I’ve pushed some more changes and seems like I’d have to >regenerate some test indexes but I’m not sure how to do that. Do you >mind taking a look at this in it’s current form, and also my commits? >It is all @ my fork here: https://github.com/anshumg/lucene-solr ><https://github.com/anshumg/lucene-solr> >>> >>> P.S: I thought it’d make more sense to do this on a feature-branch >but the upgrade script wasn’t happy about that. >>> >>> -Anshum >>> >>> >>> >>>> On Jun 29, 2017, at 9:20 AM, Anshum Gupta <ansh...@apple.com ><mailto:ansh...@apple.com>> wrote: >>>> >>>> Going with your suggestions, seems like we’d be wiping out all of >the backward-codecs folder/package, is that correct ? Also, do we need >to put in anything to ensure back-combat between 6x, and 7x? >>>> >>>> -Anshum >>>> >>>> >>>> >>>>> On Jun 29, 2017, at 7:21 AM, Anshum Gupta <ansh...@apple.com ><mailto:ansh...@apple.com>> wrote: >>>>> >>>>> Thanks Adrien, I’d want to try and do this myself as long as you >can validate the correctness :). >>>>> >>>>> I’ll be working on this in a few hours and should have an update >later today and hopefully we’d wrap it up soon. >>>>> >>>>> -Anshum >>>>> >>>>> >>>>> >>>>>> On Jun 28, 2017, at 10:39 AM, Adrien Grand <jpou...@gmail.com ><mailto:jpou...@gmail.com>> wrote: >>>>>> >>>>>> If you don't want to do it, I can do it tomorrow but if you'd >like to give it a try I'd be happy to help if you need any guidance. >>>>>> >>>>>> Le mer. 28 juin 2017 à 19:38, Adrien Grand <jpou...@gmail.com ><mailto:jpou...@gmail.com>> a écrit : >>>>>> Hi Anshum, >>>>>> >>>>>> This looks like a good start to me. You would also need to remove >the 6.x version constants so that TestBackwardCompatibility does not >think they are worth testing, as well as all codecs, postings formats >and doc values formats that are defined in the lucene/backward-codecs >module since they are only about 6.x codecs. >>>>>> >>>>>> Le mer. 28 juin 2017 à 09:57, Anshum Gupta <ansh...@apple.com ><mailto:ansh...@apple.com>> a écrit : >>>>>> Thanks for confirming that Alan, I had similar thoughts but >wasn’t sure. >>>>>> >>>>>> I don’t want to change anything that I’m not confident about so >I’m just going to create remove those and commit it to my fork. If >someone who’s confident agrees with what I’m doing, I’ll go ahead and >make those changes to the upstream :). >>>>>> >>>>>> -Anshum >>>>>> >>>>>> >>>>>> >>>>>>> On Jun 28, 2017, at 12:54 AM, Alan Woodward <a...@flax.co.uk ><mailto:a...@flax.co.uk>> wrote: >>>>>>> >>>>>>> We don’t need to support lucene5x codecs in 7, so you should be >able to just remove those tests (and the the relevant packages from >backwards-codecs too), I think? >>>>>>> >>>>>>> >>>>>>>> On 28 Jun 2017, at 08:38, Anshum Gupta <ansh...@apple.com ><mailto:ansh...@apple.com>> wrote: >>>>>>>> >>>>>>>> I tried to move forward to see this work before automatically >computing the versions but I have about 30 odd failing test. I’ve made >those changes and pushed to my local GitHub account in case you have >the time to look: https://github.com/anshumg/lucene-solr ><https://github.com/anshumg/lucene-solr> >>>>>>>> >>>>>>>> Here’s the build summary if that helps: >>>>>>>> >>>>>>>> [junit4] Tests with failures [seed: 31C3B60E557C7E14] (first >10 out of 31): >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testOutliers2 >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testShortRange >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewValues >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFullLongRange >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testRamBytesUsed >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewLargeValues >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testByteRange >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testLongRange >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene50.TestLucene50SegmentInfoFormat.testRandomExceptions >>>>>>>> [junit4] - >org.apache.lucene.codecs.lucene62.TestLucene62SegmentInfoFormat.testRandomExceptions >>>>>>>> [junit4] >>>>>>>> [junit4] >>>>>>>> [junit4] JVM J0: 0.56 .. 9.47 = 8.91s >>>>>>>> [junit4] JVM J1: 0.56 .. 4.13 = 3.57s >>>>>>>> [junit4] JVM J2: 0.56 .. 47.28 = 46.73s >>>>>>>> [junit4] JVM J3: 0.56 .. 3.89 = 3.33s >>>>>>>> [junit4] Execution time total: 47 seconds >>>>>>>> [junit4] Tests summary: 8 suites, 215 tests, 30 errors, 1 >failure, 24 ignored (24 assumptions) >>>>>>>> >>>>>>>> >>>>>>>> -Anshum >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Jun 27, 2017, at 4:15 AM, Adrien Grand <jpou...@gmail.com ><mailto:jpou...@gmail.com>> wrote: >>>>>>>>> >>>>>>>>> The test***BackwardCompatibility cases can be removed since >they make sure that Lucene 7 can read Lucene 6 norms, while Lucene 8 >doesn't have to be able to read Lucene 6 norms. >>>>>>>>> >>>>>>>>> TestSegmentInfos needs to be adapted to the new versions, we >need to replace 5 with 6 and 8 with 9. Maybe we should compute those >numbers automatically based on Version.LATEST.major so that it does not >require manual changes when moving to a new major version. That would >give 5 -> Version.LATEST.major-2 and 8 -> Version.LATEST.major+1. >>>>>>>>> >>>>>>>>> I can do those changes on Thursday if you don't feel >comfortable doing them. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Le mar. 27 juin 2017 à 08:12, Anshum Gupta <ansh...@apple.com ><mailto:ansh...@apple.com>> a écrit : >>>>>>>>> Without making any changes at all and just bumping up the >version, I hit these errors when running the tests: >>>>>>>>> >>>>>>>>> [junit4] 2> NOTE: reproduce with: ant test >-Dtestcase=TestSegmentInfos -Dtests.method=testIllegalCreatedVersion >-Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >-Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >-Dtests.file.encoding=US-ASCII >>>>>>>>> [junit4] FAILURE 0.01s J0 | >TestSegmentInfos.testIllegalCreatedVersion <<< >>>>>>>>> [junit4] > Throwable #1: >junit.framework.AssertionFailedError: Expected exception >IllegalArgumentException but no exception was thrown >>>>>>>>> [junit4] > at >__randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:CE340683BE44C211]:0) >>>>>>>>> [junit4] > at >org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2672) >>>>>>>>> [junit4] > at >org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion(TestSegmentInfos.java:35) >>>>>>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>>>>>> [junit4] 2> NOTE: reproduce with: ant test >-Dtestcase=TestSegmentInfos -Dtests.method=testVersionsOneSegment >-Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >-Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >-Dtests.file.encoding=US-ASCII >>>>>>>>> [junit4] ERROR 0.00s J0 | >TestSegmentInfos.testVersionsOneSegment <<< >>>>>>>>> [junit4] > Throwable #1: >org.apache.lucene.index.CorruptIndexException: segments file recorded >indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older >version=7.0.0 >(resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>>>>>>> [junit4] > at >__randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:A7477EE8875F2E36]:0) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>>>>>>> [junit4] > at >org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment(TestSegmentInfos.java:67) >>>>>>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>>>>>> [junit4] 2> NOTE: reproduce with: ant test >-Dtestcase=TestSegmentInfos -Dtests.method=testVersionsTwoSegments >-Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >-Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >-Dtests.file.encoding=US-ASCII >>>>>>>>> [junit4] ERROR 0.00s J0 | >TestSegmentInfos.testVersionsTwoSegments <<< >>>>>>>>> [junit4] > Throwable #1: >org.apache.lucene.index.CorruptIndexException: segments file recorded >indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older >version=7.0.0 >(resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>>>>>>> [junit4] > at >__randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:4EE9CC4194FBB648]:0) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>>>>>>> [junit4] > at >org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>>>>>>> [junit4] > at >org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments(TestSegmentInfos.java:96) >>>>>>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>>>>>> >>>>>>>>> On applying the patch here (https://pastebin.com/tM4Fpy1Q ><https://pastebin.com/tM4Fpy1Q>), I end up with the following errors: >>>>>>>>> >>>>>>>>> [junit4] Tests with failures [seed: 5B388AB1E2BEFF87]: >>>>>>>>> [junit4] - >org.apache.lucene.search.similarities.TestSimilarityBase.testLengthEncodingBackwardCompatibility >>>>>>>>> [junit4] - >org.apache.lucene.search.similarities.TestClassicSimilarity.testNormEncodingBackwardCompatibility >>>>>>>>> [junit4] - >org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>>>>>>> [junit4] - >org.apache.lucene.search.similarities.TestBM25Similarity.testLengthEncodingBackwardCompatibility >>>>>>>>> >>>>>>>>> Any ideas on what I’m missing here? >>>>>>>>> >>>>>>>>> >>>>>>>>> -Anshum >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Jun 26, 2017, at 2:55 PM, Ryan Ernst <r...@iernst.net ><mailto:r...@iernst.net>> wrote: >>>>>>>>>> >>>>>>>>>> After changing that constant check, do you get the same test >failures? What are the actual failure messages? >>>>>>>>>> >>>>>>>>>> On Mon, Jun 26, 2017 at 10:38 AM Anshum Gupta ><ans...@anshumgupta.net <mailto:ans...@anshumgupta.net>> wrote: >>>>>>>>>> Thanks Adrien, sadly, the tests aren't passing. That's the >reason why I didn't push the changes. I'll see if someone else can help >while you're away. >>>>>>>>>> >>>>>>>>>> -Anshum >>>>>>>>>> >>>>>>>>>> On Mon, Jun 26, 2017 at 9:55 AM Adrien Grand ><jpou...@gmail.com <mailto:jpou...@gmail.com>> wrote: >>>>>>>>>> Your proposed change looks correct to me. As far as whether >other changes are required, I'm currently traveling but can look when >I'm back on Thursday. Feel free to push the branches if tests are >passing, we can fix things later? >>>>>>>>>> >>>>>>>>>> Le lun. 26 juin 2017 à 07:13, Anshum Gupta ><ans...@anshumgupta.net <mailto:ans...@anshumgupta.net>> a écrit : >>>>>>>>>> The only throw case that I think needs changing is in >SegmentInfos.java L315 >>>>>>>>>> Version luceneVersion = Version.fromBits(input.readVInt(), >input.readVInt(), input.readVInt()); >>>>>>>>>> if (luceneVersion.onOrAfter(Version.LUCENE_7_0_0) == false) { >>>>>>>>>> // TODO: should we check indexCreatedVersion instead? >>>>>>>>>> throw new IndexFormatTooOldException(input, "this index is >too old (version: " + luceneVersion + ")"); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> The Lucene version here should be LUCENE_7_0_0, instead of >the original LUCENE_6_0_0. >>>>>>>>>> >>>>>>>>>> Is there anything else that's needed (and is this the correct >change?). >>>>>>>>>> >>>>>>>>>> -Anshum >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Jun 25, 2017 at 9:45 PM Anshum Gupta ><ans...@anshumgupta.net <mailto:ans...@anshumgupta.net>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I am working on creating the 7x, and 7.0 branches but I have >the following failing tests: >>>>>>>>>> [junit4] Tests with failures [seed: 4FBDDCD3F96316D3]: >>>>>>>>>> [junit4] - >org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments >>>>>>>>>> [junit4] - >org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>>>>>>>> [junit4] - >org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment >>>>>>>>>> >>>>>>>>>> I assume this is a result of me not doing anything about the >following TODO that got printed when I ran addVersion.py. >>>>>>>>>> TODO >>>>>>>>>> - Update IndexFormatTooOldException throw cases >>>>>>>>>> >>>>>>>>>> Can someone shed more light on what needs to be done here? >>>>>>>>>> >>>>>>>>>> -Anshum >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
