thanks for all the feedback, I opened https://issues.apache.org/jira/browse/LUCENE-9669 to address this further.
On Wed, Jan 13, 2021 at 2:58 PM Adrien Grand <jpou...@gmail.com> wrote: > > +1 this strikes to me as a good balance between increasing backward > compatibility guarantees and still keeping room for innovation. > > David, actually I would like to advocate in favor of still disallowing > opening N-2 indices by default, as they might not match Lucene's current > expectations (e.g. using a different encoding for norms due to LUCENE-7730), > and using Lucene's current analyzers/similarities/queries might trigger > surprising behavior. My preference would be to expose the ability to open N-2 > indices behind an expert API/flag that documents limitations with N-2 indices. > > Mike, I wondered about this question too. As you pointed out, I think that we > will generally be ok given that the N-2 compatibility layer will very likely > be the same as the N-1 compatibility layer that we need to develop anyway. I > tried to think of examples when that wouldn't work but couldn't find any > (which doesn't mean that there is none, but hopefully it would be rare). > > > > On Mon, Jan 11, 2021 at 4:57 PM Michael McCandless > <luc...@mikemccandless.com> wrote: >> >> +1, I like the idea in general. >> >> We will have to work out the details in practice as we come across "index >> breaking" changes, and where/how to draw the line of "best effort". But I >> think this is an improvement for our users over the hard check we now have >> for "only N-1", and likely not so much development effort? >> >> I think where it might get interesting is if we want to make a Codec API >> change, maybe to optimize a interesting use-cases, and then we must do some >> development to fix N-2 BWC codec (as well as N-1 BWC codec that we already >> must fix for such an example, today). >> >> Some users seem to keep their indices alive for a very long time! >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> >> On Sat, Jan 9, 2021 at 6:13 AM Simon Willnauer <simon.willna...@gmail.com> >> wrote: >>> >>> I can provide some examples of BWC issues and what we would do if it >>> happened in the future: >>> >>> - negative offsets: in this case it would be best effort to add a >>> wrapper around the older formats to check if the offsets go backwards >>> on the read side and throw an exception to prevent consumers making >>> the assumption that offsets go forward only from failing or going OOM >>> etc. >>> - norms encoding: in this case it would be best effort in the older >>> norms formats to convert to the newer encodings. >>> - the removal of numeric fields queries would not fall under the >>> promises we make with compatibility of N-2 and it would be the >>> responsibility of the user to keep the code around that understands >>> the value of a field. >>> >>> I hope this clarifies some of the aspects? >>> >>> we would only do all this for the reading end, for writing we would >>> reject indices that are older than N-1 >>> >>> simon >>> >>> >>> On Thu, Jan 7, 2021 at 8:04 PM jim ferenczi <jim.feren...@gmail.com> wrote: >>> > >>> > The proposal is only about keeping the ability to read file-format up to >>> > N-2. Everything that is done on top of the file format is not guaranteed >>> > and should be supported on a best-effort basis. >>> > That's an important aspect if we don't want to block innovation. So in >>> > practice that means that queries that require some specific file format >>> > or analyzers that change behaviors in major versions would not be part of >>> > the extended guarantee. >>> > >>> > >>> > Le mer. 6 janv. 2021 à 21:53, Yonik Seeley <ysee...@gmail.com> a écrit : >>> >> >>> >> On Wed, Jan 6, 2021 at 4:40 AM Simon Willnauer >>> >> <simon.willna...@gmail.com> wrote: >>> >>> >>> >>> You can open a reader on an index created by >>> >>> version N-2, but you cannot open an IndexWriter on it >>> >> >>> >> >>> >> +1 >>> >> There should definitely be more consideration given to back compat in >>> >> general... it's caused a ton of pain to users over time. >>> >> >>> >> -Yonik >>> >> >>> >> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> > > > -- > Adrien --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
