Hello Mayya, I can share my view. Strictly speaking, we only need a new codec version if the on-disk format changes for some type (like kNN vectors in this case). Otherwise we can just evolve the logic in the same way we do with other code, making the improvements in-place. Alessandro's PR is a good example of this -- it's a refactor that happens to touch the vectors reader code. It doesn't require a totally new codec version.
With your PR for LUCENE-10592, I don't think we technically need a new codec for it, because the on-disk format remains the same. But since it's such a big change that deeply affects the vectors writing logic, it feels convenient that we're making it in a new format (maybe feels a bit clearer/ safer). Julie On Wed, Jul 20, 2022 at 4:21 PM Mayya Sharipova <mayya.sharip...@elastic.co.invalid> wrote: > Hi Julie, > thanks for raising this point, I was also thinking about it. > I think it would be valuable to clarify what constitutes the format > changes. Is on disk changes a necessary requirement for it? Or is it enough > that codec readers and writers change their behaviour? > > For example, for LUCENE-10592, there are extensive changes about how codec > readers and writers behave, while the format on disk stays the same. > Specifically for the release 9.3, Alessandro also introduced some changes > <https://github.com/apache/lucene/pull/926>to codec readers while the > format on disk stayed the same. > > > > > On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani <juliet...@gmail.com> > wrote: > >> Hello everyone, >> >> We had preemptively created a Lucene93Codec to support the vector search >> changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592). >> However neither of these changes made it for 9.3, so there have not been >> any real format changes-- Lucene93Codec is the same as Lucene92Codec. >> Should we remove Lucene93Codec on branch_9_3 (effectively reverting >> https://github.com/apache/lucene/pull/924) ? >> >> Julie >> >> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova >> <mayya.sharip...@elastic.co.invalid> wrote: >> >>> Thanks Adrien, it is indeed a big change, also would be nice to see >>> benchmarks after it is merged. >>> >>> So, Ignacio, please don't wait for LUCENE-10592, we will not be able to >>> make it for tomorrow. >>> >>> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand <jpou...@gmail.com> wrote: >>> >>>> While I can understand the excitement about LUCENE-10592, it's also a >>>> big change, maybe we should not even try to get it in before cutting the >>>> branch? >>>> >>>> On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova >>>> <mayya.sharip...@elastic.co.invalid> wrote: >>>> >>>>> Thanks for the reminder about the release, Ignacio! >>>>> About LUCENE-10592 >>>>> <https://issues.apache.org/jira/browse/LUCENE-10592> I will see what >>>>> progress we can make today, and will let you know before Wednesday at 9:00 >>>>> CEST. >>>>> >>>>> On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov <msoko...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks for checking, but please don't wait for LUCENE-10577. It's not >>>>>> clear when that might get resolved >>>>>> >>>>>> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera <iver...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Just a quick reminder I plan to cut the 9.3 branch this Wednesday at >>>>>>> 9:00 CEST. Let me know if there is any issue. >>>>>>> >>>>>>> @Mike: I see that LUCENE-10577 >>>>>>> <https://issues.apache.org/jira/browse/LUCENE-10577> is still under >>>>>>> API discussions, Do you think it will make it to the release? >>>>>>> LUCENE-10592 <https://issues.apache.org/jira/browse/LUCENE-10592> >>>>>>> seems to have made good progress, would you think it will be ready? >>>>>>> @Nick: I gave you feedback on the PR. I think it is way too risky to >>>>>>> add an encoding in a rush. I recommend adding the change in the sandbox >>>>>>> if >>>>>>> you want to iterate in there or wait for another release where there is >>>>>>> more time to think through the encoding. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Ignacio >>>>>>> >>>>>>> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize <nkn...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I'd like to get ShapeDocValuesField in for the 9.3 release ( >>>>>>>> https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice >>>>>>>> feature for enabling facets and aggregations over XYShape and >>>>>>>> LatLonShape >>>>>>>> field types and could make for a good 9.3 geo highlight. >>>>>>>> >>>>>>>> Nicholas Knize, Ph.D., GISP >>>>>>>> Principal Engineer - Search | Amazon >>>>>>>> Apache Lucene PMC Member and Committer >>>>>>>> nkn...@apache.org >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 12, 2022 at 2:50 PM Ignacio Vera <iver...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for the heads up, I am planning to cut the brunch middle >>>>>>>>> next week, Wednesday July 20th. >>>>>>>>> Let me know at the beginning of next week if there is any issue >>>>>>>>> from your side. >>>>>>>>> >>>>>>>>> cheers, >>>>>>>>> >>>>>>>>> Ignacio >>>>>>>>> >>>>>>>>> On Mon, Jul 11, 2022 at 4:21 PM Michael Sokolov < >>>>>>>>> msoko...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I would like to see if we can get >>>>>>>>>> https://issues.apache.org/jira/browse/LUCENE-10577 in. It is >>>>>>>>>> working >>>>>>>>>> and gives nice gains, but there is some controversy about the >>>>>>>>>> API. If >>>>>>>>>> we can't get it sorted out this week(?) it can certainly slip to >>>>>>>>>> the >>>>>>>>>> next revision. I know that >>>>>>>>>> https://issues.apache.org/jira/browse/LUCENE-10592 is also >>>>>>>>>> baking and >>>>>>>>>> has a PR that seems to be progressing rapidly. >>>>>>>>>> >>>>>>>>>> On Mon, Jul 11, 2022 at 10:03 AM Ignacio Vera <iver...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > Hello! >>>>>>>>>> > >>>>>>>>>> > Lucene 9.2.0 was released around 2 months ago and we are >>>>>>>>>> accumulating a good bunch of new features, optimizations and bug >>>>>>>>>> fixes. >>>>>>>>>> Would there be support for releasing Lucene 9.3 soon? >>>>>>>>>> > >>>>>>>>>> > I am happy being the release manager. I did not see any issues >>>>>>>>>> marked "blocker", but please let me know if there are any. >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > >>>>>>>>>> > Ignacio >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>>>>>>>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>>>>>>>> >>>>>>>>>> >>>> >>>> -- >>>> Adrien >>>> >>>
