Re: The need for a Lucene 9.9.1 release
Thanks, I added a couple of 9.9.1 changelog entries, and will start the RC1 process. -Chris. > On 12 Dec 2023, at 18:42, Michael McCandless > wrote: > > OK this is merged now. Are there any other 9.9.1 blockers? I am trying to > pass all Monster tests but that can probably just run concurrently with > voting (optimistic concurrency!)? > > Mike McCandless > > http://blog.mikemccandless.com > > > On Tue, Dec 12, 2023 at 9:18 AM Chris Hegarty > wrote: > Hi Mike, > > > On 12 Dec 2023, at 12:56, Michael McCandless > > wrote: > > > > Hi Chris, > > > > I think we should also regenerate the FSTs for 9.9.1? > > Seems reasonable. > > > https://github.com/apache/lucene/pull/12924 > > I added my comments and review on the PR. > > -Chris. > > > Thanks, > > > > Mike > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: The need for a Lucene 9.9.1 release
OK this is merged now. Are there any other 9.9.1 blockers? I am trying to pass all Monster tests but that can probably just run concurrently with voting (optimistic concurrency!)? Mike McCandless http://blog.mikemccandless.com On Tue, Dec 12, 2023 at 9:18 AM Chris Hegarty wrote: > Hi Mike, > > > On 12 Dec 2023, at 12:56, Michael McCandless > wrote: > > > > Hi Chris, > > > > I think we should also regenerate the FSTs for 9.9.1? > > Seems reasonable. > > > https://github.com/apache/lucene/pull/12924 > > I added my comments and review on the PR. > > -Chris. > > > Thanks, > > > > Mike > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: The need for a Lucene 9.9.1 release
Hi Mike, > On 12 Dec 2023, at 12:56, Michael McCandless > wrote: > > Hi Chris, > > I think we should also regenerate the FSTs for 9.9.1? Seems reasonable. > https://github.com/apache/lucene/pull/12924 I added my comments and review on the PR. -Chris. > Thanks, > > Mike - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: The need for a Lucene 9.9.1 release
Hi Chris, I think we should also regenerate the FSTs for 9.9.1? https://github.com/apache/lucene/pull/12924 Thanks, Mike On Tue, Dec 12, 2023 at 7:54 AM Guo Feng wrote: > Heads up: > > The bug fix PR (https://github.com/apache/lucene/pull/12900) has been > merged to main, and backported to lucene_9x & lucene_9_9. > > On 2023/12/11 20:27:48 Chris Hegarty wrote: > > Just a quick update on this... > > > > > On 9 Dec 2023, at 09:09, Chris Hegarty > wrote: > > > > > > Hi, > > > > > > We’ve encounter two very serious issues with the recent Lucene 9.9.0 > release, both of which (even if taken by themselves) would warrant a 9.9.1. > The issues are: > > > > > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on > term dictionaries in Lucene 9.9 > > > > Great work has been done re-adding tests, creating a new test to > reproduce, and also working on an underlying fix. It feels like we’re > getting close! :-) > > > > > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash > when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 > > > > Merged to branch_9_9. > > > > Once no.1 is merged, I’ll build a 9.9.1 RC1 and start a vote. > > > > -Chris > > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: The need for a Lucene 9.9.1 release
Heads up: The bug fix PR (https://github.com/apache/lucene/pull/12900) has been merged to main, and backported to lucene_9x & lucene_9_9. On 2023/12/11 20:27:48 Chris Hegarty wrote: > Just a quick update on this... > > > On 9 Dec 2023, at 09:09, Chris Hegarty > > wrote: > > > > Hi, > > > > We’ve encounter two very serious issues with the recent Lucene 9.9.0 > > release, both of which (even if taken by themselves) would warrant a 9.9.1. > > The issues are: > > > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term > > dictionaries in Lucene 9.9 > > Great work has been done re-adding tests, creating a new test to reproduce, > and also working on an underlying fix. It feels like we’re getting close! :-) > > > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when > > compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 > > Merged to branch_9_9. > > Once no.1 is merged, I’ll build a 9.9.1 RC1 and start a vote. > > -Chris > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: The need for a Lucene 9.9.1 release
Just a quick update on this... > On 9 Dec 2023, at 09:09, Chris Hegarty wrote: > > Hi, > > We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, > both of which (even if taken by themselves) would warrant a 9.9.1. The issues > are: > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term > dictionaries in Lucene 9.9 Great work has been done re-adding tests, creating a new test to reproduce, and also working on an underlying fix. It feels like we’re getting close! :-) > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when > compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 Merged to branch_9_9. Once no.1 is merged, I’ll build a 9.9.1 RC1 and start a vote. -Chris - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: The need for a Lucene 9.9.1 release
> On 9 Dec 2023, at 09:09, Chris Hegarty wrote: > > Hi, > > We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, > both of which (even if taken by themselves) would warrant a 9.9.1. The issues > are: > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term > dictionaries in Lucene 9.9 > > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when > compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 I opened a small PR which reflows the code in computeCommonPrefixLengthAndBuildHistogram which has the affect of working around the JIT crash. https://github.com/apache/lucene/pull/12903 -Chris. > There is still a little investigation and work left to bring these issues to > a point where we’re comfortable with proposing a solution. I would be hopeful > that we’ll get there by early next week. If so, then a Lucene 9.9.1 release > can be proposed. > > Thanks, > -Chris. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: The need for a Lucene 9.9.1 release
FYI - I added the next bugfix version 9.9.1 to `branch_9_9`, in preparation for the upcoming bug fix release. https://github.com/apache/lucene/commit/1617c0b3a5624adba6e7b380dfeb7fb89b8a2feb -Chris. > On 9 Dec 2023, at 09:09, Chris Hegarty wrote: > > Hi, > > We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, > both of which (even if taken by themselves) would warrant a 9.9.1. The issues > are: > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term > dictionaries in Lucene 9.9 > > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when > compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 > > There is still a little investigation and work left to bring these issues to > a point where we’re comfortable with proposing a solution. I would be hopeful > that we’ll get there by early next week. If so, then a Lucene 9.9.1 release > can be proposed. > > Thanks, > -Chris. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: The need for a Lucene 9.9.1 release
I believe your assessment that it is "only" a read problem is correct. I can see how using the "corruption" wording may have caused confusion. It is a severe bug though that affects multi term queries and I thought it's a good idea to patch that, given that folks have reproduced it and found the root cause. I agree on adding tests that cover it and not rushing anything out, yet people upgrading to 9.9.0 are affected by it and that seems bad. Thanks for the feedback. On Sat, Dec 9, 2023 at 12:31 PM Robert Muir wrote: > I don't understand use of the word corruption, isn't it just a bug in > intersect() that only affects wildcards etc? e.g. its not gonna merge > into new segments or impact written data in any way. > > And i don't think we should rushout some bugfix release without any > test for this? > > On Sat, Dec 9, 2023 at 5:30 AM Luca Cavanna wrote: > > > > Based on the discussions in > https://github.com/apache/lucene/issues/12895 , it seems like reverting > the change that caused the corruption on read is the quickest fix, so that > we can speed up releasing 9.9.1. I opened a PR for that: > https://github.com/apache/lucene/pull/12899. Is there additional testing > that needs to be done to ensure that this is enough to address the > corruption? > > > > Regarding a fix for the JVM SIGSEGV crash, how far are we from a fix > that protects Lucene from it? Should we wait for that to be included in > 9.9.1? Asking because the corruption above looks like it needs to be > addressed rather quickly. It would be great to include both, but I don't > know how long that delays 9.9.1. > > > > Cheers > > Luca > > > > > > > > On Sat, Dec 9, 2023 at 11:13 AM Chris Hegarty > wrote: > >> > >> Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent > experience on 9.9.0) > >> > >> -Chris. > >> > >> > On 9 Dec 2023, at 09:09, Chris Hegarty < > christopher.hega...@elastic.co> wrote: > >> > > >> > Hi, > >> > > >> > We’ve encounter two very serious issues with the recent Lucene 9.9.0 > release, both of which (even if taken by themselves) would warrant a 9.9.1. > The issues are: > >> > > >> > 1. https://github.com/apache/lucene/issues/12895 - Corruption read > on term dictionaries in Lucene 9.9 > >> > > >> > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash > when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 > >> > > >> > There is still a little investigation and work left to bring these > issues to a point where we’re comfortable with proposing a solution. I > would be hopeful that we’ll get there by early next week. If so, then a > Lucene 9.9.1 release can be proposed. > >> > > >> > Thanks, > >> > -Chris. > >> > >> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: The need for a Lucene 9.9.1 release
I don't understand use of the word corruption, isn't it just a bug in intersect() that only affects wildcards etc? e.g. its not gonna merge into new segments or impact written data in any way. And i don't think we should rushout some bugfix release without any test for this? On Sat, Dec 9, 2023 at 5:30 AM Luca Cavanna wrote: > > Based on the discussions in https://github.com/apache/lucene/issues/12895 , > it seems like reverting the change that caused the corruption on read is the > quickest fix, so that we can speed up releasing 9.9.1. I opened a PR for > that: https://github.com/apache/lucene/pull/12899. Is there additional > testing that needs to be done to ensure that this is enough to address the > corruption? > > Regarding a fix for the JVM SIGSEGV crash, how far are we from a fix that > protects Lucene from it? Should we wait for that to be included in 9.9.1? > Asking because the corruption above looks like it needs to be addressed > rather quickly. It would be great to include both, but I don't know how long > that delays 9.9.1. > > Cheers > Luca > > > > On Sat, Dec 9, 2023 at 11:13 AM Chris Hegarty > wrote: >> >> Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent >> experience on 9.9.0) >> >> -Chris. >> >> > On 9 Dec 2023, at 09:09, Chris Hegarty >> > wrote: >> > >> > Hi, >> > >> > We’ve encounter two very serious issues with the recent Lucene 9.9.0 >> > release, both of which (even if taken by themselves) would warrant a >> > 9.9.1. The issues are: >> > >> > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term >> > dictionaries in Lucene 9.9 >> > >> > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when >> > compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 >> > >> > There is still a little investigation and work left to bring these issues >> > to a point where we’re comfortable with proposing a solution. I would be >> > hopeful that we’ll get there by early next week. If so, then a Lucene >> > 9.9.1 release can be proposed. >> > >> > Thanks, >> > -Chris. >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: The need for a Lucene 9.9.1 release
Based on the discussions in https://github.com/apache/lucene/issues/12895 , it seems like reverting the change that caused the corruption on read is the quickest fix, so that we can speed up releasing 9.9.1. I opened a PR for that: https://github.com/apache/lucene/pull/12899. Is there additional testing that needs to be done to ensure that this is enough to address the corruption? Regarding a fix for the JVM SIGSEGV crash, how far are we from a fix that protects Lucene from it? Should we wait for that to be included in 9.9.1? Asking because the corruption above looks like it needs to be addressed rather quickly. It would be great to include both, but I don't know how long that delays 9.9.1. Cheers Luca On Sat, Dec 9, 2023 at 11:13 AM Chris Hegarty wrote: > Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent > experience on 9.9.0) > > -Chris. > > > On 9 Dec 2023, at 09:09, Chris Hegarty > wrote: > > > > Hi, > > > > We’ve encounter two very serious issues with the recent Lucene 9.9.0 > release, both of which (even if taken by themselves) would warrant a 9.9.1. > The issues are: > > > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on > term dictionaries in Lucene 9.9 > > > > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash > when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 > > > > There is still a little investigation and work left to bring these > issues to a point where we’re comfortable with proposing a solution. I > would be hopeful that we’ll get there by early next week. If so, then a > Lucene 9.9.1 release can be proposed. > > > > Thanks, > > -Chris. > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: The need for a Lucene 9.9.1 release
Oh, and I’m happy to be Release Manager for 9.9.1 (given my recent experience on 9.9.0) -Chris. > On 9 Dec 2023, at 09:09, Chris Hegarty wrote: > > Hi, > > We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, > both of which (even if taken by themselves) would warrant a 9.9.1. The issues > are: > > 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term > dictionaries in Lucene 9.9 > > 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when > compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 > > There is still a little investigation and work left to bring these issues to > a point where we’re comfortable with proposing a solution. I would be hopeful > that we’ll get there by early next week. If so, then a Lucene 9.9.1 release > can be proposed. > > Thanks, > -Chris. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
The need for a Lucene 9.9.1 release
Hi, We’ve encounter two very serious issues with the recent Lucene 9.9.0 release, both of which (even if taken by themselves) would warrant a 9.9.1. The issues are: 1. https://github.com/apache/lucene/issues/12895 - Corruption read on term dictionaries in Lucene 9.9 2. https://github.com/apache/lucene/issues/12898 - JVM SIGSEGV crash when compiling computeCommonPrefixLengthAndBuildHistogram Lucene 9.9.0 There is still a little investigation and work left to bring these issues to a point where we’re comfortable with proposing a solution. I would be hopeful that we’ll get there by early next week. If so, then a Lucene 9.9.1 release can be proposed. Thanks, -Chris. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
