Re: New branch and feature freeze for Lucene 9.7.0

2023-06-16 Thread Chris Hegarty 
Hi Uwe,

> On 16 Jun 2023, at 13:52, Uwe Schindler  wrote:
> 
> Hi,
> 
> I also downloaded latest JDK 21 EA build (21-b27) and regenerated the apijar 
> files: No changes. So all fine! I did this because there were some late 
> changes to javadocs and API definition, but it all looks fine. Also the bug 
> with SecurityManager that hit Elasticsearch was also fixed (but we have a 
> workaround).
> 
> I will now also update Policeman Jenkins to latest EA build.
> 
Thanks for verifying the apijar, and adding the new EA build to the Jenkins.

It would be great if we could automate the apijar check, by periodically 
running a job that sucks down the latest EA build and runs the generation, 
comparing against the checked in version. But the way we do this already kinda 
minimises the risk - as you have said before (and I agree), it is very unlikely 
that the API will change during rampdown.

-Chris.
> Uwe
> 
> Am 16.06.2023 um 13:50 schrieb Adrien Grand:
>> NOTICE:
>> 
>> Branch branch_9_7 has been cut and versions updated to 9.8 on stable branch.
>> 
>> Please observe the normal rules:
>> 
>> * No new features may be committed to the branch.
>> * Documentation patches, build patches and serious bug fixes may be
>>   committed to the branch. However, you should submit all patches you
>>   want to commit as pull requests first to give others the chance to review
>>   and possibly vote against them. Keep in mind that it is our
>>   main intention to keep the branch as stable as possible.
>> * All patches that are intended for the branch should first be committed
>>   to the unstable branch, merged into the stable branch, and then into
>>   the current release branch.
>> * Normal unstable and stable branch development may continue as usual.
>>   However, if you plan to commit a big change to the unstable branch
>>   while the branch feature freeze is in effect, think twice: can't the
>>   addition wait a couple more days? Merges of bug fixes into the branch
>>   may become more difficult.
>> * Only Github issues with Milestone 9.7
>>   and priority "Blocker" will delay a release candidate build.
>> 
>> -- 
>> Adrien
> -- 
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de 
> eMail: u...@thetaphi.de

Richer Aggregations in Lucene

2023-06-16 Thread Shradha Shankar 
Hi Lucene devs,

I work on product search at Amazon, where we use Lucene faceting
to compute aggregations. There's a few functionalities I'm missing with
faceting. For example, faceting will always aggregate all the way up to the
dimension and it can't compute multiple aggregations in one pass of the
match-set.

Lucene-based search engines (like Elastic or OpenSearch) have feature-rich
aggregation engines which allow different collection modes and give the user
more control over the granularity of the scopes for which aggregations are
computed.

Are there historical reasons not to have this type of aggregation engine
directly in Lucene? If it seems like a worthwhile idea to pursue, I've
experimented a bit with how we could fulfill these needs in Lucene and I can
open an issue/PR.

Thanks,
Shradha

Re: New branch and feature freeze for Lucene 9.7.0

2023-06-16 Thread Uwe Schindler 

Hi,

I also downloaded latest JDK 21 EA build (21-b27) and regenerated the 
apijar files: No changes. So all fine! I did this because there were 
some late changes to javadocs and API definition, but it all looks fine. 
Also the bug with SecurityManager that hit Elasticsearch was also fixed 
(but we have a workaround).


I will now also update Policeman Jenkins to latest EA build.

Uwe

Am 16.06.2023 um 13:50 schrieb Adrien Grand:

NOTICE:

Branch branch_9_7 has been cut and versions updated to 9.8 on stable 
branch.


Please observe the normal rules:

* No new features may be committed to the branch.
* Documentation patches, build patches and serious bug fixes may be
  committed to the branch. However, you should submit all patches you
  want to commit as pull requests first to give others the chance to 
review

  and possibly vote against them. Keep in mind that it is our
  main intention to keep the branch as stable as possible.
* All patches that are intended for the branch should first be committed
  to the unstable branch, merged into the stable branch, and then into
  the current release branch.
* Normal unstable and stable branch development may continue as usual.
  However, if you plan to commit a big change to the unstable branch
  while the branch feature freeze is in effect, think twice: can't the
  addition wait a couple more days? Merges of bug fixes into the branch
  may become more difficult.
* Only Github issues with Milestone 9.7
  and priority "Blocker" will delay a release candidate build.

--
Adrien


--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:u...@thetaphi.de

New branch and feature freeze for Lucene 9.7.0

2023-06-16 Thread Adrien Grand 
NOTICE:

Branch branch_9_7 has been cut and versions updated to 9.8 on stable branch.

Please observe the normal rules:

* No new features may be committed to the branch.
* Documentation patches, build patches and serious bug fixes may be
  committed to the branch. However, you should submit all patches you
  want to commit as pull requests first to give others the chance to review
  and possibly vote against them. Keep in mind that it is our
  main intention to keep the branch as stable as possible.
* All patches that are intended for the branch should first be committed
  to the unstable branch, merged into the stable branch, and then into
  the current release branch.
* Normal unstable and stable branch development may continue as usual.
  However, if you plan to commit a big change to the unstable branch
  while the branch feature freeze is in effect, think twice: can't the
  addition wait a couple more days? Merges of bug fixes into the branch
  may become more difficult.
* Only Github issues with Milestone 9.7
  and priority "Blocker" will delay a release candidate build.

-- 
Adrien

TAC Applications for Community Over Code North America and Asia now open

2023-06-16 Thread Gavin McDonald 
Hi All,

(This email goes out to all our user and dev project mailing lists, so you
may receive this
email more than once.)

The Travel Assistance Committee has opened up applications to help get
people to the following events:


*Community Over Code Asia 2023 - *
*August 18th to August 20th in Beijing , China*

Applications for this event closes on the 6th July so time is short, please
apply as soon as possible. TAC is prioritising applications from the Asia
and Oceania regions.

More details on this event can be found at:
https://apachecon.com/acasia2023/

More information on how to apply please read: https://tac.apache.org/


*Community Over Code North America - *
*October 7th to October 10th in Halifax, Canada*

Applications for this event closes on the 22nd July. We expect many
applications so please do apply as soon as you can. TAC is prioritising
applications from the North and South America regions.

More details on this event can be found at: https://communityovercode.org/

More information on how to apply please read: https://tac.apache.org/


*Have you applied to be a Speaker?*

If you have applied or intend to apply as a Speaker at either of these
events, and think you
may require assistance for Travel and/or Accommodation - TAC advises that
you do not
wait until you have been notified of your speaker status and to apply
early. Should you
not be accepted as a speaker and still wish to attend you can amend you
application to
include Conference fees, or, you may withdraw your application.

The call for presentations for Halifax is here:
https://communityovercode.org/call-for-presentations/
and you have until the 13th of July to apply.

The call for presentations for Beijing is here:
https://apachecon.com/acasia2023/cfp.html
and you have until the 18th June to apply.

*IMPORTANT Note on Visas:*

It is important that you apply for a Visa as soon as possible - do not wait
until you know if you have been accepted for Travel Assistance or not, as
due to current wait times for Interviews in some Countries, waiting that
long may be too late, so please do apply for a Visa right away. Contact
tac-ap...@tac.apache.org if you need any more information or assistance in
this area.

*Spread the Word!!*

TAC encourages you to spread the word about Travel Assistance to get to
these events, so feel free to repost as you see fit on Social Media, at
work, schools, universities etc etc...

Thank You and hope to see you all soon

Gavin McDonald on behalf of the ASF Travel Assistance Committee.

Re: [JENKINS] Lucene-9.x-Linux (64bit/hotspot/jdk-11.0.15) - Build # 11132 - Unstable!

2023-06-16 Thread Dawid Weiss 
I ran it quickly and it does reproduce on 9x:

gradlew :lucene:grouping:test  -Ptests.jvms=6 -Ptests.haltonfailure=false
 -Ptests.seed=48999F57C5128295 -Ptests.multiplier=3 -Ptests.badapples=false
-Ptests.gui=true -Ptests.file.encoding=UTF-8

   > java.lang.IndexOutOfBoundsException: Range [0, 0 + 332) out of
bounds for length 80
   > at
__randomizedtesting.SeedInfo.seed([48999F57C5128295:3AD5BA58747234E6]:0)
   > at
java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
   > at
java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromIndexSize(Preconditions.java:82)
   > at
java.base/jdk.internal.util.Preconditions.checkFromIndexSize(Preconditions.java:361)
   > at
java.base/java.util.Objects.checkFromIndexSize(Objects.java:411)
   > at
java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:180)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.store.ByteBuffersDataInput.readBytes(ByteBuffersDataInput.java:159)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.store.ByteBuffersIndexInput.readBytes(ByteBuffersIndexInput.java:85)
   > at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
/org.apache.lucene.tests.store.MockIndexInputWrapper.readBytes(MockIndexInputWrapper.java:148)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1235)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
   > at
org.apache.lucene.search.grouping.TermGroupFacetCollector$MV$SegmentResult.nextTerm(TermGroupFacetCollector.java:437)
   > at
org.apache.lucene.search.grouping.GroupFacetCollector.mergeSegmentResults(GroupFacetCollector.java:97)
   > at
org.apache.lucene.search.grouping.TestGroupFacetCollector.testRandom(TestGroupFacetCollector.java:429)



On Fri, Jun 16, 2023 at 7:40 AM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/11132/
> Java: 64bit/hotspot/jdk-11.0.15 -XX:-UseCompressedOops
> -XX:+UseConcMarkSweepGC
>
> 1 tests failed.
> FAILED:
> org.apache.lucene.search.grouping.TestGroupFacetCollector.testRandom
>
> Error Message:
> java.lang.IndexOutOfBoundsException
>
> Stack Trace:
> java.lang.IndexOutOfBoundsException
> at
> __randomizedtesting.SeedInfo.seed([48999F57C5128295:3AD5BA58747234E6]:0)
> at java.base/java.nio.Buffer.checkBounds(Buffer.java:714)
> at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:179)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.store.ByteBuffersDataInput.readBytes(ByteBuffersDataInput.java:159)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.store.ByteBuffersIndexInput.readBytes(ByteBuffersIndexInput.java:85)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.store.MockIndexInputWrapper.readBytes(MockIndexInputWrapper.java:148)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1235)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
> at
> org.apache.lucene.search.grouping.TermGroupFacetCollector$MV$SegmentResult.nextTerm(TermGroupFacetCollector.java:437)
> at
> org.apache.lucene.search.grouping.GroupFacetCollector.mergeSegmentResults(GroupFacetCollector.java:97)
> at
> org.apache.lucene.search.grouping.TestGroupFacetCollector.testRandom(TestGroupFacetCollector.java:429)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
>