[ANNOUNCE] Apache Lucene 9.11.0 released

2024-06-06 Thread Benjamin Trent
The Lucene PMC is pleased to announce the release of Apache Lucene 9.11.0. Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for nearly any application that requires structured search, full-text search, faceting,

Re: [VOTE] Release Lucene 9.11.0 RC1

2024-06-06 Thread Benjamin Trent
It's been >72h since the vote was initiated and the result is: +1 12 (11 binding) 0 0 -1 0 This vote has PASSED Thanks! Ben Trent On Thu, Jun 6, 2024 at 12:27 AM Patrick Zhai wrote: > +1 > > SUCCESS! [1:01:30.064666] > > On Wed, Jun 5, 2024 at 11:08 AM Houston Putman wrote: > >>

[VOTE] Release Lucene 9.11.0 RC1

2024-06-03 Thread Benjamin Trent
Please vote for release candidate 1 for Lucene 9.11.0 The artifacts can be downloaded from: https://dist.apache.org/repos/dist/dev/lucene/lucene-9.11.0-RC1-rev-d433394b292e3562e0bb34222f7dd4f307e2b8ca You can run the smoke tester directly with this command: python3 -u

Re: Lucene 9.11

2024-05-29 Thread Benjamin Trent
at 12:45 AM Stefan Vodita > wrote: > >> Ben, I just merged #13414 <https://github.com/apache/lucene/pull/13414>, >> so it's not a blocker for the release. >> Thanks again for volunteering to be release manager! >> >> Stefan >> >> On Tue,

Re: Lucene 9.11

2024-05-28 Thread Benjamin Trent
gt; > +1 the 9.11 changelog looks great! > > > > On Tue, May 14, 2024 at 4:50 PM Benjamin Trent > wrote: > > Hey y'all, > > > > Looking at changes for 9.11, we are building a significant list. I > propose we do a release in the next couple of weeks. > &

Lucene 9.11

2024-05-14 Thread Benjamin Trent
Hey y'all, Looking at changes for 9.11, we are building a significant list. I propose we do a release in the next couple of weeks. While this email is a little early (I am about to go on vacation for a bit), I volunteer myself as release manager. Unless there are objections, I plan on kicking

Format metadata versioning vs. new named Formats

2024-04-12 Thread Benjamin Trent
Hey y'all, I am confused about when we should supply a new format name (e.g. Lucene911... vs. Lucene99) versus using a new metadata header version (incrementing VERSION_CURRENT). Are there general rules to follow? At first glance, using a new Lucene format name prefix is functionally the same

Re: [apache/lucene] Run failed: Run nightly: buildAndPushRelease and smokeTestRelease.py - main (df154cd)

2024-04-05 Thread Benjamin Trent
ists all releases available at: > https://archive.apache.org/dist/lucene/java/ > > version 9.10.1 isn't there so it complains with: > RuntimeError: tested version=9.10.1 but it was not released? > > I'm not sure how to deal with yet-unreleased minor versions myself! > > D

Re: [apache/lucene] Run failed: Run nightly: buildAndPushRelease and smokeTestRelease.py - main (df154cd)

2024-04-04 Thread Benjamin Trent
This seems related to us forgetting to make the back-compat indices & versions when 9.10.1 was released and me adding them later. I have since added the 9.10.1 to Version.java and version.txt in main and 9x. Now, both main and 9x have the back-compat indices (these changes were not at the same

Re: [JENKINS] Lucene » Lucene-NightlyTests-main - Build # 1315 - Still Unstable!

2024-04-02 Thread Benjamin Trent
This is me. We missed the 9.10.1 version in the 9x branch and the main branch. So, I added it. But, obviously, I didn't think about generating all the bwc indices that we didn't generate when that release was pushed. We can remove it, I would just need to adjust some new BWC tests I added that

Re: [Vote] Bump the Lucene main branch to Java 21

2024-02-23 Thread Benjamin Trent
+1 On Fri, Feb 23, 2024 at 8:54 AM Adrien Grand wrote: > +1 > > On Fri, Feb 23, 2024 at 12:54 PM Uwe Schindler wrote: > > > > Here is my +1 > > > > Uwe > > > > Am 23.02.2024 um 12:24 schrieb Chris Hegarty: > > > Hi, > > > > > > Since the discussion on bumping the Lucene main branch to Java 21

Re: [VOTE] Release Lucene 9.9.2 RC1

2024-01-25 Thread Benjamin Trent
+1 SUCCESS! [0:47:01.998711] And I verified via a local monster test that this bug is fixed: https://github.com/apache/lucene/pull/13027 I need to contribute back the monster integration test to fully exercise that code path. Thanks Chris! On Thu, Jan 25, 2024 at 11:01 AM Michael McCandless <

Re: [VOTE] Release Lucene 9.9.1 RC1

2023-12-13 Thread Benjamin Trent
SUCCESS! [1:06:02.232333] + 1! On Wed, Dec 13, 2023 at 3:26 PM Greg Miller wrote: > SUCCESS! [2:27:01.875939] > > +1 > > Thanks! > -Greg > > On Wed, Dec 13, 2023 at 3:58 AM Chris Hegarty > wrote: > >> And (short) release note: >> >>

Re: [VOTE] Release Lucene 9.9.0 RC2

2023-11-30 Thread Benjamin Trent
SUCCESS! [0:44:05.132154] +1 On Thu, Nov 30, 2023 at 1:09 PM Chris Hegarty wrote: > Please vote for release candidate 2 for Lucene 9.9.0 > > > The artifacts can be downloaded from: > > > https://dist.apache.org/repos/dist/dev/lucene/lucene-9.9.0-RC2-rev-06070c0dceba07f0d33104192d9ac98ca16fc500

Re: [VOTE] Release Lucene 9.9.0 RC1

2023-11-30 Thread Benjamin Trent
SUCCESS! [0:47:11.013106] +1 On Thu, Nov 30, 2023 at 7:16 AM Ignacio Vera wrote: > SUCCESS! [0:52:59.891964] > > > +1 > > On Thu, Nov 30, 2023 at 12:42 PM Michael McCandless < > luc...@mikemccandless.com> wrote: > >> +1 to release. >> >> I hit a corner-case test failure and opened a PR to fix

Re: Lucene 9.9.0 Release

2023-11-21 Thread Benjamin Trent
+1 9.9 will be a stellar release! Thank you Chris! On Tue, Nov 21, 2023 at 7:31 AM Adrien Grand wrote: > +1 9.9 has plenty of great changes indeed! Thanks for volunteering as a > RM, Chris. > > It would be good to try and fix the PKLookup regression that was > introduced since 9.8: >

Re: Quantization for vector search

2023-11-04 Thread Benjamin Trent
Hey Michael, In short, it's being worked on :). Could you point to the LinkedIN post? Is Nils talking about the model output quantized output or that their default output is easily compressible because of how the embeddings are built? I have done a bad job of linking back against that original

Re: Squash vs merge of PRs

2023-11-04 Thread Benjamin Trent
TL;DR, forcing non-committers to squash things is a good idea. Enforcing through some measure for committers is a bad idea. Since this thread is now in Robert's spam, I am guessing it won't have any impact :). I do not think Robert is actively trying hurt the project in any way. It seems to me

Re: Weird HNSW merge performance result

2023-10-11 Thread Benjamin Trent
Heya Patrick, What version of Lucene Util are you using? There was a bug where `forceMerge` was not actually using your configured maxConn & beamWidth. See: https://github.com/mikemccand/luceneutil/pull/232 Do you have that commit and rebuilt the KnnGraphTester? On Wed, Oct 11, 2023 at 10:10 AM

Re: Disconnectedness in HNSW graphs in Lucene

2023-08-24 Thread Benjamin Trent
o the > exact relative neighborhood graphs, allowing controlling the number of the > connections which is important for search performance. > ``` > > On 2023/08/23 16:07:55 Benjamin Trent wrote: > > Nitiraj, > > > > Good experimentation! Connectedness within layers is ind

Re: Disconnectedness in HNSW graphs in Lucene

2023-08-23 Thread Benjamin Trent
Nitiraj, Good experimentation! Connectedness within layers is indeed important. The algorithm itself should ensure connectedness of disjoint NSWs as it mutually connects nodes (selected over diversity). However, if the data is extremely clustered, this can cause connectedness to drop (few

Re: [VOTE] Release PyLucene 9.7.0-rc1

2023-07-07 Thread Benjamin Trent
+1 I tested getting ann-benchmarks updated and it worked just fine. Was also able to build locally and run some tests (non-exhaustive) on my M1 macbook. Hope everyone else has the same success! On Thu, Jul 6, 2023 at 3:47 AM Andi Vajda wrote: > > The PyLucene 9.7.0 (rc1) release tracking the

New release for PyLucene?

2023-07-05 Thread Benjamin Trent
Lucene 9.7 was just released and contains multiple desirable improvements: https://lucene.apache.org/core/9_7_0/changes/Changes.html Could we kick off a new release for PyLucene? Thanks! Ben

Re: [VOTE] Dimension Limit for KNN Vectors

2023-05-16 Thread Benjamin Trent
My vote is for option 3. Prevents Lucene from having the limit increased. Allows others who implement a different codec to set a limit of their choosing. Though I don't know the historical reasons for putting specific configuration items at the codec level. This limit is performance related and

Re: Lucene 9.6 release

2023-04-19 Thread Benjamin Trent
+1 ! You rock Alan! On Wed, Apr 19, 2023, 9:54 AM Ignacio Vera wrote: > +1 > > Thanks Alan! > > On Wed, Apr 19, 2023 at 1:27 PM Alan Woodward > wrote: > >> Hi all, >> >> It’s been a while since our last release, and we have a number of nice >> improvements and optimisations sitting in the 9x

Re: [Proposal] Remove max number of dimensions for KNN vectors

2023-04-07 Thread Benjamin Trent
>From all I have seen when hooking up JFR when indexing a medium number of vectors(1M +), almost all the time is spent simply comparing the vectors (e.g. dot_product). This indicates to me that another algorithm won't really help index build time tremendously. Unless others do dramatically fewer

Re: Welcome Ben Trent as Lucene committer

2023-01-27 Thread Benjamin Trent
Hey y'all! This is truly an honor! Well, I am Ben Trent and have been writing code for over a decade now. Which I know is not a very long time compared to most folks. I originally wanted to do research and work in pure mathematics (my baccalaureate), but quickly realized I am nowhere near smart

Adding new extension point

2022-11-10 Thread Benjamin Trent
Hey y'all, I am new to this type of workflow, I am used to github and Pull-requests. What is the process for adding a new extension point? With a recent foray into getting Lucene KNN into the ann-benchmarks repository, I found the need to adjust the current codec (Lucene94Codec) to allow us to

Re: [VOTE] Release PyLucene 9.4.1-rc3

2022-11-01 Thread Benjamin Trent
+1 from me On Tue, Nov 1, 2022 at 4:37 PM Andi Vajda wrote: > > The PyLucene 9.4.1 (rc3) release tracking the recent release of > Apache Lucene 9.4.1 is ready. > > A release candidate is available from: > https://dist.apache.org/repos/dist/dev/lucene/pylucene/9.4.1-rc3/ > > PyLucene 9.4.1

Re: [VOTE] Release PyLucene 9.4.1-rc2

2022-11-01 Thread Benjamin Trent
Andi, I pulled down the rc-2 and tested in Docker on Ubuntu 18.04.6 LTS (Bionic Beaver), with Python 3.6.9, I get the following error attempting to build JCC jcc3/sources/functions.cpp: In function 'void installType(PyTypeObject**, PyType_Def*, PyObject*, char*, int)':

Re: [VOTE] Release PyLucene 9.4.1

2022-11-01 Thread Benjamin Trent
+1 On Mon, Oct 31, 2022 at 6:50 PM Jeff Breidenbach wrote: > +1 > > On Mon, Oct 31, 2022, 3:50 PM Andi Vajda wrote: > > > > > The PyLucene 9.4.1 (rc1) release tracking the recent release of > > Apache Lucene 9.4.1 is ready. > > > > A release candidate is available from: > >

Re: New release of PyLucene?

2022-10-31 Thread Benjamin Trent
, 2022 at 11:50 AM Andi Vajda wrote: > > > On Oct 31, 2022, at 07:55, Benjamin Trent wrote: > > > > Lucene 9.4.1 was recently released. The last version of PyLucene > released > > was 9.1.0. There have been some improvements to Lucene since then. My > > par

New release of PyLucene?

2022-10-31 Thread Benjamin Trent
Lucene 9.4.1 was recently released. The last version of PyLucene released was 9.1.0. There have been some improvements to Lucene since then. My particular concern is around KNN search. What is the process to start a new release of PyLucene? Thank you! Ben