Hi Benjamin Thank you for your thoughtful reply.
I see the problem and I think i understand the tradeoffs. The challenge I see is specifically with these bullet points: > - encourage lucene legacy indices migration to lucene-ng > - deprecate lucene indices and use lucene2026 as default > - remove lucene legacy from the code base Experience shows that encouraging migration is challenging and it can take years to "encourage" the stragglers. Therefore, if feasible with reasonable effort, I believe a solution that leaves the option of automatic migration is preferable. Maybe that could be the case with if we eventually implement "let lucene-ng which is now alone also read indices of type 'lucene'". But then we have to keep supporting the two types forever into the future. That's where I would encourage looking into enhancing the way indexes are selected. I don't have the answer how that should work. And I do understand that you want the ability to toggle the new lucene format on/off in order to enable experimentation and progressive rollout. Some unstructured ideas, in the hope that they might inspire further thought. It is entirely possible that they are not good or impractical to implement. - index selection could look at the type and at the file-format version (i.e. in the :data node; indexes could even record the version in a property) - lucene-ng could be toggled on with a temporary additional property (e.g. flavor=lucene-ng, or minVersion=9) - once we want to drop the old lucene module, we could auto-migrate from the old index format to the new; this might be achieved by transcribing the old format files to the new format, if feasible, or simply by reindexing with the new format - the new format could take over type=lucene entirely and delegate to the old implementation (if present) when desired; I think this would allow for keeping the old module (largely?) unchanged - consider the future: is it likely/possible that Lucene create a new incompatible index-format? will this migration scenario repeat eventually? In any case, I welcome to see this effort to move to a more recent Lucene version! So thank you for tackling it. Regards Julian On Mon, 31 Aug 2026 at 08:39, Benjamin Habegger <[email protected]> wrote: > > Hi Julian, > > This is quite tricky. lucene 4.7 (+13 years old) has an index format that > has evolved and lucene 9 does not know how to handle them. There is some > capability in lucene to read indices in the formats of previous versions, > but the capability of reading 4.7 indices is long gone. So this > technically means that lucene 4.7 and lucene 9 indices are incompatible > formats. There were many attempts to upgrade lucene version by version but > going from 4.7 to 9 step major by major is way too bumpy. > > This implementation is based on the idea of accepting this fact and moving > forward in reimplementing (or rather porting) the oak search capabilities > for a new type of index (which technically it is besides the name). So > instead of "upgrading" indices from lucene 4.7 to 9 we are going to migrate > indices from lucene to luceneNg (lucene9, lucene2, lucene2026) just like we > migrate indices from lucene to elastic. > > Having 2 modules taking care of the same index type would introduce some > internal gymnastics that I find is added complexity for very low benefit > and loss of clarity. Also having a new type also allows to keep the legacy > lucene implementation completely untouched. If both lucene and lucene-ng > were to use the same type this would require lucene and lucene-ng to be > aware that there could be another implementation picking up responsibility > for that type of index. Internally, the type is what decides if this or > that module picks it up. > > We could add yet another field (implementation) but I don’t believe this > adds value because type is already describing the supporting index format > (lucene legacy, lucene ng, elastic, property, etc.). > > For me the transition from lucene legacy to lucene ng could look like this: > - add lucene-ng module with a new type lucene2026 in a non disturbing beta > version (no index type lucene on the runtime system, the code does not > enter into play at all) > - test live the new implementation for unseen missed and used functionality > - release a fully working version for most if not all > production environments > - encourage lucene legacy indices migration to lucene-ng > - deprecate lucene indices and use lucene2026 as default > - remove lucene legacy from the code base > > We could additionally consider in a followup (if really the type is > important to keep as lucene) > - let lucene-ng which is now alone also read indices of type "lucene" > > My argument basically summarizes to: the type of index lucene-ng handles is > technically a completely different format that the one lucene (legacy) > handles. Not the same type, not the same name. > > Best, > Benjamin > > > On Wed, Aug 26, 2026 at 4:31 PM Julian Sedding <[email protected]> wrote: > > > Hi > > > > I understand that the name of the maven module needs to be different. > > > > However, I am wondering if the index type name really needs to change? > > > > In my opinion it is unfortunate if this implementation detail leaks > > into the index definitions. Given that we control both oak-lucene > > implementations, can we not find a way to chose the correct > > implementation for type=lucene automatically? > > > > While this adds some complexity and requires some thought, I think it > > would help tremendously with adoption. > > > > Julian > > > > On Tue, 25 Aug 2026 at 09:10, Benjamin Habegger <[email protected]> wrote: > > > > > > Hello all, > > > > > > Currently OAK is using a very old version of lucene (4.7) for it's > > indices > > > of type "lucene". There have been different attempts in upgrading lucene > > to > > > newer versions but this have not succeeded and with time this has become > > > more complicated. Recently (see apache#2817 > > > <https://github.com/apache/jackrabbit-oak/pull/2817>) a new approach to > > > solving the issue was not to upgrade lucene but rather reimplement a new > > > type of index directly based on a more recent version of lucene (9+) with > > > in mind to make the code base easier to upgrade (also noting that since > > 4.7 > > > the lucene SPIs have stabilized). > > > > > > We've set up a poll here > > https://github.com/orgs/oak-indexing/discussions/1289 > > > t <https://github.com/orgs/oak-indexing/discussions/1289>o get your > > > thoughts on this. > > > > > > Best, > > > Benjamin > >
