Hi Julian, This is quite tricky. lucene 4.7 (+13 years old) has an index format that has evolved and lucene 9 does not know how to handle them. There is some capability in lucene to read indices in the formats of previous versions, but the capability of reading 4.7 indices is long gone. So this technically means that lucene 4.7 and lucene 9 indices are incompatible formats. There were many attempts to upgrade lucene version by version but going from 4.7 to 9 step major by major is way too bumpy.
This implementation is based on the idea of accepting this fact and moving forward in reimplementing (or rather porting) the oak search capabilities for a new type of index (which technically it is besides the name). So instead of "upgrading" indices from lucene 4.7 to 9 we are going to migrate indices from lucene to luceneNg (lucene9, lucene2, lucene2026) just like we migrate indices from lucene to elastic. Having 2 modules taking care of the same index type would introduce some internal gymnastics that I find is added complexity for very low benefit and loss of clarity. Also having a new type also allows to keep the legacy lucene implementation completely untouched. If both lucene and lucene-ng were to use the same type this would require lucene and lucene-ng to be aware that there could be another implementation picking up responsibility for that type of index. Internally, the type is what decides if this or that module picks it up. We could add yet another field (implementation) but I don’t believe this adds value because type is already describing the supporting index format (lucene legacy, lucene ng, elastic, property, etc.). For me the transition from lucene legacy to lucene ng could look like this: - add lucene-ng module with a new type lucene2026 in a non disturbing beta version (no index type lucene on the runtime system, the code does not enter into play at all) - test live the new implementation for unseen missed and used functionality - release a fully working version for most if not all production environments - encourage lucene legacy indices migration to lucene-ng - deprecate lucene indices and use lucene2026 as default - remove lucene legacy from the code base We could additionally consider in a followup (if really the type is important to keep as lucene) - let lucene-ng which is now alone also read indices of type "lucene" My argument basically summarizes to: the type of index lucene-ng handles is technically a completely different format that the one lucene (legacy) handles. Not the same type, not the same name. Best, Benjamin On Wed, Aug 26, 2026 at 4:31 PM Julian Sedding <[email protected]> wrote: > Hi > > I understand that the name of the maven module needs to be different. > > However, I am wondering if the index type name really needs to change? > > In my opinion it is unfortunate if this implementation detail leaks > into the index definitions. Given that we control both oak-lucene > implementations, can we not find a way to chose the correct > implementation for type=lucene automatically? > > While this adds some complexity and requires some thought, I think it > would help tremendously with adoption. > > Julian > > On Tue, 25 Aug 2026 at 09:10, Benjamin Habegger <[email protected]> wrote: > > > > Hello all, > > > > Currently OAK is using a very old version of lucene (4.7) for it's > indices > > of type "lucene". There have been different attempts in upgrading lucene > to > > newer versions but this have not succeeded and with time this has become > > more complicated. Recently (see apache#2817 > > <https://github.com/apache/jackrabbit-oak/pull/2817>) a new approach to > > solving the issue was not to upgrade lucene but rather reimplement a new > > type of index directly based on a more recent version of lucene (9+) with > > in mind to make the code base easier to upgrade (also noting that since > 4.7 > > the lucene SPIs have stabilized). > > > > We've set up a poll here > https://github.com/orgs/oak-indexing/discussions/1289 > > t <https://github.com/orgs/oak-indexing/discussions/1289>o get your > > thoughts on this. > > > > Best, > > Benjamin >
