Re: Intra-segment search concurrency implementation

2024-08-01 Thread Luca Cavanna
Hey Alan, Thanks for the feedback. I need to give it some more thought, but I kind of assumed that we would not want to create different instances of leaf reader context for partitions of the same segment. The mapping between the physical layout of a segment and leaf reader context should remain 1

Re: Intra-segment search concurrency implementation

2024-07-31 Thread Alan Woodward
Hi Luca, This is very exciting! I haven’t followed the dev process very closely so far, so this may already have been looked at and dismissed as unworkable for various reasons, but I’m wondering if we definitely need a new abstraction for a LeafReaderContext partition? Could we instead find a

Intra-segment search concurrency implementation

2024-07-29 Thread Luca Cavanna
Hey all, I have been working on an initial implementation of intra-segment search concurrency for Lucene. My goal is to introduce the ability to concurrently search partitions of the same segment, think of a force-merged segment for instance, in a way that's as transparent as possible to users. Th