jimczi commented on PR #16418: URL: https://github.com/apache/lucene/pull/16418#issuecomment-5122027821
Thanks both. @mikemccand right, it's a write-for-read trade: a read merges the doc across the layers with a small min-heap, newest wins. On never-updated docs paying the most, the delta layers are sparse (`IndexedDISI`), so a doc that isn't in a layer is skipped via `advance` rather than scanned, so it's ~O(log N) over the layers, not O(N) per doc, and N is bounded by `maxDocValuesOverlays` (default 16) with a merge collapsing it back to one column. I measured the sequential scan (aggregate over all live docs): ~118 ms vs ~35 ms for a plain column, 5M docs at 16 overlays (numeric), so a few times slower but bounded. I haven't isolated random-access lookup (sorting/faceting) yet, that's the honest gap, and I agree it's likely dominated by the rest of search. I'll add a probe for it. @shaie on the dynamic per-field-per-segment policy: that's already the shape of it, the fold is per field per segment and it collapses back to a single dense column once the deltas cover the whole column. Right now that triggers at full coverage; your %-threshold (say 30%) is a nice way to cap read cost earlier instead of waiting for 100%, and it lines up with the size-tiered compaction I flagged as a TODO. Today the bound is the overlay count; a coverage-% trigger could complement or replace it. Your fold-at-open idea is exactly the read-side optimization I want, and it answers Mike's question: merge the deltas into one buffer at reader open so reads behave like a plain column and you pay the merge once. Not in the first cut, to keep the mechanism minimal, but it's the top follow-up, along with `intoBitSet` and streaming the base while only patching changed docids. Good to know this connects to the old postings-update discussions too, and agreed the sweet spot is sparse fields (price, ratings_count) rather than the update-everything timestamp case DV updates were originally built for. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
