jimczi commented on PR #16418: URL: https://github.com/apache/lucene/pull/16418#issuecomment-5109773219
Yeah I looked at that. Deltas need to survive commits because the whole point is to commit often without rewriting the full column each time. Fold-at-commit only helps when commits are rare compared to refreshes. In the case I care about a refresh is also a commit, so you'd fold to dense on every refresh, which is just what we do today. Your 29/30 holds when refresh:commit is big, at 1:1 it's 0/1. I don't think the compat cost is a real blocker either. An old reader rejects the index up front (`IndexFormatTooNewException`, before any codec runs), and I made the bump deterministic like you said in the other comment (it always bumps when the feature is on now, see 3eb7cf1) so it only depends on the config. Commits with the feature off are byte-for-byte the same as today, so backports are safe. The real cost of keeping deltas around is on the read side (the overlay merge). But there's a lot we can do there: `intoBitSet`, reading the dense base directly and only patching the changed docids, or folding the deltas into one small patch at reader open. That's a bigger scope though, so I'd rather land the mechanism first and do those as follow-ups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
