ChrisHegarty commented on PR #16418: URL: https://github.com/apache/lucene/pull/16418#issuecomment-5102949655
I like this a lot. The sparse deltas with the existing codec encoding is a clean approach, and the write amplification numbers speak for themselves. The thing I keep coming back to is whether deltas need to survive across commits. That's the decision that forces the segments version bump and the compatibility implications (old readers can't open the index while overlays are active). What if you just folded back to dense at commit time? You'd still get most of the win: if refreshes are every 1s and commits every 30s, that's 30 cheap sparse delta writes between commits and then one dense rewrite when you commit. You've avoided 29 out of 30 full-column rewrites compared to today. Not as optimal as never rewriting until merge, but zero compatibility cost, segments_N stays unchanged, old Lucene can still read it, and the committed state is always a plain dense column. Curious whether you considered that path and rejected it, or whether it just fell out naturally that deltas persist. It might be worth it as a simpler first step even if the full "persist across commits" mode comes later? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
