shaie commented on PR #16418: URL: https://github.com/apache/lucene/pull/16418#issuecomment-5118142773
Thanks @mikemccand for CC'ing me. I like the idea and have few comments, although I haven't reviewed the changes so if you've implemented them already, it's even better :). On the per-field override / cost of writing deltas vs full vectors: we can make the decision dynamic per the number of updates. For example, say up to 30% of docs were updated, you store the deltas. More than 30% fold all the updates to a single vector. If a segment is updated over several flushes and accumulates over 30% total updates, they're collapsed too. That way, it's per-field per-segment policy which changes dynamically with the rate of updates. The threshold (30%) is a hyperparameter you can evaluate how it impacts indexing and search throughput. On the runtime cost, how about if we folded all the updates to a single buffer, just like a segment without deltas? That way, you pay the cost once at segment open time, but otherwise reads behave just as if the segment had no deltas? And yes @mikemccand this idea resembles a discussion we've had many years ago about posting updates too - you maintain the updates in "layers/overlays" of the postings and resolve the final values at read time. At the time when we've implemented DV updates, the dominating scenario was updating numeric values such as update timestamps etc., which naturally affected the majority of the index. But I can totally see usecases for this capability, such as updating sparse fields (price, ratings_count) which will benefit from this enhancement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
