chrevanthreddy opened a new issue, #19500: URL: https://github.com/apache/hudi/issues/19500
Parent: #19094 Related: #18676 ## Summary Implement LIRE (Local Incremental Rebalancing) for sustained vector-cluster skew after Tier-1 vector compaction has landed and soaked. LIRE changes local centroid geometry but does not use persisted centroid epochs. One completed MDT delta commit publishes the replacement routing and posting state atomically; `routingVersion` exists only to invalidate caches. ## Scope - Evaluate `ClusterStats` split/merge triggers for clusters touched by recent maintenance. - Fetch authoritative raw vectors using the positional reader shared with exact rerank. - Run local balanced 2-means for oversized clusters. - Mint replacement cluster IDs and re-encode every affected vector against its replacement centroid. - Merge undersized clusters into an appropriate surviving centroid and re-encode their vectors. - Evaluate neighboring leaves directly; do not route LIRE neighbor discovery through the coarse map. - Apply bounded neighbor reassignment and publish replacement centroid chunks, posting blocks, tombstones, and statistics in one MDT delta commit. - Rewrite the whole affected cluster when its shard geometry permits; escalate to a new generation when the bounded one-commit rewrite cannot be guaranteed. - Bump `routingVersion` for cache invalidation only. - Track assignment-mismatch metrics and trigger coarse-routing rebuild/escalation when thresholds are exceeded. ## Consistency requirements - Readers observe either the pre-rebalance or post-rebalance MDT snapshot, never a mixed routing view. - Persisted schema and keys contain no centroid epoch dimension. - Residual codes are always regenerated when a centroid moves. - Old clusters/postings are tombstoned in the same commit that publishes replacements. ## Acceptance criteria - Adversarial skew fixtures deterministically trigger split and merge paths. - Tests cover split, merge, bounded neighbor reassignment, one-commit visibility, failure before commit, retry/idempotence, cache invalidation, and generation escalation. - Coarse-routing mismatch metrics and rebuild triggers are observable. ## Dependencies - Epoch removal in #19097 / PR #19317. - Incremental update hook and additive `ClusterStats`. - Freshness/frontier contract. - Tier-1 vector compaction/repacking and locator refresh. ## Scheduling Deferred behind the correctness and bounded-growth critical path. Synthetic skew is required for certification because organic production-like churn may not trigger LIRE for weeks or months. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
