rangareddy commented on issue #3975: URL: https://github.com/apache/hudi/issues/3975#issuecomment-5101235479
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: The core constraint behind this thread -- "global indexes prevent the duplicates I need to avoid, but they do not scale as partition count grows" -- has since been addressed, though not in the form originally requested. Your literal proposal was to restrict a global index to a named subset of partitions. That was never implemented; as noted on 2022-09-13 it is difficult to generalise beyond date-style partitioning. What landed instead is a global index that actually scales: the **record index**, a metadata-table-backed key-to-location index with sharding, added in **0.14.0** (confirmed against `HoodieIndex.IndexType` -- absent at `release-0.13.1`, present at `release-0.14.0`). On current master it has been split into `GLOBAL_RECORD_LEVEL_INDEX` (uniqueness across all partitions, which is your requirement) and `RECORD_LEVEL_INDEX` (per-partition), with the original `RECORD_INDEX` deprecated in favour of those. That removes the need for the delete-then-upsert dance you built to work around `GLOBAL_SIMPLE`, and with it the slow delete pass. Bucket index and the consistent-hashing index are also available if you prefer a hash-based approach. On the delete slowness specifically: your observation that a 9k-row delete cost about the same as a 500-row delete was the right read -- the time was going into the per-partition small-file lookup, not into rewriting base files. In current code that lookup is skipped outright when `hoodie.parquet.small.file.limit` is `0` (`UpsertPartitioner.java:277`) and is otherwise parallelised across partitions rather than run serially (`UpsertPartitioner.java:287`), so the `hoodie.parquet.small.file.limit=0` workaround suggested at the time is now a genuine bypass rather than a partial one. Closing as stale -- there has been no substantive discussion since 2022-11-04, and re-measuring a 0.10-era delete-vs-upsert timing would not be meaningful against a write path that has since changed substantially. If you are still on this pattern, the concrete thing to try is `GLOBAL_RECORD_LEVEL_INDEX` on a recent version and dropping the separate delete step entirely. If that still underperforms for you, please open a fresh issue with the Hudi version, partition count, table size and the Spark UI stage timings -- a current-version report we can act on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
