rangareddy commented on issue #3733: URL: https://github.com/apache/hudi/issues/3733#issuecomment-5102141478
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: The root cause here was correctly identified in the thread itself. As you noted on 2021-10-04, the spikes coincided with a backfill loading old records into the Kafka stream, so a single batch spanned many `year/month/day/hour` partitions instead of the usual 1-8, and bloom index lookup fanned out across all of them. With roughly 8,750 partitions per year of data, that fan-out is the expected cost of a partition-scoped bloom index rather than a defect -- which is why the reply at the time invited a contribution rather than proposing a fix. Two things have landed since that bear directly on it: - **Metadata table on by default from 0.12.** At your partition count, the per-partition file listing against S3 was a large part of what you were measuring; the metadata table removes it. - **Record index, added in 0.14.0** (confirmed against `HoodieIndex.IndexType`: absent at `release-0.13.1`, present at `release-0.14.0`; on current master split into `GLOBAL_RECORD_LEVEL_INDEX` and `RECORD_LEVEL_INDEX`). It stores key-to-location mappings in the metadata table with sharding, so lookup cost does not scale with how many partitions a batch happens to touch. That is the direct answer to a backfill batch spanning thousands of partitions. For the record, this issue was already declared closed for inactivity on 2022-09-13 but left open; closing it now to match that. If you still run this pipeline and index-lookup spikes during backfills are still a problem, `RECORD_LEVEL_INDEX` on a recent version is the concrete thing to try, and a fresh issue with before/after numbers would be genuinely useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
