rangareddy commented on issue #3751: URL: https://github.com/apache/hudi/issues/3751#issuecomment-5102148156
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: This was a configuration-tuning thread and it largely resolved in-thread. Two concrete corrections were made along the way: - The reason for choosing COW over MOR was Athena support, but Athena has supported MOR since 2021-07-16; and if you prefer COW-like query semantics, Hive-syncing a MOR table gives you both `foo` and `foo_rt`, with `foo` behaving like a read-optimized COW table. - The resource shape was the main bottleneck: 19 executors at `executor-cores 1` caps effective concurrency at roughly 19-38 tasks, while the Spark and Hudi parallelism settings were much higher, so those settings could not take effect. Raising cores to 3-5 and aligning `spark.default.parallelism`, `spark.sql.shuffle.partitions` and the `hoodie.*.shuffle.parallelism` values is what moved the needle. Note also that `hoodie.datasource.write.row.writer.enable` applied only to `bulk_insert` at the time, so it was not affecting your upsert path. You reported a reworked, more stable setup on 2022-04-25 after upgrading to 0.10.0 and moving from `GLOBAL_BLOOM` to `BLOOM`. The follow-up request on 2022-09-07 for partition count and Spark stage screenshots was never answered, and there has been no activity since. Closing as stale, since no Hudi-side defect was isolated and the report predates several rewrites of the write path. For the underlying "many small input files" shape, the current recommendation is `bulk_insert` (or `insert`) plus scheduled **clustering** rather than paying small-file packing on every micro-batch. If write speed is still a problem on a recent version, please open a fresh issue with the Hudi version, partition count and Spark UI stage timings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
