gudladona commented on issue #20489: URL: https://github.com/apache/hudi/issues/20489#issuecomment-6670304408
Thanks for the quick review. I'll send a PR for (1)–(5), with both of your notes applied: - **(3)/(4):** `withBulkInsertParallelism` is dropped instead of moved. I confirmed on master that the clustering write paths pass `numOutputGroups` explicitly. The only readers of `BULKINSERT_PARALLELISM_VALUE` are the regular bulk-insert action (`SparkBulkInsertHelper.bulkInsert(..., config.getBulkInsertShuffleParallelism(), ...)`) and `HoodieDatasetBulkInsertHelper.prepareForBulkInsert`, and clustering calls neither. All groups of a run then share one write config. It is re-derived if the strategy's write config is replaced; `SparkBinaryCopyClusteringExecutionStrategy` does that before clustering. - **(6)** stays out of this PR. I'll follow up separately once this lands. Two smaller things the PR also covers, since they're the same pattern: - `SparkConsistentBucketClusteringExecutionStrategy` also builds a write config per group; it uses the shared one as well. - Each `SparkReaderContextFactory` creates three broadcasts (base-file reader, Hadoop configuration, table config). Building the factory once per run instead of once per group also removes three broadcasts per group. The reproducing test from the issue (`TestClusteringTaskBinaryPayloads`) will be part of the PR as the regression gate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
