voonhous commented on PR #19906: URL: https://github.com/apache/hudi/pull/19906#issuecomment-5637694386
Per-class test times in test-spark-java17-fta, the one job above baseline in all three runs. Seconds per class from the surefire lines of each job log; "mean d" is the mean of the three runs minus the baseline. | class | #19892 run | run 3 | run 4 | run 5 | mean d | |---|---|---|---|---|---| | TestHoodieClientOnCopyOnWriteStorage | 417 | 518 | 530 | 489 | +95 | | TestHoodieBackedMetadata | 543 | 612 | 637 | 580 | +66 | | TestHoodieClientMultiWriter | 288 | 338 | 332 | 327 | +45 | | TestHoodieSparkMergeOnReadTableRollback | 143 | 170 | 176 | 163 | +26 | | TestRecordLevelIndex | 185 | 207 | 231 | 190 | +24 | | TestLSMDataSource | 76 | 98 | 103 | 91 | +21 | | TestHoodieIndex | 123 | 137 | 157 | 132 | +19 | | TestSavepointRestoreMergeOnRead | 105 | 127 | 125 | 117 | +18 | | TestMORDataSourceStorage | 82 | 98 | 107 | 93 | +17 | | TestRliLookupMetricsOnSparkSql | 44 | 60 | 64 | 57 | +16 | | whole job, 45 classes | 2769 | 3291 | 3469 | 3119 | +524 | No class in this job got faster, and the slowdown is broad rather than one outlier, so it is the setting and not noise. These are write-heavy client tests driven through RDDs: many commits of a few thousand records each, where every Hudi write stage now runs two tasks instead of four on a four-core runner. The SQL suites that gained (dml.others, dml.insert, the functional package) are the opposite shape: many tiny shuffles where 4 partitions and adaptive planning were overhead. That points at `spark.default.parallelism` as the part that hurts and `spark.sql.shuffle.partitions` plus AQE off as the part that pays. Next step is a local A/B on TestHoodieClientOnCopyOnWriteStorage and TestMergeIntoTable across three settings: baseline (4/4, AQE on), this PR (2/2, AQE off), and a hybrid (shuffle 2, AQE off, default parallelism 4). If the hybrid keeps the SQL gain without the RDD loss, the PR changes to that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
