wombatu-kun commented on code in PR #19115:
URL: https://github.com/apache/hudi/pull/19115#discussion_r3497525581
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SparkSampleWritesUtils.java:
##########
@@ -108,7 +110,14 @@ private static Pair<Boolean, String>
doSampleWrites(JavaSparkContext jsc, Option
try (SparkRDDWriteClient sampleWriteClient = new SparkRDDWriteClient(new
HoodieSparkEngineContext(jsc), sampleWriteConfig, Option.empty())) {
int size = writeConfig.getIntOrDefault(SAMPLE_WRITES_SIZE);
return recordsOpt.map(records -> {
- List<HoodieRecord> samples = records.coalesce(1).take(size);
+ // Rewrite each record with an empty partition path so the
sample-writes table is
+ // effectively non-partitioned. The bulk-insert writer routes records
to files by
+ // HoodieRecord.getPartitionPath(); without this, an input that spans
many source
+ // partitions fans out into many tiny files even though parallelism is
1, slowing
+ // the sample write and inflating per-file metadata in the size
estimate.
+ List<HoodieRecord> samples = records.coalesce(1).take(size).stream()
+ .map(r -> r.newInstance(new HoodieKey(r.getRecordKey(), "")))
Review Comment:
Impact is bounded to the first commit: getWriteConfigWithRecordSizeEstimate
early-returns once the timeline is non-empty (SparkSampleWritesUtils.java:69),
and from the next commit on AverageRecordSizeEstimator.averageBytesPerRecord
recomputes from real partitioned commit stats, using this config only as the
empty-timeline fallback (AverageRecordSizeEstimator.java:70-105). That
estimator also subtracts a per-file metadata estimate and skips below-threshold
commits (AverageRecordSizeEstimator.java:86,90,93), so the target semantic is
data bytes per record, not footer overhead - which is what the single-file
sample now measures. A sparse-partition under-estimate self-corrects after the
first commit, and a bounded-partition layout would partly reintroduce the
footer overhead this removes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
