yanghua commented on a change in pull request #2785:
URL: https://github.com/apache/hudi/pull/2785#discussion_r609610921
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/BucketAssignFunction.java
##########
@@ -159,8 +164,11 @@ public void processElement(I value, Context ctx,
Collector<O> out) throws Except
final BucketInfo bucketInfo;
final HoodieRecordLocation location;
- if (!partitionLoadState.contains(hoodieKey.getPartitionPath())) {
+ if (bootstrapIndex &&
!partitionLoadState.contains(hoodieKey.getPartitionPath())) {
// If the partition records are never loaded, load the records first.
+
+ // The dataset may be huge, thus the processing would block for long,
Review comment:
Is it the right place to add this comment?
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -255,8 +264,14 @@ private FlinkOptions() {
public static final ConfigOption<Double> WRITE_BATCH_SIZE = ConfigOptions
.key("write.batch.size.MB")
.doubleType()
- .defaultValue(2D) // 2MB
- .withDescription("Batch buffer size in MB to flush data into the
underneath filesystem");
+ .defaultValue(64D) // 64MB
+ .withDescription("Batch buffer size in MB to flush data into the
underneath filesystem, default 64MB");
+
+ public static final ConfigOption<Integer> WRITE_LOG_BLOCK_SIZE =
ConfigOptions
+ .key("write.log_block.size")
Review comment:
Can we also add a unit like `WRITE_BATCH_SIZE `?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
