vinothchandar commented on a change in pull request #1602:
URL: https://github.com/apache/hudi/pull/1602#discussion_r436636150
##########
File path:
hudi-client/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java
##########
@@ -54,6 +54,12 @@
public static final String PARQUET_SMALL_FILE_LIMIT_BYTES =
"hoodie.parquet.small.file.limit";
// By default, treat any file <= 100MB as a small file.
public static final String DEFAULT_PARQUET_SMALL_FILE_LIMIT_BYTES =
String.valueOf(104857600);
+ // Hudi will use the previous commit to calculate the estimated record size
by totalBytesWritten/totalRecordsWritten.
+ // If the previous commit is too small to make an accurate estimation, Hudi
will search commits in the reverse order,
+ // until find a commit has totalBytesWritten larger than
(PARQUET_SMALL_FILE_LIMIT_BYTES * RECORD_SIZE_ESTIMATION_THRESHOLD)
+ public static final String RECORD_SIZE_ESTIMATION_THRESHOLD =
"hoodie.record.size.estimation.threshold";
Review comment:
rename: RECORD_SIZE_ESTIMATION_THRESHOLD_PROP
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
