WinkerDu commented on a change in pull request #3073:
URL: https://github.com/apache/iceberg/pull/3073#discussion_r710365448
##########
File path: core/src/main/java/org/apache/iceberg/util/TableScanUtil.java
##########
@@ -56,7 +57,10 @@ public static boolean hasDeletes(FileScanTask task) {
Preconditions.checkArgument(lookback > 0, "Invalid split planning lookback
(negative or 0): %s", lookback);
Preconditions.checkArgument(openFileCost >= 0, "Invalid file open cost
(negative): %s", openFileCost);
- Function<FileScanTask, Long> weightFunc = file -> Math.max(file.length(),
openFileCost);
+ // Check the size of delete file as well to avoid unbalanced bin-packing
+ Function<FileScanTask, Long> weightFunc = file -> Math.max(
+ file.length() +
file.deletes().stream().mapToLong(ContentFile::fileSizeInBytes).sum(),
Review comment:
Thanks for your reply @openinx . Actually I don't think it's necessary
to separate delete file to eq-delete and pos-delete when calculating cost.
besides the algorithm here would be complicated, the top-level logic to apply
delete files is
`applyEqDeletes(applyPosDeletes(records))`
Generally I think the base logic above for eq-delete and pos-delete is the
same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
