zhangjun0x01 commented on a change in pull request #1704:
URL: https://github.com/apache/iceberg/pull/1704#discussion_r522857330
##########
File path:
core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -226,11 +226,16 @@ public RewriteDataFilesActionResult execute() {
return TableScanUtil.planTasks(splitTasks, targetSizeInBytes,
splitLookback, splitOpenFileCost);
})
.flatMap(Streams::stream)
+ .filter(task -> task.files().size() > 1)
.collect(Collectors.toList());
+ if (combinedScanTasks.isEmpty()) {
+ return RewriteDataFilesActionResult.empty();
+ }
+
List<DataFile> addedDataFiles = rewriteDataForTasks(combinedScanTasks);
- List<DataFile> currentDataFiles = filteredGroupedTasks.values().stream()
- .flatMap(tasks -> tasks.stream().map(FileScanTask::file))
+ List<DataFile> currentDataFiles = combinedScanTasks.stream()
Review comment:
`filteredGroupedTasks` contains all the datafiles that meet the
conditions of the query. `combinedScanTasks` removes some datafiles that are
smaller than targetSizeInBytes but do not need to be compressed. If when
constructing `currentDataFiles`,we use filteredGroupedTasks instead of
combinedScanTasks, those large files that do not need to be compressed will be
deleted, resulting in results Incorrect.
When I doing this PR, I forgot to consider the datefile whose size greater
than the target file size. I think it is another issue, so I opened a new PR
#1762 to fix it when I found this issue. I think we should modify #1762 first,
then modify this PR, or merge #1762 into this PR ? what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
