difin commented on code in PR #5052:
URL: https://github.com/apache/hive/pull/5052#discussion_r1473198987
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java:
##########
@@ -328,4 +334,25 @@ public static PartitionData toPartitionData(StructLike
key, Types.StructType key
}
return data;
}
+
+ public static void getFiles(Table table, List<DataFile> dataFiles,
List<DeleteFile> deleteFiles) {
+ CloseableIterable<FileScanTask> fileScanTasks =
table.newScan().planFiles();
+ Set<String> dataFilesPath = Sets.newHashSet();
+ Set<String> deleteFilesPath = Sets.newHashSet();
+
+ for (FileScanTask fileScanTask : fileScanTasks) {
+ // filter repeated data files
Review Comment:
I am not sure under which cases there can be repeated files, I added that
because I saw such duplicates prevention when getting data and delete files in
a code of another engine which also implements Iceberg compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
