difin commented on code in PR #5393:
URL: https://github.com/apache/hive/pull/5393#discussion_r1736910943
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -2095,6 +2097,55 @@ public List<Partition>
getPartitionsByExpr(org.apache.hadoop.hive.ql.metadata.Ta
}
}
+ @Override
+ public List<Partition>
getPartitionsWithFilter(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
+ ExprNodeDesc filter, boolean currentSpec) {
+ SearchArgument sarg = ConvertAstToSearchArg.create(conf,
(ExprNodeGenericFuncDesc) filter);
+ Expression exp = HiveIcebergFilterFactory.generateFilterExpression(sarg);
+ Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ int tableSpecId = table.spec().specId();
+ List<Partition> partitions = Lists.newArrayList();
+
+ TableScan scan =
table.newScan().filter(exp).caseSensitive(false).includeColumnStats().ignoreResiduals();
+
+ try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
+ tasks.forEach(task -> {
+ DataFile file = task.file();
+ PartitionSpec spec = task.spec();
+ if ((currentSpec && file.specId() == tableSpecId) || (!currentSpec &&
file.specId() != tableSpecId)) {
Review Comment:
I want this method to return all files for the latest spec when
latestSpecOnly=True, or all files with older spec when latestSpecOnly=False.
If we change the condition to the one above, it will always return files for
the latest spec regardless of latestSpecOnly value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
