LuciferYang commented on code in PR #10663:
URL: https://github.com/apache/paimon/pull/10663#discussion_r4321377776
##########
paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/utils/HiveSplitGenerator.java:
##########
@@ -130,6 +128,29 @@ public static InputSplit[] generateSplits(
return splits.toArray(new InputSplit[0]);
}
+ private static List<String> locationsToPlan(
+ FileStoreTable table, JobConf jobConf, String locations) {
+ List<String> belonging = new ArrayList<>();
+ String foreign = null;
+ for (String location : locations.split(",")) {
+ // Hive create dummy file for empty table or partition. If this
location doesn't
+ // belong to this table, nothing to do.
+ if (location.startsWith(table.location().toString())) {
+ belonging.add(location);
+ } else if (foreign == null && !location.isEmpty()) {
+ foreign = location;
+ }
+ }
+ // With LOCATION_IN_PROPERTIES, Hive gives the table a location under
its own warehouse,
+ // which does not start with the paimon location. If the table has no
Hive partitions,
+ // that location stands for the whole table. Hive partitions of such a
table all share
+ // it, so they cannot be told apart and are still skipped.
+ if (belonging.isEmpty() && foreign != null &&
!jobConf.getBoolean(HIVE_PARTITIONED, true)) {
+ return Collections.singletonList(foreign);
Review Comment:
Fixed in fadf12aa03. The Hive location is now read with the pushed-down
query filter only: no partition predicate or tag is taken from its path, and
the split keeps the path for Hive. `HiveSplitGeneratorTest#testForeignLocation`
adds `/hive/pt=plain/warehouse/t`, which now plans both partitions instead of
only `plain`.
The rest of this PR was already merged as #10589, so after merging master
this PR only carries this fix. I updated the title and description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
