github-actions[bot] commented on code in PR #66481:
URL: https://github.com/apache/doris/pull/66481#discussion_r3755944900
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java:
##########
@@ -665,7 +670,9 @@ private List<ConnectorScanRange> planScanInternal(
}
int formatVersion = getFormatVersion(table);
- List<String> orderedPartitionKeys =
IcebergPartitionUtils.getIdentityPartitionColumns(table);
+ Schema scanSchema = pinnedSchema(table, iceHandle);
+ List<String> orderedPartitionKeys =
+ IcebergPartitionUtils.getIdentityPartitionColumns(table,
scanSchema);
Review Comment:
This list now intentionally unions identity keys across all specs, but
`partitioned` two lines below is still computed once from the latest
`table.spec()`. After the current spec removes its last partition field,
retained files can still have `dataFile.specId()` pointing to an old identity
spec; `computePerFileInvariants` then skips that file's partition metadata
entirely even though FE classifies the old field as a `PARTITION_KEY`. For an
imported old-spec file whose `p` exists only in manifest partition data, BE
receives no `columns_from_path` and materializes NULL/default instead of the
manifest value. Make the partition-presence decision from each data file's spec
and add eager/streaming coverage for a retained identity-spec file after the
current spec becomes unpartitioned.
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java:
##########
@@ -301,7 +302,7 @@ basePath, inputFormat, serdeLib, columnNames, columnTypes,
partitionFieldNames(m
List<ConnectorScanRange> ranges = new ArrayList<>();
for (String partitionPath : partitionPaths) {
Map<String, String> partValues = parsePartitionValues(
- partitionPath, hudiHandle.getPartitionKeyNames());
+ partitionPath, hudiHandle.getPartitionKeyNames(),
hiveStylePartitioning);
Review Comment:
This new layout argument assumes `partitionPath` came from the physical Hudi
layout, but `applyFilter` can source it from HMS. With Doris
`use_hive_sync_partition=true` and Hudi
`hoodie.datasource.write.hive_style_partitioning=false`, the handle contains
the HMS name `city=Beijing` while the physical directory/FileSystemView key is
positional `Beijing`. This call parses the HMS name as the literal positional
value `city=Beijing`, and the same nonphysical string is passed to the exact
COW/MOR FileSystemView lookup, so the predicate-pruned scan produces no split.
Preserve source/layout provenance and translate matched HMS partitions to
physical relative paths (or prune the physical Hudi list), then add a
sync-enabled positional-layout test that returns the split and value `Beijing`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
