boneanxs commented on code in PR #8452:
URL: https://github.com/apache/hudi/pull/8452#discussion_r1201575964
##########
hudi-common/src/main/java/org/apache/hudi/metadata/FileSystemBackedTableMetadata.java:
##########
@@ -96,11 +109,32 @@ public List<String>
getPartitionPathWithPathPrefixes(List<String> relativePathPr
}
private List<String> getPartitionPathWithPathPrefix(String
relativePathPrefix) throws IOException {
+ return
getPartitionPathWithPathPrefixUsingFilterExpression(relativePathPrefix, null,
null);
+ }
+
+ private List<String>
getPartitionPathWithPathPrefixUsingFilterExpression(String relativePathPrefix,
+
Types.RecordType partitionFields,
+
Expression expression) throws IOException {
List<Path> pathsToList = new CopyOnWriteArrayList<>();
pathsToList.add(StringUtils.isNullOrEmpty(relativePathPrefix)
- ? new Path(datasetBasePath) : new Path(datasetBasePath,
relativePathPrefix));
+ ? dataBasePath.get() : new Path(dataBasePath.get(),
relativePathPrefix));
List<String> partitionPaths = new CopyOnWriteArrayList<>();
+ int partitionLevel = -1;
+ boolean needPushDownExpressions;
+ // Not like `HoodieBackedTableMetadata`, since we don't know the exact
partition levels here,
+ // given it's possible that partition values contains `/`, which could
affect
+ // the result to get right `partitionValue` when listing paths, here we
have
+ // to make it more strict that `urlEncodePartitioningEnabled` must be
enabled.
+ // TODO better enable urlEncodePartitioningEnabled if
hiveStylePartitioningEnabled is enabled?
Review Comment:
Checking hudi configure here is difficult, since`HoodieConfig` is not passed
when construct `FileSystemBackedTableMetadata`, that's why this class lefts
some TODOs here like `TODO: Get the parallelism from HoodieWriteConfig` and
`get `caseSensitive` from HoodieConfig`.
For now, I think it's safe if we both check `hiveStylePartitioningEnabled`
and `urlEncodePartitioningEnabled` when using `FileSystemBackedTableMetadata`
and enable partition push down filters.
And I'll try to resolve TODOs relate to HoodieConfig in a follow-up pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
