cshuo commented on code in PR #19992:
URL: https://github.com/apache/hudi/pull/19992#discussion_r4045165209
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/FileStatsIndex.java:
##########
@@ -384,20 +384,26 @@ private static Object doUnpack(
return converter.convert(rawVal);
}
+ /**
+ * Reads statistics for the requested columns and relative partition paths.
+ * An empty partition list reads all partitions using column-only prefixes.
+ */
@VisibleForTesting
- public List<RowData> readColumnStatsIndexByColumns(String[] targetColumns) {
- // NOTE: If specific columns have been provided, we can considerably trim
down amount of data fetched
- // by only fetching Column Stats Index records pertaining to the
requested columns.
- // Otherwise, we fall back to read whole Column Stats Index
+ public List<RowData> readColumnStatsIndexByColumns(String[] targetColumns,
List<String> candidatePartitions) {
ValidationUtils.checkArgument(targetColumns.length > 0,
"Column stats is only valid when push down filters have referenced
columns");
// Read Metadata Table's column stats Flink's RowData list by
- // - Fetching the records by key-prefixes (column names)
+ // - Fetching the records by key-prefixes (column names and candidate
partitions, when provided)
// - Deserializing fetched records into [[RowData]]s
- List<ColumnStatsIndexPrefixRawKey> rawKeys = Arrays.stream(targetColumns)
- .map(ColumnStatsIndexPrefixRawKey::new) // Just column name, no
partition
- .collect(Collectors.toList());
+ List<ColumnStatsIndexPrefixRawKey> rawKeys;
+ if (candidatePartitions.isEmpty()) {
+ rawKeys =
Arrays.stream(targetColumns).map(ColumnStatsIndexPrefixRawKey::new).collect(Collectors.toList());
+ } else {
+ rawKeys = candidatePartitions.stream().distinct()
+ .flatMap(partition -> Arrays.stream(targetColumns).map(column -> new
ColumnStatsIndexPrefixRawKey(column, partition)))
+ .collect(Collectors.toList());
Review Comment:
One concern for queries whose candidate files still span all partitions:
this expands the lookup from C column prefixes to P × C partition-column
prefixes without reducing the statistics returned. Although the prefixes are
sorted and the reader can advance through the keys, the additional prefix
encoding, sorting, and lookup processing could increase query-planning overhead
for highly partitioned tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
