voonhous commented on code in PR #20328:
URL: https://github.com/apache/hudi/pull/20328#discussion_r4488277082
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2253,27 +2215,34 @@ private static
List<HoodieColumnRangeMetadata<Comparable>> getFileStatsRangeMeta
return readColumnRangeMetadataFrom(partitionPath, fileName,
datasetMetaClient, columnsToIndex, maxBufferSize, indexVersion);
}
- public static HoodieData<HoodieRecord>
convertMetadataToPartitionStatsRecords(HoodiePairData<String,
List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadataPartitionPair,
-
HoodieTableMetaClient dataMetaClient,
-
Map<String, HoodieSchema> colsToIndexSchemaMap,
-
HoodieIndexVersion partitionStatsIndexVersion
- ) {
- try {
- return columnRangeMetadataPartitionPair
- .flatMapValues(List::iterator)
- .groupByKey()
- .map(pair -> {
- final String partitionName = pair.getLeft();
- return collectAndProcessColumnMetadata(pair.getRight(),
partitionName, isShouldScanColStatsForTightBound(dataMetaClient),
Option.empty(), colsToIndexSchemaMap, partitionStatsIndexVersion);
- })
- .flatMap(recordStream -> recordStream.iterator());
- } catch (Exception e) {
- throw new HoodieException("Failed to generate column stats records for
metadata table", e);
- }
- }
-
- public static boolean
isShouldScanColStatsForTightBound(HoodieTableMetaClient dataMetaClient) {
- return
MetadataPartitionType.COLUMN_STATS.isMetadataPartitionAvailable(dataMetaClient);
+ /**
+ * Generates partition statistics only for columns with statistics covering
every expected base and log file.
+ * Incomplete columns produce partition stats delete records to invalidate
any previously stored bounds for those columns.
+ * The expected file names must come from the file inventory, independently
of statistics extraction.
+ */
+ public static Stream<HoodieRecord>
convertMetadataToPartitionStatsRecords(String partitionPath,
+
List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata,
+
Set<String> expectedFileNames,
+
Map<String, HoodieSchema> colsToIndexSchemaMap,
+
HoodieIndexVersion partitionStatsIndexVersion) {
+ Map<String, List<HoodieColumnRangeMetadata<Comparable>>> columnMetadataMap
= fileColumnMetadata.stream()
+ // Negative counts belong to missing/deleted statistics, whereas
all-null and empty files have valid counts.
+ .filter(range -> range.getValueCount() >= 0 && range.getNullCount() >=
0)
+
.collect(Collectors.groupingBy(HoodieColumnRangeMetadata::getColumnName));
+ return colsToIndexSchemaMap.keySet().stream().flatMap(column -> {
+ List<HoodieColumnRangeMetadata<Comparable>> ranges =
columnMetadataMap.getOrDefault(column, Collections.emptyList());
+ Set<String> coveredFiles = ranges.stream()
+ .map(range -> new StoragePath(range.getFilePath()).getName())
+ .collect(Collectors.toSet());
+ if (expectedFileNames.isEmpty() ||
!coveredFiles.containsAll(expectedFileNames)) {
Review Comment:
When a column is added via schema evolution and default list at play, the
new column joins the index automatically. However,
`ParquetUtils#readColumnStatsFrommetadata` only emit ranges from columns that
exist in the file footer, hence, files that were written before the schema
evolution never "cover" it IIUC.
Could we treat a file that lacks the column as an all-null range, or at
least add an ADD COLUMNS test that pins the intended behaviour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]