cshuo commented on code in PR #20328:
URL: https://github.com/apache/hudi/pull/20328#discussion_r4488665943


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2253,27 +2215,34 @@ private static 
List<HoodieColumnRangeMetadata<Comparable>> getFileStatsRangeMeta
     return readColumnRangeMetadataFrom(partitionPath, fileName, 
datasetMetaClient, columnsToIndex, maxBufferSize, indexVersion);
   }
 
-  public static HoodieData<HoodieRecord> 
convertMetadataToPartitionStatsRecords(HoodiePairData<String, 
List<HoodieColumnRangeMetadata<Comparable>>> columnRangeMetadataPartitionPair,
-                                                                               
  HoodieTableMetaClient dataMetaClient,
-                                                                               
  Map<String, HoodieSchema> colsToIndexSchemaMap,
-                                                                               
  HoodieIndexVersion partitionStatsIndexVersion
-  ) {
-    try {
-      return columnRangeMetadataPartitionPair
-          .flatMapValues(List::iterator)
-          .groupByKey()
-          .map(pair -> {
-            final String partitionName = pair.getLeft();
-            return collectAndProcessColumnMetadata(pair.getRight(), 
partitionName, isShouldScanColStatsForTightBound(dataMetaClient), 
Option.empty(), colsToIndexSchemaMap, partitionStatsIndexVersion);
-          })
-          .flatMap(recordStream -> recordStream.iterator());
-    } catch (Exception e) {
-      throw new HoodieException("Failed to generate column stats records for 
metadata table", e);
-    }
-  }
-
-  public static boolean 
isShouldScanColStatsForTightBound(HoodieTableMetaClient dataMetaClient) {
-    return 
MetadataPartitionType.COLUMN_STATS.isMetadataPartitionAvailable(dataMetaClient);
+  /**
+   * Generates partition statistics only for columns with statistics covering 
every expected base and log file.
+   * Incomplete columns produce partition stats delete records to invalidate 
any previously stored bounds for those columns.
+   * The expected file names must come from the file inventory, independently 
of statistics extraction.
+   */
+  public static Stream<HoodieRecord> 
convertMetadataToPartitionStatsRecords(String partitionPath,
+                                                                           
List<HoodieColumnRangeMetadata<Comparable>> fileColumnMetadata,
+                                                                           
Set<String> expectedFileNames,
+                                                                           
Map<String, HoodieSchema> colsToIndexSchemaMap,
+                                                                           
HoodieIndexVersion partitionStatsIndexVersion) {
+    Map<String, List<HoodieColumnRangeMetadata<Comparable>>> columnMetadataMap 
= fileColumnMetadata.stream()
+        // Negative counts belong to missing/deleted statistics, whereas 
all-null and empty files have valid counts.
+        .filter(range -> range.getValueCount() >= 0 && range.getNullCount() >= 
0)
+        
.collect(Collectors.groupingBy(HoodieColumnRangeMetadata::getColumnName));
+    return colsToIndexSchemaMap.keySet().stream().flatMap(column -> {
+      List<HoodieColumnRangeMetadata<Comparable>> ranges = 
columnMetadataMap.getOrDefault(column, Collections.emptyList());
+      Set<String> coveredFiles = ranges.stream()
+          .map(range -> new StoragePath(range.getFilePath()).getName())
+          .collect(Collectors.toSet());
+      if (expectedFileNames.isEmpty() || 
!coveredFiles.containsAll(expectedFileNames)) {

Review Comment:
   Thanks for raising this. This PR invalidates partition stats for the new 
column when older files lack coverage. Queries remain correct, though pruning 
may be reduced.
   
   I added an `ADD COLUMNS` test for COW and MOR. Follow-up issue for 
optimizing partition stats after schema evolution: 
https://github.com/apache/hudi/issues/20535.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to