cshuo opened a new issue, #20441: URL: https://github.com/apache/hudi/issues/20441
Follow-up to https://github.com/apache/hudi/pull/20328 and https://github.com/apache/hudi/issues/20296. Before this fix, expanding `hoodie.metadata.index.column.stats.column.list` and subsequently writing new files into an existing partition could produce partition statistics covering only the files with statistics for the newly indexed column. These aggregates can have non-null counts and min/max values, making them indistinguishable from complete statistics to the read-time missing-statistics guard. For example, an older file may contain `name = 'alice'`, while the stored partition bounds cover only a newer file containing `name = 'bob'`. A query for `name = 'alice'` can then incorrectly prune the entire partition. The PR checks file coverage when generating partition statistics and invalidates incomplete aggregates. It does not detect or repair historical aggregates in partitions that are not subsequently processed. Upgrading alone therefore does not repair all affected tables. This follow-up should address detection and repair of historical incomplete partition statistics, including partitions untouched after upgrade, and document a supported rebuild procedure. Regression coverage should start with partial aggregates created before the fix and verify correct query results after repair, including cases where some file-level column statistics remain missing. Tables that expanded the indexed column list before the fix should rebuild the partition stats index using a version containing the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
