yihua opened a new pull request, #20216: URL: https://github.com/apache/hudi/pull/20216
### Describe the issue this Pull Request addresses Follow-up to the discussion in https://github.com/apache/hudi/pull/19802#discussion_r4163300788. An index built by the async indexer (schedule, then execute) silently misses data commits that are requested after the indexing instant and complete before the indexing run starts. Such a commit does not update the new metadata partition, because the partition is not yet inflight in the table config. The index initialization then lists files only up to the indexing instant, and the catchup skips any instant that already has a commit in the metadata table. Indexes initialized from the file listing (column stats and bloom filters) therefore get no entries for the files written by those commits. Indexes initialized from file slices (record index, secondary index, expression index, partition stats) read the slices as of the run and are not affected. ### Summary and Changelog - `HoodieBackedTableMetadataWriter#initializeFromFilesystem` lists files up to the latest completed instant on the data timeline instead of the instant passed in. Regular initialization already passes the latest completed instant, so it is unchanged. For an indexing action, the listing now also covers the commits completed after the action was scheduled. The initialization still commits to the metadata table at the indexing instant. - `TestHoodieIndexer#testIndexerCoversCommitBetweenScheduleAndExecute` schedules an index, writes a commit, executes the index, and checks that the index covers both commits, for column stats, bloom filters, and record index. Without the fix the column stats and bloom filter cases fail with `expected: <6> but was: <3>`. The record index case passes either way and guards the file-slice bound of the initialization. ### Impact Column stats and bloom filter indexes built by the async indexer are complete for commits made between scheduling and execution. No public API or config change. ### Risk Level low. Only the file listing bound of an indexing action changes, and files of pending instants are still excluded. A narrower race remains for a writer whose commit overlaps the start of the indexing run (it can read the table config before the partition is marked inflight and complete after the initialization takes its timeline snapshot). That needs a handoff under the table lock and is not addressed here. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
