codope opened a new pull request, #20558: URL: https://github.com/apache/hudi/pull/20558
### Describe the issue this Pull Request addresses Closes #20557 A Spark writer with `hoodie.metadata.index.async=true` and the record index enabled fails its first commit with: ``` java.lang.IllegalStateException: Partition record_index should be part of inflight metadata partitions here [] ``` With async indexing on, `initializeIfNeeded` leaves the record index to `HoodieIndexer`, so `record_index` is neither completed nor inflight in `hoodie.properties`. The non-streaming write path takes the partitions to update from the table config and is fine. The streaming path (the Spark default) takes `record_index` from the write config and then fails when it finds no file groups for it. ### Summary and Changelog The first commit now succeeds and writes only the metadata partitions that exist. Once `HoodieIndexer` has scheduled and built `record_index`, the writer updates it, as it already does with `hoodie.metadata.streaming.write.enabled=false`. Both partition selectors used by a streaming commit in `HoodieBackedTableMetadataWriter` now apply the table-config rule from `getMetadataPartitionsToUpdate` (completed or inflight partitions): - `getStreamingMetadataPartitionsToUpdate`: stream to `record_index` only when the table config lists it. - `getNonStreamingMetadataPartitionsToUpdate`: send to the batch path only enabled partitions that the table config lists. The second change is needed too. With only the first, `record_index` falls through to the batch path in `completeStreamingCommit`, and `tagRecordsWithLocation` fails the same check. The second change also covers other enabled partitions that the async indexer hasn't built yet, such as column stats. Test: `TestMetadataWriterCommit#testStreamingWriteSkipsRecordIndexLeftToAsyncIndexer` runs a streaming commit with async indexing and the global record index enabled. It asserts that only `files` is bootstrapped and only `files` is written. It fails on master with the exception above. ### Impact Fixes the async record-index workflow (ingestion keeps running while `HoodieIndexer` builds the index) with streaming metadata writes, which Spark uses by default. When async indexing is off, every enabled partition is already in the table config after initialization, so nothing changes. Flink is out of scope. Flink streams to the metadata table only when `index.type` is a record-level index. The writer then needs the record index to tag records, and it has no fallback. So the async workflow can't work there. With this change, a Flink job configured that way fails on its first record-index lookup with "Record index is not initialized in MDT", instead of failing at the first commit. A fail-fast config check for Flink can follow separately. ### Risk Level low. Partitions are only ever dropped from the update set, never added. The ones dropped are not completed or inflight in the table config, and writing to them would have failed the same check. Ran `TestMetadataWriterCommit`, `TestCreateHandle`, `TestMergeHandle`, `TestAppendHandle` and `TestHoodieIndexer` locally (Spark 3.5). ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
