codope opened a new issue, #20557:
URL: https://github.com/apache/hudi/issues/20557

   ### Bug Description
   
   ### Bug Description
   
   On Spark, a writer that sets `hoodie.metadata.index.async=true` and enables 
the record index fails its first commit. This is the documented setup for 
building the record index with `HoodieIndexer` while ingestion keeps running. 
The writer is meant to leave building the index to `HoodieIndexer` and keep it 
up to date afterwards, and it must keep the index enabled, because a writer 
with it disabled drops the partition 
(`HoodieTable#deleteMetadataIndexIfNecessary`).
   
   The error:
   
   ```
   java.lang.IllegalStateException: Partition record_index should be part of 
inflight metadata partitions here []
   ```
   
   The commit fails before `HoodieIndexer` is ever scheduled, so the 
async-indexing workflow can't be used with Spark's default streaming metadata 
writes.
   
   **Cause.** The streaming and non-streaming metadata write paths disagree 
about which partitions to update. In `HoodieBackedTableMetadataWriter` on 
master (`7888bf90969`):
   
   - `initializeIfNeeded` (L276–283) skips initializing every partition except 
FILES when `isMetadataAsyncIndex()` is true, because `HoodieIndexer` 
initializes them. So `record_index` is neither completed nor inflight in 
`hoodie.properties`.
   - The non-streaming path, `getMetadataPartitionsToUpdate` (L800–814), uses 
the table config, completed plus inflight partitions, as the source of truth. 
Its comment calls out the async indexer case.
   - The streaming path, `getStreamingMetadataPartitionsToUpdate` (L931–936), 
adds `RECORD_INDEX` whenever it is in `enabledIndexerMap`, which reflects the 
write config. It doesn't check the table config. Secondary indexes in the same 
method *are* taken from the table config.
   - `tagRecordsWithLocationForStreamingWrites` (L1017) then finds no file 
slices for `record_index` and fails the `checkState` at L1034.
   
   **Expected.** The first commit succeeds and writes only the partitions that 
exist. After `HoodieIndexer` schedules (inflight) and completes `record_index`, 
the writer updates it. This is what happens with 
`hoodie.metadata.streaming.write.enabled=false`.
   
   **Suggested fix.** In `getStreamingMetadataPartitionsToUpdate`, add 
`RECORD_INDEX` only if `record_index` is in the table config's completed or 
inflight metadata partitions, the same rule `getMetadataPartitionsToUpdate` 
applies.
   
   **Workaround.** Set `hoodie.metadata.streaming.write.enabled=false` on the 
ingestion writer. With that, the full async flow works: HoodieStreamer in 
continuous mode, `HoodieIndexer --mode schedule` then `--mode execute`, and the 
writer keeps logging to `record_index` afterwards (verified on 1.2.1).
   
   **Reproduction.** spark-shell with the Spark 3.5 bundle; fresh table, one 
upsert:
   
   ```scala
   val path = "/tmp/rli_async_repro/tbl"
   val opts = Map(
     "hoodie.table.name" -> "rli_async",
     "hoodie.datasource.write.recordkey.field" -> "id",
     "hoodie.datasource.write.partitionpath.field" -> "dt",
     "hoodie.table.ordering.fields" -> "ts",
     "hoodie.datasource.write.operation" -> "upsert",
     "hoodie.metadata.enable" -> "true",
     "hoodie.metadata.index.async" -> "true",
     "hoodie.metadata.global.record.level.index.enable" -> "true")
   spark.range(0, 100).selectExpr("cast(id as string) as id", "'2026-01-01' as 
dt", "1L as ts")
     .write.format("hudi").options(opts).mode("append").save(path)
   // fails; adding "hoodie.metadata.streaming.write.enabled" -> "false" 
succeeds,
   // and hoodie.table.metadata.partitions=files
   ```
   
   HoodieStreamer hits the same failure on its first commit 
(`StreamSync.writeToSinkAndDoMetaSync` → `SparkRDDWriteClient.commit`).
   
   Related: #17208 hits the same assertion through a different path, enabling 
the metadata table on an existing table and then running a DML.
   
   
   ### Environment
   
   - Hudi: 1.2.1 (`hudi-spark3.5-bundle_2.12`) and master `7888bf90969` (built 
locally, Spark 3.5 profile)
   - Spark 3.5.6, Scala 2.12, Java 17, local mode, macOS
   - Table: COPY_ON_WRITE, default table version (9 on 1.2.1, 10 on master); 
fails on both
   - Relevant config: `hoodie.metadata.enable=true`, 
`hoodie.metadata.index.async=true`, 
`hoodie.metadata.global.record.level.index.enable=true`, 
`hoodie.metadata.streaming.write.enabled` left at the Spark default
   
   ### Logs and Stack Trace
   
   ```
   java.lang.IllegalStateException: Partition record_index should be part of 
inflight metadata partitions here []
        at 
org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:78)
        at 
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.lambda$tagRecordsWithLocationForStreamingWrites$37(HoodieBackedTableMetadataWriter.java:1558)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at 
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.tagRecordsWithLocationForStreamingWrites(HoodieBackedTableMetadataWriter.java:1552)
        at 
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.streamWriteToMetadataPartitions(HoodieBackedTableMetadataWriter.java:1420)
        at 
org.apache.hudi.client.SparkStreamingMetadataWriteHandler.streamWriteToMetadataTable(SparkStreamingMetadataWriteHandler.java:52)
        at 
org.apache.hudi.client.SparkStreamingMetadataWriteHandler.streamWriteToMetadataTable(SparkStreamingMetadataWriteHandler.java:44)
        at 
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:112)
        at 
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:94)
        at 
org.apache.hudi.utilities.streamer.StreamSync.writeToSinkAndDoMetaSync(StreamSync.java:992)
        at 
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:535)
        at 
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:844)
   ```
   
   (Stack trace from HoodieStreamer on 1.2.1; line numbers differ on master.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to