codope opened a new issue, #20557:
URL: https://github.com/apache/hudi/issues/20557
### Bug Description
### Bug Description
On Spark, a writer that sets `hoodie.metadata.index.async=true` and enables
the record index fails its first commit. This is the documented setup for
building the record index with `HoodieIndexer` while ingestion keeps running.
The writer is meant to leave building the index to `HoodieIndexer` and keep it
up to date afterwards, and it must keep the index enabled, because a writer
with it disabled drops the partition
(`HoodieTable#deleteMetadataIndexIfNecessary`).
The error:
```
java.lang.IllegalStateException: Partition record_index should be part of
inflight metadata partitions here []
```
The commit fails before `HoodieIndexer` is ever scheduled, so the
async-indexing workflow can't be used with Spark's default streaming metadata
writes.
**Cause.** The streaming and non-streaming metadata write paths disagree
about which partitions to update. In `HoodieBackedTableMetadataWriter` on
master (`7888bf90969`):
- `initializeIfNeeded` (L276–283) skips initializing every partition except
FILES when `isMetadataAsyncIndex()` is true, because `HoodieIndexer`
initializes them. So `record_index` is neither completed nor inflight in
`hoodie.properties`.
- The non-streaming path, `getMetadataPartitionsToUpdate` (L800–814), uses
the table config, completed plus inflight partitions, as the source of truth.
Its comment calls out the async indexer case.
- The streaming path, `getStreamingMetadataPartitionsToUpdate` (L931–936),
adds `RECORD_INDEX` whenever it is in `enabledIndexerMap`, which reflects the
write config. It doesn't check the table config. Secondary indexes in the same
method *are* taken from the table config.
- `tagRecordsWithLocationForStreamingWrites` (L1017) then finds no file
slices for `record_index` and fails the `checkState` at L1034.
**Expected.** The first commit succeeds and writes only the partitions that
exist. After `HoodieIndexer` schedules (inflight) and completes `record_index`,
the writer updates it. This is what happens with
`hoodie.metadata.streaming.write.enabled=false`.
**Suggested fix.** In `getStreamingMetadataPartitionsToUpdate`, add
`RECORD_INDEX` only if `record_index` is in the table config's completed or
inflight metadata partitions, the same rule `getMetadataPartitionsToUpdate`
applies.
**Workaround.** Set `hoodie.metadata.streaming.write.enabled=false` on the
ingestion writer. With that, the full async flow works: HoodieStreamer in
continuous mode, `HoodieIndexer --mode schedule` then `--mode execute`, and the
writer keeps logging to `record_index` afterwards (verified on 1.2.1).
**Reproduction.** spark-shell with the Spark 3.5 bundle; fresh table, one
upsert:
```scala
val path = "/tmp/rli_async_repro/tbl"
val opts = Map(
"hoodie.table.name" -> "rli_async",
"hoodie.datasource.write.recordkey.field" -> "id",
"hoodie.datasource.write.partitionpath.field" -> "dt",
"hoodie.table.ordering.fields" -> "ts",
"hoodie.datasource.write.operation" -> "upsert",
"hoodie.metadata.enable" -> "true",
"hoodie.metadata.index.async" -> "true",
"hoodie.metadata.global.record.level.index.enable" -> "true")
spark.range(0, 100).selectExpr("cast(id as string) as id", "'2026-01-01' as
dt", "1L as ts")
.write.format("hudi").options(opts).mode("append").save(path)
// fails; adding "hoodie.metadata.streaming.write.enabled" -> "false"
succeeds,
// and hoodie.table.metadata.partitions=files
```
HoodieStreamer hits the same failure on its first commit
(`StreamSync.writeToSinkAndDoMetaSync` → `SparkRDDWriteClient.commit`).
Related: #17208 hits the same assertion through a different path, enabling
the metadata table on an existing table and then running a DML.
### Environment
- Hudi: 1.2.1 (`hudi-spark3.5-bundle_2.12`) and master `7888bf90969` (built
locally, Spark 3.5 profile)
- Spark 3.5.6, Scala 2.12, Java 17, local mode, macOS
- Table: COPY_ON_WRITE, default table version (9 on 1.2.1, 10 on master);
fails on both
- Relevant config: `hoodie.metadata.enable=true`,
`hoodie.metadata.index.async=true`,
`hoodie.metadata.global.record.level.index.enable=true`,
`hoodie.metadata.streaming.write.enabled` left at the Spark default
### Logs and Stack Trace
```
java.lang.IllegalStateException: Partition record_index should be part of
inflight metadata partitions here []
at
org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:78)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.lambda$tagRecordsWithLocationForStreamingWrites$37(HoodieBackedTableMetadataWriter.java:1558)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.tagRecordsWithLocationForStreamingWrites(HoodieBackedTableMetadataWriter.java:1552)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.streamWriteToMetadataPartitions(HoodieBackedTableMetadataWriter.java:1420)
at
org.apache.hudi.client.SparkStreamingMetadataWriteHandler.streamWriteToMetadataTable(SparkStreamingMetadataWriteHandler.java:52)
at
org.apache.hudi.client.SparkStreamingMetadataWriteHandler.streamWriteToMetadataTable(SparkStreamingMetadataWriteHandler.java:44)
at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:112)
at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:94)
at
org.apache.hudi.utilities.streamer.StreamSync.writeToSinkAndDoMetaSync(StreamSync.java:992)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:535)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:844)
```
(Stack trace from HoodieStreamer on 1.2.1; line numbers differ on master.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]