Davis-Zhang-Onehouse opened a new issue, #19719:
URL: https://github.com/apache/hudi/issues/19719
**Describe the problem**
`TestSecondaryIndexPruning` fails deterministically on current `master`
(`2460f8bed20a`) in a local run (macOS arm64, JDK 11, `-Dspark3.5
-Dscala-2.12`): every scenario that creates a second secondary index loses the
first index's records.
```
Expected Array([1$row1], [2$row2], [3$row3], [abc$row1], [cde$row2],
[def$row3]),
but got Array([1$row1], [2$row2], [3$row3])
```
8 of 8 parameterized cases error the same way
(`testSecondaryIndexWithFilters`, `testSecondaryIndexWithPartitionStatsIndex`,
`testUpdatesReInsertsDeletes`). The same signature appeared in a PR CI run's
`test-spark-java17-java-tests-part3 (scala-2.13, spark4.2)` job, while the
`master` push run on the same SHA was green, so the trigger is timing or
environment sensitive, not code-path availability.
**Mechanism (from the logs)**
During the second `create index`, the metadata writer's bootstrap commit
finds its target instant already completed on the metadata-table timeline and
rolls it back before recommitting:
```
Initializing secondary_index_idx_ts index with 10 file groups
Begin rollback of instant 20260823193207664 ...
Rolling back instant
[20260823193207664__20260823193208269__deltacommit__COMPLETED]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.commitInternal(HoodieBackedTableMetadataWriter.java:1402)
at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.bulkCommit(SparkHoodieBackedTableMetadataWriter.java:198)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.initializeMetadataPartition(HoodieBackedTableMetadataWriter.java:493)
```
The rolled-back completed deltacommit carries the earlier secondary index's
records, which is exactly the data that goes missing.
`generateUniqueInstantTime` returns an indexing instant unchanged, so when the
metadata table already holds a completed deltacommit at that instant,
`commitInternal`'s rollback-and-recommit destroys it.
**Reproduction**
```
mvn clean test -pl hudi-spark-datasource/hudi-spark -am -Dspark3.5
-Dscala-2.12 \
-Dtest='TestSecondaryIndexPruning#testSecondaryIndexWithPartitionStatsIndex'
```
on `master` `2460f8bed20a`. Reproduced on two independent checkouts (clean
build, incremental compilation disabled).
**Expected**
Creating a second secondary index must not roll back the metadata-table
commit that carries the first one; the bootstrap commit needs an instant that
cannot collide with a completed metadata-table deltacommit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
