JoyMichael7842 opened a new issue, #19225:
URL: https://github.com/apache/hudi/issues/19225
### Bug Description
**What happened:**
Writing to a COPY_ON_WRITE table with the Record Level Index enabled results
in the record index having **fewer entries than the base data**, silently — no
error or warning. The dropped records:
- are **absent from the RLI** (record index count < distinct base record
keys),
- cause **silent read loss on RLI point-lookups** (a query by record key
returns nothing though the row exists in the base files).
The dropped set is **different on every write** (zero overlap across runs —
so it is not specific "bad" keys), and it correlates with record-key
**entropy**: sequential keys drop 0; high-entropy keys drop the most. The count
scales super-linearly with row count.
This reproduces with high-entropy record keys such as:
`concat('USL', substr(sha2(cast(id as string), 256), 1, 16))`
Observed shortfalls (same config/cluster, only the keys/size differ):
| record keys | N | shortfall |
|---|---|---|
| sequential (`SYN000…`) | 441M | 0 |
| high-entropy hex | 50M | 2,226 |
| high-entropy hex | 441M | 37,221 |
**What you expected:**
The record index should contain exactly one entry per distinct record key —
RLI count == distinct base data record keys — with no silent drops.
**Steps to reproduce:**
1. Generate N unique high-entropy string keys and write to a COW table with
RECORD_INDEX enabled:
```python
N = 50_000_000
synth = (spark.range(0, N)
.withColumn("id", expr("concat('USL', substr(sha2(cast(id as string),
256), 1, 16))"))
.withColumn("c", lit(1.0)).select("id", "c").dropDuplicates(["id"]))
opts = {
"hoodie.table.name": "rli_repro",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.precombine.field": "id",
"hoodie.datasource.write.keygenerator.class":
"org.apache.hudi.keygen.NonpartitionedKeyGenerator",
"hoodie.metadata.enable": "true",
"hoodie.metadata.record.index.enable": "true",
"hoodie.index.type": "RECORD_INDEX",
}
synth.write.format("hudi").options(**opts).mode("overwrite").save(path)
```
2. Compare base-data distinct keys vs record-index entry count:
```python
data = spark.read.format("hudi").option("hoodie.metadata.enable",
"false").load(path) \
.select("_hoodie_record_key").distinct().count()
rli = spark.read.format("hudi").load(path +
"/.hoodie/metadata").where("type=5").count()
print(data, rli, data - rli) # -> 50000000 49997774 2226
```
3. data - rli is nonzero (2,226 here) — those keys are missing from the RLI.
A point-lookup for a missing key (with data skipping enabled) returns 0 rows
even though the record exists in the base files.
Full runnable script and raw output are attached (`rli_repro.py.txt`,
`rli_repro_output.txt`).
### Environment
```markdown
**Hudi version:** 1.2.0 (also reproduced on 1.1.0)
**Query engine:** Spark 3.5
**Relevant configs:** COPY_ON_WRITE, hoodie.index.type=RECORD_INDEX,
hoodie.metadata.record.index.enable=true,
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator,
single writer, operation=upsert
### Logs and Stack Trace
No error or stack trace — the drop is silent. The only signal is the
record-index count being lower than the distinct base-data record-key count.
See attached rli_repro_output.txt (data=50000000, rli=49997774, shortfall=2226).
[rli_repro.py.txt](https://github.com/user-attachments/files/29812080/rli_repro.py.txt)
[rli_repro_output.txt](https://github.com/user-attachments/files/29812081/rli_repro_output.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
