voonhous commented on issue #20631:
URL: https://github.com/apache/hudi/issues/20631#issuecomment-6822385204

   Finding from reading the write and read paths (master `db90a0da`); nothing 
was run.
   
   **Which writers emit usable record positions**
   - `hoodie.write.record.positions` defaults to true (since 1.0.0), but 
positions are only written for write version 8 and above, and only when the 
index tags each record with a base file row position.
   - Emit by default: Spark and the Java client, for upserts and key-based 
deletes, with SIMPLE, GLOBAL_SIMPLE, BLOOM or GLOBAL_BLOOM.
   - Never emit: Flink (explicitly disabled); the record-level, BUCKET, 
INMEMORY and HBase indexes; Spark SQL `UPDATE` and `DELETE` (prepped writes); 
`MERGE INTO` on tables without a primary key; log compaction; any block that 
also holds an insert; HFile blocks.
   - Table versions 6 and 7: not usable. The reader requires the 
`BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` header, which exists from version 
8.
   - Storage: versions 8 and 9 keep positions in the block header as a Base64 
Roaring bitmap; version 10 native log files keep them in the parquet footer 
key-value metadata as an ordered list. Neither is a column.
   
   **Read side**
   - A block is rejected when the header's base instant is missing or differs 
from the file slice's base file commit time (compaction, clustering). The first 
rejected block switches the whole file group reader to key-based merging; the 
fallback is one-way per reader, not per block.
   - Position-based merging still reads `_hoodie_record_key` from every base 
row: `FileGroupReaderSchemaHandler.getMandatoryFieldsForMerging` always adds 
it, and the projection is fixed before the reader knows whether it will fall 
back. So the exit criterion "the record-key column is no longer read" cannot be 
met without a hudi-common change.
   - The saving available to Trino today is the per-row string key extraction 
and hash lookup, replaced by a long lookup. The per-row Avro conversion stays.
   
   **What blocks it in the connector today**
   - `HudiUtil.convertToFileSlice` builds the base file with the split's commit 
time, which is the table's latest commit, not the base file's own instant. 
Every block would fail the instant check and fall back.
   - Without a real row index column the reader counts base rows, which is only 
valid while the base file is read in full and in order; base-file pruning on 
the merge path (#20633) would break that. Trino's parquet reader can expose a 
row index column (`$parquet$row_index`), which would need a record context that 
reports `supportsParquetRowIndex()`.
   - The local benchmark generator uses the INMEMORY index, so its logs carry 
no positions; switching it to SIMPLE produces them.
   
   Suggest lowering the priority: the feature applies to Spark tables written 
with default configs, but the expected gain is modest until the key column can 
be dropped from the projection.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to