voonhous commented on issue #20631: URL: https://github.com/apache/hudi/issues/20631#issuecomment-6822385204
Finding from reading the write and read paths (master `db90a0da`); nothing was run. **Which writers emit usable record positions** - `hoodie.write.record.positions` defaults to true (since 1.0.0), but positions are only written for write version 8 and above, and only when the index tags each record with a base file row position. - Emit by default: Spark and the Java client, for upserts and key-based deletes, with SIMPLE, GLOBAL_SIMPLE, BLOOM or GLOBAL_BLOOM. - Never emit: Flink (explicitly disabled); the record-level, BUCKET, INMEMORY and HBase indexes; Spark SQL `UPDATE` and `DELETE` (prepped writes); `MERGE INTO` on tables without a primary key; log compaction; any block that also holds an insert; HFile blocks. - Table versions 6 and 7: not usable. The reader requires the `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` header, which exists from version 8. - Storage: versions 8 and 9 keep positions in the block header as a Base64 Roaring bitmap; version 10 native log files keep them in the parquet footer key-value metadata as an ordered list. Neither is a column. **Read side** - A block is rejected when the header's base instant is missing or differs from the file slice's base file commit time (compaction, clustering). The first rejected block switches the whole file group reader to key-based merging; the fallback is one-way per reader, not per block. - Position-based merging still reads `_hoodie_record_key` from every base row: `FileGroupReaderSchemaHandler.getMandatoryFieldsForMerging` always adds it, and the projection is fixed before the reader knows whether it will fall back. So the exit criterion "the record-key column is no longer read" cannot be met without a hudi-common change. - The saving available to Trino today is the per-row string key extraction and hash lookup, replaced by a long lookup. The per-row Avro conversion stays. **What blocks it in the connector today** - `HudiUtil.convertToFileSlice` builds the base file with the split's commit time, which is the table's latest commit, not the base file's own instant. Every block would fail the instant check and fall back. - Without a real row index column the reader counts base rows, which is only valid while the base file is read in full and in order; base-file pruning on the merge path (#20633) would break that. Trino's parquet reader can expose a row index column (`$parquet$row_index`), which would need a record context that reports `supportsParquetRowIndex()`. - The local benchmark generator uses the INMEMORY index, so its logs carry no positions; switching it to SIMPLE produces them. Suggest lowering the priority: the feature applies to Spark tables written with default configs, but the expected gain is modest until the key column can be dropped from the projection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
