anoopj opened a new issue, #3014:
URL: https://github.com/apache/iceberg-rust/issues/3014
### Is your feature request related to a problem or challenge?
`CachingDeleteFileLoader::parse_positional_deletes_record_batch_stream`
currently allocates and hashes the map key on every row. A positional delete
file is sorted by (file_path, pos) and typically targets a single data file, so
the same (often ~60–80 char) path string repeats for every one of potentially
millions of rows. Each
iteration heap-allocates a throwaway String and hashes it purely to look
up a map entry that, after the first row, already exists. On merge-on-read
tables with large delete files this is pure overhead on
a hot path.
### Describe the solution you'd like
Within each Arrow batch, buffer the contiguous run of positions for one path
and merge it into the map with a single entry lookup, so the key is allocated
and hashed once per run instead of once per row. Positions are still inserted
one-by-one (no ordering precondition), and a recurring path merges into its
existing delete vector via entry().or_default(), so results are identical
regardless of ordering.
Within each Arrow batch, buffer the contiguous run of positions for one path
and merge it into the map with a single entry lookup.
## Java parity
This brings the Rust reader in line with the Java implementation, which
already avoids per-row key allocation.
## Scope
Applies to the Parquet positional-delete read path (the v2 representation).
v3 deletion vectors are a separate, Puffin-encoded bitmap path
### Willingness to contribute
I can contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
