anoopj opened a new issue, #2879:
URL: https://github.com/apache/iceberg-rust/issues/2879
### Is your feature request related to a problem or challenge?
We already write v3 row lineage (row ranges get assigned on the snapshot and
manifest during commit), but there's no way to read the two row-lineage
metadata columns back out during a scan:
- `_row_id`
- `_last_updated_sequence_number`
Both are defined in `metadata_columns.rs` but nothing actually produces their
values, so selecting `_row_id` in a scan currently errors out in the arrow
read
pipeline — the field id passes projection validation and then there's no
constant/source for it.
For reference, Iceberg Java derives these on read rather than storing them
per row.
`_last_updated_sequence_number` maps to the file's **data** sequence number —
the one that's preserved when a file is carried forward across a rewrite, not
the file sequence number (which changes on every commit that re-adds the
file).
`_row_id` is `first_row_id +
position_in_file`, where the per-file `first_row_id` is inherited when the
manifest is read (manifest-level `first_row_id` plus a running record count).
We don't do any of that inheritance/derivation on the read side yet.
Note the write side already does the right thing here . v3 stores compact
ranges
and expects readers to derive the per-row values, so this is really just a
missing piece on the read path.
This is a building block for row-lineage consumers (CDC/MERGE-style engines
on
top of the scan API), not a changelog/CDC scan feature itself.
### Describe the solution you'd like
Happy to work on this. Part of the v3 row-lineage work (#2411) and
complements
the write side (#1652); would follow the same metadata-column materialization
pattern as #2746 (`_pos`) and #2695 (`_spec_id`).
### Willingness to contribute
I can contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
