jkylling opened a new issue, #7299:
URL: https://github.com/apache/arrow-rs/issues/7299

   **Is your feature request related to a problem or challenge? Please describe 
what you are trying to do.**
   <!--
   A clear and concise description of what the problem is. Ex. I'm always 
frustrated when [...] 
   (This section helps Arrow developers understand the context and *why* for 
this feature, in addition to  the *what*)
   -->
   Deletion vectors in the Delta Lake and Iceberg table formats are defined in 
terms of row numbers within individual Parquet files. To be able to filter out 
rows defined as deleted by deletion vectors we need a way to know the file row 
number of the rows read by the Arrow Parquet reader.
   
   **Describe the solution you'd like**
   <!--
   A clear and concise description of what you want to happen.
   -->
   The  Arrow Parquet reader should optionally return a column containing the 
row number of each row. We add a method 
`ArrowReaderBuilder::with_row_numbers(self, with_row_numbers: bool) -> Self`, 
which configures the Arrow Parquet reader to add an extra column named 
`row_number` to its schema (possibly the method could be 
`ArrowReaderBuilder::with_row_number_column(self, with_row_numbers: 
Option<String>) -> Self` to make the column name configurable). This column 
contains the row number within the file.
   
   **Describe alternatives you've considered**
   <!--
   A clear and concise description of any alternative solutions or features 
you've considered.
   -->
   There is a corresponding issue on Datafusion 
https://github.com/apache/datafusion/issues/13261. It considers an alternative 
using primary keys and existing SQL primitives, but this comes with a 
performance penalty. The consensus on the issue is
   
   > I agree with the assessment that the information must be coning from the 
file reader itself.
   
   That is, the Arrow Parquet reader.
   
   **Additional context**
   <!--
   Add any other context or screenshots about the feature request here.
   -->
   Please see https://github.com/apache/datafusion/issues/13261 for the 
corresponding issue in Datafusion. There is also a discussion in Datafusion to 
add system/metadata columns in https://github.com/apache/datafusion/pull/14057 
through which this additional file row number column could be exposed. Though, 
we do not need system/metadata columns to be available to support deletion 
vectors in delta-rs or iceberg-rs, since the delta-rs and iceberg-rs Datafusion 
based readers use the Datafusion ParquetSource directly to construct the 
execution plans for the scans of their TableProviders.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to