wjones127 commented on issue #35301: URL: https://github.com/apache/arrow/issues/35301#issuecomment-1540486938
Have you thought about how you want to handle these masks in conjunction with filter pushdown? I wonder if the best approach would be to expose a new hidden column `__row_index` in the scanner. Then you can pass down a filter (which can prune row groups or even data pages), and then the data returned with have the original row indices in that `__row_index` column. From there you could filter the results for where the `__filename` column matches the filename and `__row_index` matches doesn't match any of the deleted rows in the mask. What do you think of that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
