[ https://issues.apache.org/jira/browse/ARROW-16616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17557566#comment-17557566 ]
Will Jones commented on ARROW-16616: ------------------------------------ {quote}In the R package we handle this by building a separate query object that contains the Dataset (or RecordBatchReader, or Table, or whatever), and the query object contains the filtering etc. methods. {quote} Yeah, I think it seems like the PR as-is would be trying to make {{Dataset}} into this kind of query object, which I don't think is it's purpose. {{Scanner}} seems closer to that to me, but it's also a bit of a mismatch. {quote}That said, going that route (or the FilteredDataset path, for that matter) feels like a path to essentially creating ibis in pyarrow. {quote} IMO we need to have a larger conversation about how much of Acero we want to expose in PyArrow. We might want a low-level API for creating exec plans, and maybe even some high-level functions like this. But seems like we should have a design doc to think about what our vision in here and how that does relate to developments in Ibis. > [Python] Allow lazy evaluation of filters in Dataset and add Datset.filter > method > --------------------------------------------------------------------------------- > > Key: ARROW-16616 > URL: https://issues.apache.org/jira/browse/ARROW-16616 > Project: Apache Arrow > Issue Type: Sub-task > Components: Python > Reporter: Alessandro Molina > Assignee: Alessandro Molina > Priority: Major > Labels: pull-request-available > Fix For: 9.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > To keep the {{Dataset}} api compatible with the {{Table}} one in terms of > analytics capabilities, we should add a {{Dataset.filter}} method. The > initial POC was based on {{_table_filter}} but that required materialising > all the {{Dataset}} content after filtering as it returned an > {{{}InMemoryDataset{}}}. > Given that {{Scanner}} can filter a dataset without actually materialising > the data until a final step happens, it would be good to have > {{Dataset.filter}} return some form of lazy dataset when the filter is only > stored aside and the Scanner is created when data is actually retrieved. > PS: Also update {{test_dataset_filter}} test to use the {{Dataset.filter}} > method -- This message was sent by Atlassian Jira (v8.20.7#820007)