[ https://issues.apache.org/jira/browse/ARROW-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072496#comment-17072496 ]
Joris Van den Bossche commented on ARROW-7965: ---------------------------------------------- This depends on ARROW-8164, but if that gets merged quickly, it would be nice tackle this issue for 0.17 since that would enable us to remove `factory()` from the high-level user API (which wasn't there yet in 0.16, so this would avoid it ever being in a released version) > [Python] Hold a reference to the dataset factory for later reuse > ---------------------------------------------------------------- > > Key: ARROW-7965 > URL: https://issues.apache.org/jira/browse/ARROW-7965 > Project: Apache Arrow > Issue Type: Improvement > Components: Python > Reporter: Krisztian Szucs > Priority: Major > Labels: pull-request-available > Fix For: 0.17.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Provide a more intuitive way to construct nested dataset: > ```python > # instead of using confusing factory function > dataset([ > factory("s3://old-taxi-data", format="parquet"), > factory("local/path/to/new/data", format="csv") > ]) > # let the user to construct a new dataset directly from dataset objects > dataset([ > dataset("s3://old-taxi-data", format="parquet"), > dataset("local/path/to/new/data", format="csv") > ]) > ``` > In the future we might want to introduce a new Dataset class which wraps > functionality of both the dataset actory and the materialized dataset > enabling optimizations over rediscovery of already materialized datasets. -- This message was sent by Atlassian Jira (v8.3.4#803005)