[jira] [Commented] (ARROW-11981) [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode
[ https://issues.apache.org/jira/browse/ARROW-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302912#comment-17302912 ] David Li commented on ARROW-11981: -- (I usually go by lidavidm since my name is rather common, but I wonder if I shouldn't just change my name sometimes…) To Weston's question - I think that's the plan, to have datasets as just another note in the pipeline, and to move projection/filtering into the pipeline and out of the datasets code. > [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode > > > Key: ARROW-11981 > URL: https://issues.apache.org/jira/browse/ARROW-11981 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 3.0.0 >Reporter: Ben Kietzman >Priority: Major > Labels: dataset > > UnionDataset allows Fragments of multiple schemas and differing file formats > to be scanned together as a single Dataset. This is useful functionality but > makes the Dataset interface somewhat difficult to reason about since it must > be general enough to accommodate UnionDataset. > After ARROW-11928 it will probably be more natural to support unioning of > datasets through a subclass of ExecNode. Reconciliation of differing schemas > can then be trivially handled by a full ProjectNode. > Note this would obviate both ARROW-11001 and ARROW-11749. In addition, > Dataset could be simplified to a concrete class containing a set of > compatibly typed/formatted Fragments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11981) [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode
[ https://issues.apache.org/jira/browse/ARROW-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302910#comment-17302910 ] Weston Pace commented on ARROW-11981: - [~lidavidm] > [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode > > > Key: ARROW-11981 > URL: https://issues.apache.org/jira/browse/ARROW-11981 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 3.0.0 >Reporter: Ben Kietzman >Priority: Major > Labels: dataset > > UnionDataset allows Fragments of multiple schemas and differing file formats > to be scanned together as a single Dataset. This is useful functionality but > makes the Dataset interface somewhat difficult to reason about since it must > be general enough to accommodate UnionDataset. > After ARROW-11928 it will probably be more natural to support unioning of > datasets through a subclass of ExecNode. Reconciliation of differing schemas > can then be trivially handled by a full ProjectNode. > Note this would obviate both ARROW-11001 and ARROW-11749. In addition, > Dataset could be simplified to a concrete class containing a set of > compatibly typed/formatted Fragments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11981) [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode
[ https://issues.apache.org/jira/browse/ARROW-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302908#comment-17302908 ] David Li commented on ARROW-11981: -- [~bkietz] Wrong David Li > [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode > > > Key: ARROW-11981 > URL: https://issues.apache.org/jira/browse/ARROW-11981 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 3.0.0 >Reporter: Ben Kietzman >Priority: Major > Labels: dataset > > UnionDataset allows Fragments of multiple schemas and differing file formats > to be scanned together as a single Dataset. This is useful functionality but > makes the Dataset interface somewhat difficult to reason about since it must > be general enough to accommodate UnionDataset. > After ARROW-11928 it will probably be more natural to support unioning of > datasets through a subclass of ExecNode. Reconciliation of differing schemas > can then be trivially handled by a full ProjectNode. > Note this would obviate both ARROW-11001 and ARROW-11749. In addition, > Dataset could be simplified to a concrete class containing a set of > compatibly typed/formatted Fragments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11981) [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode
[ https://issues.apache.org/jira/browse/ARROW-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302832#comment-17302832 ] Weston Pace commented on ARROW-11981: - It does raise the question of whether dataset scanning itself should be an exec node? > [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode > > > Key: ARROW-11981 > URL: https://issues.apache.org/jira/browse/ARROW-11981 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 3.0.0 >Reporter: Ben Kietzman >Priority: Major > Labels: dataset > > UnionDataset allows Fragments of multiple schemas and differing file formats > to be scanned together as a single Dataset. This is useful functionality but > makes the Dataset interface somewhat difficult to reason about since it must > be general enough to accommodate UnionDataset. > After ARROW-11928 it will probably be more natural to support unioning of > datasets through a subclass of ExecNode. Reconciliation of differing schemas > can then be trivially handled by a full ProjectNode. > Note this would obviate both ARROW-11001 and ARROW-11749. In addition, > Dataset could be simplified to a concrete class containing a set of > compatibly typed/formatted Fragments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11981) [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode
[ https://issues.apache.org/jira/browse/ARROW-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302515#comment-17302515 ] Ben Kietzman commented on ARROW-11981: -- [~davidli] [~westonpace] > [C++][Dataset][Compute] Replace UnionDataset with Union ExecNode > > > Key: ARROW-11981 > URL: https://issues.apache.org/jira/browse/ARROW-11981 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Affects Versions: 3.0.0 >Reporter: Ben Kietzman >Priority: Major > Labels: dataset > > UnionDataset allows Fragments of multiple schemas and differing file formats > to be scanned together as a single Dataset. This is useful functionality but > makes the Dataset interface somewhat difficult to reason about since it must > be general enough to accommodate UnionDataset. > After ARROW-11928 it will probably be more natural to support unioning of > datasets through a subclass of ExecNode. Reconciliation of differing schemas > can then be trivially handled by a full ProjectNode. > Note this would obviate both ARROW-11001 and ARROW-11749. In addition, > Dataset could be simplified to a concrete class containing a set of > compatibly typed/formatted Fragments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
