[
https://issues.apache.org/jira/browse/ARROW-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17157378#comment-17157378
]
Andy Grove commented on ARROW-8205:
-----------------------------------
I think I am fine with removing Expr::Column(usize) from the logical plan and
removing the complexity there. I hope I'm not forgetting any important reason
for still having it there.
I also think that projections in the physical plan should generally be based on
name so that we can better handle cases where parquet partitions don't all have
the same schema, and also to handle schemaless use cases better e.g. querying
JSON.
However, the output from the parquet scan should generally have a known schema
(based on the projection) so it would be nice for all the upstream operators to
be able to operate on indices rather than names for good performance. I would
be interested to see any impact on performance from your changes, perhaps by
running the benchmark crate in the project.
> [Rust] [DataFusion] DataFusion should enforce unique field names in a schema
> ----------------------------------------------------------------------------
>
> Key: ARROW-8205
> URL: https://issues.apache.org/jira/browse/ARROW-8205
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust, Rust - DataFusion
> Affects Versions: 0.16.0
> Reporter: Andy Grove
> Priority: Major
> Fix For: 2.0.0
>
>
> There does not seem to be any validation to avoid schemas being created with
> duplicate field names. We should add this along with unit tests.
> This will require changing the signature of the constructors to try_new with
> a Result return type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
