alamb opened a new pull request #619: URL: https://github.com/apache/arrow-datafusion/pull/619
# Which issue does this PR close? Closes https://github.com/apache/arrow-datafusion/issues/617 but I am still not sure if this is a bug or not (explained below) # Rationale for this change As explained on https://github.com/apache/arrow-datafusion/issues/617 the projection pushdown operation removes columns from a scan when the `LogicalPlan::Projection` had unqualified columns (so an expression like `a`, rather than `table.a`). In my case in IOx this was occuring due to an extension node, when my code was creating the exprs without qualifiers. When I updated to the code to create the expression with qualifiers things are fine again ```diff let exprs = input .schema() .fields() .iter() - .map(|field| logical_plan::col(field.name())) + .map(|field| Expr::Column(field.qualified_column())) .collect::<Vec<_>>(); ``` Thus I am not sure if the projection pushdown code should actually handle this case, or if we should have some sort of error / warning instead. Having it silently produce the wrong answer (which is what happened in IOx) was hard to debug # What changes are included in this PR? 1. Only check relational qualifier if there a relational qualifier on the expression, otherwise just check field name 2. Ensure no duplicate field names by using BTreeSet # Are there any user-facing changes? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
