Leon-Weigel commented on issue #25244:
URL: https://github.com/apache/datafusion/issues/25244#issuecomment-5649096150
Follow-up with a negative result and a correction, both of which I think
make this report clearer rather than weaker.
**Negative result: plain SQL does not reproduce it.** I tried five shapes
through `ctx.sql` — a left-deep three-join chain keyed on a qualifier, a
derived table carrying a duplicate `id` with the top join on `j.id`, three
left-deep joins on `b.id`, and two parenthesized variants placing the chain on
either side. All five returned the same row count with the option on and off,
so all five are correct. The reason is that the SQL path attaches a
`projection=[id@1]` to the join and drops the duplicate `id` before the
downstream join, so the second same-named physical field the misroute needs
never survives. Derived tables also lose the qualifier needed to name the
second `id`. So this is not reachable through `ctx.sql`, and I would rather say
that here than have someone discover it.
**Correction, and this is the part that matters.** My original text said the
reproducer "bypasses the logical optimizer", which invited the reading that
this only happens to hand-built plans that your planner would never produce.
That reading is wrong, and the bypass in the minimal reproducer is a shrinking
convenience, not the trigger.
The originating case is a production query engine that builds plans through
the `DataFrame` API and then calls `SessionState::create_physical_plan` on the
result — the full pipeline, analyzer and logical optimizer included, nothing
skipped. The misroute happens there, with optimization fully enabled:
```
HashJoinExec: on=[(id@0, id@1)], projection=[id@1, id@3]
FilterExec: id@0 = x1
HashJoinExec: on=[(end_id@2, id@0)], projection=[id@0, id@1, id@3]
...
DataSourceExec: test.parquet, projection=[id], predicate=DynamicFilter
[ empty ], dynamic_rg_pruning=eligible
```
with the flag on returning no rows and the flag off returning the correct
one.
So the accurate scope is: **reachable through the `DataFrame` /
`LogicalPlanBuilder` API with the logical optimizer running, not reachable
through `ctx.sql`.** The precondition is two same-named columns surviving into
a downstream join key, which SQL's projection pushdown happens to eliminate and
the builder API does not. That is unremarkable for a graph engine, where nearly
every output column is named `id`, so duplicate names are the normal case
rather than a contrived one.
The underlying name-based remap in `FilterRemapper::try_remap` is unchanged
by any of this: a physical `Column` carries an index, and resolving it by name
against a schema with duplicate names discards that information regardless of
which front end built the plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
