andygrove commented on issue #2046:
URL:
https://github.com/apache/datafusion-ballista/issues/2046#issuecomment-4994192951
Update: this issue turned out to bundle two unrelated root causes, one per
pair of queries.
**q38 / q87 (set operations)** — `datafusion-proto`'s logical `JoinNode`
decode rebuilt the join via `LogicalPlanBuilder`, whose `join_with_expr_keys` /
`join_using` hardcode `NullEqualsNothing`, so the encoded `null_equality` was
dropped. Ballista ships the client's logical plan to the scheduler, so
`INTERSECT` / `EXCEPT` (which carry `NullEqualsNull`) lost NULL-matching: NULL
stopped matching itself, so INTERSECT dropped NULL rows and EXCEPT retained
them. Fixed upstream by apache/datafusion#22104, backported to `branch-54` as
apache/datafusion#22785 — not in the released 54.0.0 tag. PR #2064 (draft until
a 54.x patch release is out).
**q4 / q78 (joins)** — a separate bug, #2065: broadcast promotion rewrites a
`SortMergeJoinExec` into a `HashJoinExec(CollectLeft)`, which does not preserve
input order, so a parent `SortMergeJoinExec` merges unsorted input and silently
drops rows. Fixed in #2066.
Note the q78 description above ("distributed LIMIT drops rows") is a red
herring — the wrong answer does not depend on `ORDER BY` or `LIMIT` and
reproduces with neither. Wrapping the query in `count(*)` changes the plan
enough to hide the bug, which is what made it look limit-related.
With both PRs the TPC-DS correctness gate goes from 86 to 90 queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
