0xdeafbeef opened a new pull request, #2837: URL: https://github.com/apache/datafusion-ballista/pull/2837
# Which issue does this PR close? No issue filed; the bug is described below. # Rationale for this change With adaptive query planning on, this query never finishes when `empty` has no rows and the join is planned as partitioned: ```sql SELECT c.id, e.id AS other FROM customers c LEFT JOIN empty e ON e.email = c.email ``` Setting `ballista.planner.propagate_empty.enabled = false` avoids it. The job creates a new stage on every replan: 1. The stage that reads `empty` finishes with zero rows, and `PropagateEmptyExecRule` replaces its exchange with an `EmptyExec`. 2. The join above stays, because the rule keeps a join that has an embedded projection (the #2200 guard). 3. The join still requires hash-partitioned input, and `EmptyExec` reports `UnknownPartitioning`. So `EnsureRequirements` adds a hash `RepartitionExec` over the `EmptyExec`, and `DistributedExchangeRule` turns it into a new stage. 4. The new stage finishes with zero rows, and the next replan does the same with it. The same happens to other outer and anti joins with an embedded projection, to a mark join, and to a window function with `PARTITION BY`. The rule has no rewrite for a mark join or for the window's `SortExec`, so they stay too. # What changes are included in this PR? `PropagateEmptyExecRule` now leaves the finished exchange in place when an operator above it stays and requires its hash partitioning. That operator reads the empty stage through a shuffle reader, which reports the hash partitioning, so `EnsureRequirements` adds nothing and no new stage is created. The rule's rewrites are unchanged: it still removes an inner join with an empty side, and the #2200 guard still applies. A join whose empty side finishes while the join is still a `DynamicJoinSelectionExec` runs one more empty stage before the plan settles. # Are there any user-facing changes? Queries that hung on an empty input now finish. No API or configuration change. Maybe someone's cat will get cold :( https://xkcd.com/1172/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
