NoahKusaba commented on code in PR #2852:
URL: 
https://github.com/apache/datafusion-ballista/pull/2852#discussion_r4595552870


##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -127,6 +129,12 @@ fn restrict(
         return Ok(rewritten);
     }
 
+    // Any other leaf the task runs none of, such as one in a `UnionExec`
+    // branch another task owns, is never executed.
+    if !under_collect && partitions.is_empty() && plan.children().is_empty() {

Review Comment:
   I went back to your suggestion as written (a819effa5). The planner-only 
resolver kept a node over a shuffle in the union's stage, which changed 7 
TPC-DS static plans for an optimization neither fix needs. The planner now asks 
without a resolver, so a node over a multi-partition shuffle keeps its own 
stage, and all 99 TPC-DS static plans match main's. A one-partition shuffle 
stays inline, like your one-partition custom leaf.
   



##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -141,7 +149,11 @@ fn restrict(
             .into_iter()
             .cloned()
             .zip(per_child)
-            .map(|(child, sub)| restrict(child, &sub, false, resolver))
+            .map(|(child, sub)| {
+                let restricted = restrict(child, &sub, false, resolver)?;
+                ensure_branch_restricted(&restricted, &sub)?;

Review Comment:
   Done: binding now runs such a stage as one task, as it does a collapse 
stage, instead of failing (35e9ea595). Planning and binding share one check, so 
a custom leaf without a resolver packs too. Queries succeed whether or not the 
stage is split, so #2854 is only about restoring parallelism and needn't block 
56.0.0.
   



##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -236,6 +260,35 @@ fn ensure_leaf_runs_whole(
     )
 }
 
+/// Check that a restricted `UnionExec` branch has one partition for each of
+/// `partitions`. The executor runs every partition of the task's plan, so a
+/// branch that keeps partitions the task wasn't given runs them in this task
+/// too, and the union's later partitions no longer line up with the task's.
+/// When those partitions read their input whole, as below a `SortExec`, a
+/// `LIMIT` or an aggregate without `GROUP BY` that merges the branch into one
+/// partition, the task returns rows that another task returns as well. A node
+/// the scheduler doesn't know that keeps its own count does the same.
+fn ensure_branch_restricted(

Review Comment:
   Superseded by :154: these stages no longer fail, so the error is just a 
backstop with one message (7c2711465). To keep the hint you asked for, binding 
now logs each stage it packs at `info`, naming the branch and pointing at the 
resolver for a node of your own (efb3b25f4).
   



##########
ballista/client/tests/split_union.rs:
##########
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod common;
+
+/// End-to-end tests of `UNION ALL` queries whose stages are split between
+/// tasks. Each task runs only the union branches it owns, so every other
+/// branch is restricted to the task's share of it, often none. They must
+/// return the rows DataFusion returns.
+#[cfg(test)]
+#[cfg(feature = "standalone")]
+mod split_union {
+    use crate::common::{
+        ballista_state, collect_lines, datafusion_context, 
orders_customers_dir,
+        register_alltypes, register_orders_customers,
+    };
+    use ballista::prelude::SessionContextExt;
+    use ballista_core::config::{
+        BALLISTA_ADAPTIVE_PLANNER_ENABLED, 
BALLISTA_SCHEDULER_MAX_PARTITIONS_PER_TASK,
+    };
+    use datafusion::prelude::SessionContext;
+    use rstest::rstest;
+    use std::path::Path;
+
+    /// The adaptive planner replaces a stage that gives no rows with an
+    /// `EmptyExec` of that stage's partition count, so in most of these a
+    /// union branch is an `EmptyExec` of several partitions.
+    const QUERIES: &[&str] = &[
+        // An aggregate that matches nothing, beside one that matches all.
+        "SELECT id % 2 AS k, count(*) AS c FROM test WHERE id > 100 GROUP BY 
id % 2 \
+         UNION ALL SELECT id % 2, count(*) FROM test GROUP BY id % 2",
+        // The same, the other way round.
+        "SELECT id % 2 AS k, count(*) AS c FROM test GROUP BY id % 2 \
+         UNION ALL SELECT id % 2, count(*) FROM test WHERE id > 100 GROUP BY 
id % 2",
+        // An empty middle branch of three.
+        "SELECT o_cust AS k, count(*) AS c FROM orders GROUP BY o_cust \
+         UNION ALL SELECT o_cust, count(*) FROM orders WHERE o_id < 0 GROUP BY 
o_cust \
+         UNION ALL SELECT c_id, 1 FROM customers WHERE c_id < 5",
+        // An aggregate over a union with an empty branch.
+        "SELECT k, sum(c) AS s FROM ( \
+           SELECT o_cust AS k, count(*) AS c FROM orders WHERE o_id < 0 GROUP 
BY o_cust \
+           UNION ALL SELECT o_cust, count(*) FROM orders GROUP BY o_cust) \
+         GROUP BY k",
+        // A join that matches nothing below an aggregate.
+        "SELECT c_region AS r, count(*) AS n FROM orders JOIN customers ON 
o_cust = c_id \
+           WHERE o_id < 0 GROUP BY c_region \
+         UNION ALL SELECT c_region, count(*) FROM customers GROUP BY c_region",
+        // A literal beside an empty aggregate.
+        "SELECT 7 AS k, 7 AS c \
+         UNION ALL SELECT id % 2, count(*) FROM test WHERE id > 100 GROUP BY 
id % 2",
+        // A union with an empty branch joined to a table.
+        "SELECT x.k, t.c_name FROM ( \
+           SELECT o_cust AS k FROM orders WHERE o_id < 0 GROUP BY o_cust \
+           UNION ALL SELECT c_id FROM customers WHERE c_id < 3) x \
+         JOIN customers t ON x.k = t.c_id",
+        // A window over a union with an empty branch.
+        "SELECT k, sum(c) OVER (PARTITION BY k) AS s FROM ( \
+           SELECT id % 2 AS k, count(*) AS c FROM test WHERE id > 100 GROUP BY 
id % 2 \
+           UNION ALL SELECT id % 2, count(*) FROM test GROUP BY id % 2)",
+        // Scans, literals and values.
+        "SELECT id FROM test UNION ALL SELECT id FROM test",
+        "SELECT 1 AS a UNION ALL SELECT 2 UNION ALL SELECT 3",
+        "SELECT 100 AS id UNION ALL SELECT id FROM test",
+        "SELECT column1 AS id FROM (VALUES (10), (11)) UNION ALL SELECT id 
FROM test",
+    ];
+
+    async fn register(ctx: &SessionContext, dir: &Path) {
+        register_alltypes(ctx, "test").await;
+        register_orders_customers(ctx, dir).await;
+    }
+
+    /// The formatted rows of `sql`, sorted, since a union's rows come in no
+    /// particular order.
+    async fn sorted_lines(ctx: &SessionContext, sql: &str) -> Vec<String> {
+        let mut lines = collect_lines(ctx, sql).await;
+        lines.sort();
+        lines
+    }
+
+    #[rstest]
+    #[tokio::test]
+    async fn split_unions_return_the_rows_datafusion_returns(

Review Comment:
   The error test went with the fallback. The matrix now has the merging 
queries instead, including your `ORDER BY id` repro, so it fails on main; with 
packing disabled, both adaptive cases fail.
   



##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -466,33 +519,24 @@ fn select_output_partitions(
         return Ok(None);
     }
 
-    // Self-generating leaves: PlaceholderRow (1 all-null row per partition)
-    // and EmptyExec (0 rows per partition). Every partition of one of these
-    // is semantically identical, so `with_partitions(indices.len())` is the
-    // algebraically correct restriction.
-    //
-    // Caveat: DataFusion's proto codec for these two doesn't roundtrip the
-    // `partitions` field — encode preserves only schema; decode always
-    // reconstructs with `::new(schema)` (partitions = 1). So a plan we
-    // restrict to `with_partitions(0)` here would arrive at the executor as
-    // a 1-partition leaf, undoing the restriction and reproducing the
-    // duplicated-output bug this whole function is meant to prevent.
-    //
-    // Workaround for the empty-slice case: swap in a 0-partition
-    // `MemorySourceConfig` (which DOES roundtrip its partition list).
-    // For non-empty slices we return the leaf as it is — currently safe 
because
-    // the only shape emitted is `partitions == 1` and UnionExec routing gives
-    // sub-slices of `[0]` (identity, no restriction needed) or `[]` (handled 
by
-    // this branch). Returned rather than `None` so the walker's leaf check
-    // accepts it: an `EmptyExec` under the round-robin `RepartitionExec` that
-    // `make_empty_exec_serde_safe` adds sees that repartition's partition
-    // numbers, not its own.
+    // Self-generating leaves: PlaceholderRow (1 row per partition) and
+    // EmptyExec (0 rows per partition). Every partition is the same, so a
+    // slice is the leaf with one partition for each index it has. As for the
+    // sources above, an index past its own count is dropped: under a
+    // round-robin `RepartitionExec` the leaf sees the repartition's numbers,
+    // and only the task that owns partition 0 may run a one-partition leaf.
     if plan.is::<PlaceholderRowExec>() || plan.is::<EmptyExec>() {

Review Comment:
   Added to the PR body.
   



##########
docs/source/user-guide/extending-components.md:
##########
@@ -179,7 +179,7 @@ let config = 
SchedulerConfig::default().with_partition_sliceable_resolver(Arc::n
 
 - `children` are the node's inputs, none for a scan, restricted to 
`partitions`. An input the node declares `SinglePartition` arrives whole, as a 
join's build side does. Partition `i` of the node must read partition `i` of 
every other input.
 - The restricted node must have exactly `partitions.len()` partitions. The 
scheduler rejects any other count.
-- `partitions` is empty for a `UnionExec` branch that belongs to other tasks. 
Return a node without partitions.
+- `partitions` is empty for a `UnionExec` branch that belongs to other tasks. 
Return a node without partitions. A leaf is never asked for none of its 
partitions: the scheduler replaces it with an empty leaf itself.

Review Comment:
   Now says the stage runs as one task, logged at `info`, rather than failing.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to