andygrove commented on code in PR #2837:
URL:
https://github.com/apache/datafusion-ballista/pull/2837#discussion_r4579331795
##########
ballista/scheduler/src/state/aqe/optimizer_rule/propagate_empty.rs:
##########
@@ -231,6 +234,54 @@ impl PropagateEmptyExecRule {
Ok(Transformed::no(plan))
}
}
+
+ /// Rewrites `plan` bottom-up with [`Self::transform`], except below a
+ /// parent that [`Self::transform`] keeps: there a child whose rewrite no
+ /// longer satisfies the parent's required input distribution stays as it
+ /// was, without the rewrites inside it.
+ ///
+ /// [`EmptyExec`] reports `UnknownPartitioning`, so it does not satisfy a
+ /// hash distribution that the exchange it replaced did. Under a parent
+ /// that requires one, such as a partitioned join or a window,
+ /// `EnsureRequirements` would add a hash repartition over the
+ /// [`EmptyExec`], and `DistributedExchangeRule` would turn that into a new
+ /// stage on every replan.
+ fn propagate(
+ plan: Arc<dyn ExecutionPlan>,
+ ) -> datafusion::error::Result<Arc<dyn ExecutionPlan>> {
+ let originals: Vec<_> = plan.children().into_iter().cloned().collect();
+ if originals.is_empty() {
+ return Ok(Self::transform(plan)?.data);
+ }
+ let rewritten = originals
+ .iter()
+ .cloned()
+ .map(Self::propagate)
+ .collect::<datafusion::error::Result<Vec<_>>>()?;
+ let result = Self::transform(replace_children_if_necessary(plan,
rewritten)?)?;
+ if result.transformed {
+ return Ok(result.data);
+ }
+ let plan = result.data;
+ let required = plan.input_distribution_requirements();
+ let satisfies = |index, child: &Arc<dyn ExecutionPlan>| {
+ let options =
ChildSatisfactionOptions::new().with_allow_subset(true);
+ required
+ .child_satisfaction(index, child.as_ref(), options)
+ .map(|satisfaction| satisfaction.is_satisfied())
+ };
+ let children = originals
+ .into_iter()
+ .zip(plan.children().into_iter().cloned())
+ .enumerate()
+ .map(|(index, (original, rewritten))| {
+ let lost = satisfies(index, &original)? && !satisfies(index,
&rewritten)?;
Review Comment:
Nice approach, and the doc comment says clearly that the rewrites inside a
put-back child are dropped. One case to think about for a follow-up: if the
child is an unresolved exchange over an inner join whose other side hasn't run
yet, putting it back means that other side still runs in full. Would it work to
keep the unresolved exchange but swap in its rewritten input, so the job pays
for one empty stage instead? Happy for this to be a separate PR.
##########
ballista/scheduler/src/state/aqe/optimizer_rule/propagate_empty.rs:
##########
@@ -231,6 +234,54 @@ impl PropagateEmptyExecRule {
Ok(Transformed::no(plan))
}
}
+
+ /// Rewrites `plan` bottom-up with [`Self::transform`], except below a
+ /// parent that [`Self::transform`] keeps: there a child whose rewrite no
+ /// longer satisfies the parent's required input distribution stays as it
+ /// was, without the rewrites inside it.
+ ///
+ /// [`EmptyExec`] reports `UnknownPartitioning`, so it does not satisfy a
+ /// hash distribution that the exchange it replaced did. Under a parent
+ /// that requires one, such as a partitioned join or a window,
+ /// `EnsureRequirements` would add a hash repartition over the
+ /// [`EmptyExec`], and `DistributedExchangeRule` would turn that into a new
+ /// stage on every replan.
+ fn propagate(
+ plan: Arc<dyn ExecutionPlan>,
+ ) -> datafusion::error::Result<Arc<dyn ExecutionPlan>> {
+ let originals: Vec<_> = plan.children().into_iter().cloned().collect();
+ if originals.is_empty() {
+ return Ok(Self::transform(plan)?.data);
+ }
+ let rewritten = originals
+ .iter()
+ .cloned()
+ .map(Self::propagate)
+ .collect::<datafusion::error::Result<Vec<_>>>()?;
+ let result = Self::transform(replace_children_if_necessary(plan,
rewritten)?)?;
+ if result.transformed {
+ return Ok(result.data);
+ }
+ let plan = result.data;
+ let required = plan.input_distribution_requirements();
+ let satisfies = |index, child: &Arc<dyn ExecutionPlan>| {
+ let options =
ChildSatisfactionOptions::new().with_allow_subset(true);
Review Comment:
Could we add a short comment on why `with_allow_subset(true)` is used here?
`EnsureRequirements` only allows subset satisfaction for some parents, and I
think `true` is the safe side because it catches every child the parent might
have accepted. Spelling that out would save the next reader some digging.
##########
ballista/scheduler/src/state/aqe/test/empty_input.rs:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Jobs in which one input finishes with no rows.
+//!
+//! Neither table declares a row count, so the planner learns that `empty` is
+//! empty only when its stage finishes.
+
+use crate::state::aqe::planner::AdaptivePlanner;
+use crate::state::aqe::test::stats_table::{StatsExec, StatsTable};
+use crate::state::aqe::test::{
+ mock_partitions_with_size, mock_partitions_with_statistics_no_data,
+};
+use ballista_core::assert_plan;
+use ballista_core::execution_plans::{ShuffleReaderExec, ShuffleWriter};
+use ballista_core::extension::SessionConfigExt;
+use datafusion::arrow::datatypes::{DataType, Field, Schema};
+use datafusion::common::Statistics;
+use datafusion::physical_plan::{ExecutionPlan, displayable};
+use datafusion::prelude::{SessionConfig, SessionContext};
+use rstest::rstest;
+use std::collections::HashSet;
+use std::sync::Arc;
+
+/// The partition count of `customers`, which tells its scans from those of
+/// `empty`.
+const CUSTOMERS_PARTITIONS: usize = 3;
+
+/// The most stages a job of these tests may run: one per table, one that a
+/// join plans over `empty` after the first `empty` stage finishes, and the
+/// final stage.
+const STAGE_RUNS_MAX: usize = 4;
+
+const LEFT_JOIN: &str = "SELECT c.id, e.eid AS other FROM customers c LEFT
JOIN empty e ON e.eemail = c.email";
+
+/// Each query reads the finished `empty` stage below a different operator that
+/// requires its hash partitioning: a join that `PropagateEmptyExecRule` does
+/// not remove because of its embedded projection, a mark join, and a window.
+#[rstest]
+#[case::left_join(LEFT_JOIN)]
+#[case::mark_join(
+ "SELECT c.id FROM customers c \
+ WHERE c.id > 5 OR EXISTS (SELECT 1 FROM empty e WHERE e.eemail = c.email)"
+)]
+#[case::window(
+ "SELECT eemail, row_number() OVER (PARTITION BY eemail ORDER BY eid) AS rn
FROM empty"
+)]
+#[tokio::test]
+async fn job_with_an_empty_input_finishes(
+ #[case] sql: &str,
+) -> datafusion::error::Result<()> {
+ assert_job_finishes(sql).await?;
+ Ok(())
+}
+
+/// The join reads the finished `empty` stage through a shuffle reader, which
+/// keeps the hash partitioning the join was planned against.
+#[tokio::test]
+async fn left_join_reads_the_empty_stage() -> datafusion::error::Result<()> {
Review Comment:
The tests read well and the `STAGE_RUNS_MAX` guard is a nice way to catch
the hang without a timeout. Could we add a case where the empty input sits one
join deeper, for example `customers LEFT JOIN (SELECT ... FROM customers c2
JOIN empty e ON ...) x ON x.cid = c.id`? That would cover putting back a whole
subtree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]