mbutrovich commented on code in PR #7662:
URL: https://github.com/apache/datafusion-comet/pull/7662#discussion_r4604943134


##########
spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala:
##########
@@ -7033,6 +7033,133 @@ class CometIcebergNativeSuite
     }
   }
 
+  // A storage-partitioned join is a sort-merge join that Spark plans without 
a shuffle, because
+  // both scans report a KeyGroupedPartitioning. On the build side of a DPP 
broadcast, AQE plans
+  // the converted join again, and EnsureRequirements has to find the join's 
required distribution
+  // already met rather than put an exchange under each side, which AQE has no 
logical link for
+  // (#7441).
+  test("AQE DPP - storage-partitioned join on the DPP build side (#7441)") {
+    assume(icebergAvailable, "Iceberg not available")
+    withTempIcebergDir { warehouseDir =>
+      withSQLConf(
+        "spark.sql.catalog.aqe_cat" -> "org.apache.iceberg.spark.SparkCatalog",
+        "spark.sql.catalog.aqe_cat.type" -> "hadoop",
+        "spark.sql.catalog.aqe_cat.warehouse" -> warehouseDir.getAbsolutePath,
+        SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true",
+        SQLConf.V2_BUCKETING_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
+        "spark.sql.iceberg.planning.preserve-data-grouping" -> "true",
+        CometConf.COMET_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_ENABLED.key -> "true",
+        CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+
+        spark.sql("""
+          CREATE TABLE aqe_cat.db.spj_dpp_fact (amount INT, store_id INT)
+          USING iceberg PARTITIONED BY (store_id)
+        """)
+        spark.sql("""
+          CREATE TABLE aqe_cat.db.spj_dpp_dim_a (id INT, country STRING)
+          USING iceberg PARTITIONED BY (id)
+        """)
+        spark.sql("""
+          CREATE TABLE aqe_cat.db.spj_dpp_dim_b (id INT, name STRING)
+          USING iceberg PARTITIONED BY (id)
+        """)
+        spark.sql(
+          "INSERT INTO aqe_cat.db.spj_dpp_fact SELECT CAST(id AS INT), CAST(id 
% 10 AS INT) " +
+            "FROM range(0, 100)")
+        spark.sql(
+          "INSERT INTO aqe_cat.db.spj_dpp_dim_a SELECT CAST(id AS INT), " +
+            "IF(id < 3, 'X', 'Y') FROM range(0, 10)")
+        spark.sql(
+          "INSERT INTO aqe_cat.db.spj_dpp_dim_b SELECT CAST(id AS INT), 
concat('n', id) " +
+            "FROM range(0, 10)")
+
+        // The merge join of the two dimensions is the build side of the 
broadcast that prunes the
+        // fact scan. It needs no shuffle because both of its scans are 
key-grouped on id.
+        val query =
+          """SELECT /*+ BROADCAST(d) */ f.store_id, f.amount, d.name
+            |FROM aqe_cat.db.spj_dpp_fact f JOIN (
+            |  SELECT /*+ MERGE(a, b) */ a.id, b.name

Review Comment:
   `CometHashJoinExec` gets the new trait too, but both tests use `MERGE`. 
Could we also run them with `SHUFFLE_HASH(a, b)` and assert that the plan has a 
`CometHashJoinExec`? I tried that variant of the join test locally on Spark 
4.1. It fails with the same `AssertionError` in 
`setLogicalLinkForNewQueryStage` when `operators.scala` is reverted to `main`, 
and passes with this PR.
   
   The two tests also share all of their setup, including the table names. When 
I ran them without the `operators.scala` change, the aggregate test failed with 
`TABLE_OR_VIEW_ALREADY_EXISTS` on `spj_dpp_fact`, because the join test failed 
before its `DROP TABLE` lines ran. So a failure in one test hides the result of 
the other. Could we use one setup with the drops in a `finally`, and generate 
the cases with a `for` over the two queries and the two hints, like the test at 
[L6170-L6174](https://github.com/apache/datafusion-comet/blob/48ee06f3fd28fa742fd53ec6575b8d748b10fb70/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala#L6170-L6174)?
 That would also give the aggregate query the no-exchange assertion that only 
the join query has now.



##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -741,17 +741,53 @@ abstract class CometExec extends CometPlan {
  * Declares the distribution of the Spark operator that a Comet operator 
replaces. Without it
  * AQE's OptimizeSkewInRebalancePartitions can split a skewed partition under 
an operator that
  * needs its input clustered, because ValidateRequirements has nothing to 
reject, and one key then
- * yields several rows. Mix it into operators that replace a clustering 
aggregate, window or join.
+ * yields several rows. Mix it into operators that replace a clustering 
aggregate or window. A
+ * join mixes in [[CometDelegatesJoinChildDistribution]] instead.
  */
 trait CometDelegatesRequiredChildDistribution extends CometExec {
   override def requiredChildDistribution: Seq[Distribution] =
     originalPlan.requiredChildDistribution
 }
 
+/**
+ * [[CometDelegatesRequiredChildDistribution]] for a join. A 
storage-partitioned join, whose sides
+ * are key-grouped scans that Spark joins with no exchange, declares no 
requirement. No shuffle
+ * sits below it for AQE to split. EnsureRequirements only accepts such a join 
when it is Spark's
+ * own join operator, and in Spark 4.1 only over Spark's own scans. So when 
AQE plans a DPP
+ * subquery's converted plan again, it would put an exchange under each side 
that it has no
+ * logical link for (#7441). The sides are the join's current children, which 
are Comet operators,
+ * because conversion works bottom-up and AQE can later swap in query stages.
+ */
+trait CometDelegatesJoinChildDistribution extends 
CometDelegatesRequiredChildDistribution {
+  override def requiredChildDistribution: Seq[Distribution] =
+    if (children.forall(CometExec.isStoragePartitioned)) {
+      children.map(_ => UnspecifiedDistribution)
+    } else {
+      super.requiredChildDistribution
+    }
+}
+
 object CometExec {
   final class InvalidSparkFallbackException(message: String)
       extends IllegalArgumentException(message)
 
+  /**
+   * Whether `plan` reads only key-grouped scans, with no shuffle above them. 
This is how Spark
+   * plans each side of a storage-partitioned join. A scan counts only when 
the scan it replaces
+   * or wraps was planned key-grouped, so a table read without grouping does 
not.
+   */
+  private[comet] def isStoragePartitioned(plan: SparkPlan): Boolean =
+    !plan.exists(_.isInstanceOf[ShuffleExchangeLike]) &&
+      plan.collectLeaves().forall {
+        // Spark's own scans. Comet's non-native V2 scan, CometBatchScanExec, 
is a leaf under a
+        // CometScanWrapper that this does not unwrap, so a join over it keeps 
its requirement.

Review Comment:
   This comment describes a path that no scan takes today. `CometBatchScanExec` 
is only created for CSV, Text, and Iceberg ([`CometScanRule.scala` 
L415](https://github.com/apache/datafusion-comet/blob/0fec45f232eac64aea65971c97956a64811af1dd/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L415),
 
[L574](https://github.com/apache/datafusion-comet/blob/0fec45f232eac64aea65971c97956a64811af1dd/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L574),
 
[L1345](https://github.com/apache/datafusion-comet/blob/0fec45f232eac64aea65971c97956a64811af1dd/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L1345)),
 and `CometExecRule` handles each of those in its own case before the 
`CometScanWrapper` one ([`CometExecRule.scala` 
L470-L483](https://github.com/apache/datafusion-comet/blob/0fec45f232eac64aea65971c97956a64811af1dd/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala#L470-L483)).
 An Iceberg scan that the native conversion declines sta
 ys a bare `CometBatchScanExec`. That class extends `DataSourceV2ScanExecBase`, 
so it matches the case below this comment and isn't excluded.
   
   Could we reword the comment to match? The paragraph in the PR description 
about Comet's non-native Iceberg reader has the same problem, so it could go 
too.
   
   ```suggestion
           // Spark's own scans, and CometBatchScanExec, which extends 
DataSourceV2ScanExecBase.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to