sunchao commented on code in PR #57437:
URL: https://github.com/apache/spark/pull/57437#discussion_r3693561275
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -434,6 +434,9 @@ object PushDownUtils extends Logging {
runtimeFilters: Seq[Expression],
partitionFields: Seq[PartitionPredicateField]):
Seq[PartitionPredicateImpl] = {
val catalystExprs = runtimeFilters.flatMap {
+ case DynamicPruningExpression(in: InSubqueryExec) if
in.isResultUnavailable =>
+ None
+ case DynamicPruningExpression(Literal.TrueLiteral) => None
Review Comment:
Fixed in `4db1868f`, thanks @peter-toth. I removed the
`DynamicPruningExpression(Literal.TrueLiteral) => None` special case,
preserving pre-existing iterative V2 behavior even when projected broadcast
pruning is disabled. The `isResultUnavailable` guard remains, so unavailable
projected domains still cannot reach connector filtering. The new `disabled
broadcast projection preserves existing no-op iterative V2 filtering`
regression checks the surviving true DPP filter, exactly one zero-reference
partition predicate, all five original partitions, and unchanged results.
I also updated the PR description to clarify that all four settings are
internal, the optimization is disabled by default, and there is no user-facing
configuration or API, addressing @viirya's earlier concern.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectedBroadcastValueSubqueryExec.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import java.util.concurrent.{Future => JFuture}
+
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference, Cast, Expression, NamedExpression, UnsafeRow}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.execution.BroadcastValueResult.{Available,
Unavailable}
+import org.apache.spark.sql.execution.joins.HashedRelation
+import org.apache.spark.sql.execution.metric.SQLMetrics
+import org.apache.spark.util.ThreadUtils
+
+/** Collects pruning values from the full rows of an already required hash
broadcast. */
+case class ProjectedBroadcastValueSubqueryExec(
Review Comment:
Fixed in `4db1868f`, thanks @peter-toth.
`SQLLastAttemptAccumulator.extractStageRDDScopes` now handles
`ProjectedBroadcastValueSubqueryExec` alongside `SubqueryBroadcastExec`,
returning `Nil` for both DPP-only subqueries instead of taking the
unsupported-subquery bailout. The existing projected-broadcast regression now
also asserts `lastAttemptValueForDataset(df) == Some(0L)` after confirming the
projected subquery is present; it runs against V1, V2, and runtime-filtering V2
scans with AQE both enabled and disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
