Peter Toth created SPARK-60703:
----------------------------------
Summary: Make the broadcast value projection for DPP work under
AQE when the broadcast plan has no shuffle
Key: SPARK-60703
URL: https://issues.apache.org/jira/browse/SPARK-60703
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Peter Toth
SPARK-58265 lets an {{onlyInBroadcast}} DPP filter take its values from the
rows of an earlier broadcast
({{spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.enabled}},
off by default). Under AQE this does not happen when the plan under that
broadcast has no shuffle. That is the usual case, for example a filtered scan.
The DPP filter is planned as {{true}} instead, so the query gets no pruning, as
before SPARK-58265. Results are not affected.
Cause:
* {{PlanAdaptiveDynamicPruningFilters.projectedBroadcast}} plans the source
with {{QueryExecution.prepareExecutedPlan(projection.sourcePlan, context)}}. It
goes on only when it gets an {{AdaptiveSparkPlanExec}} back.
* {{InsertAdaptiveSparkPlan.apply}} passes {{isSubquery = false}}. So a plan
with no exchange, no distribution requirement and no subquery comes back
unwrapped, and {{projectedBroadcast}} returns {{None}}.
The tests miss it:
* {{EnableAdaptiveExecutionSuite}} sets {{spark.sql.adaptive.forceApply=true}},
so the {{AEOn}} suites always wrap the plan.
* The V2 projection tests in
{{DataSourceV2EnhancedRuntimePartitionFilterSuite}} run with AQE off.
With {{forceApply = false}} in {{EnableAdaptiveExecutionSuite}}, all 3
projection tests of {{DynamicPartitionPruningV1SuiteAEOn}} fail. The plan has
no {{ProjectedBroadcastValueSubqueryExec}}.
Found while working on SPARK-60702.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
