[ 
https://issues.apache.org/jira/browse/SPARK-60703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60703:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the broadcast value projection for DPP work under AQE when the broadcast 
> plan has no shuffle
> -------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60703
>                 URL: https://issues.apache.org/jira/browse/SPARK-60703
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Peter Toth
>            Priority: Major
>              Labels: pull-request-available
>
> SPARK-58265 lets an {{onlyInBroadcast}} DPP filter take its values from the 
> rows of an earlier broadcast 
> ({{spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.enabled}}, 
> off by default). Under AQE this does not happen when the plan under that 
> broadcast has no shuffle. That is the usual case, for example a filtered 
> scan. The DPP filter is planned as {{true}} instead, so the query gets no 
> pruning, as before SPARK-58265. Results are not affected.
> Cause:
> * {{PlanAdaptiveDynamicPruningFilters.projectedBroadcast}} plans the source 
> with {{QueryExecution.prepareExecutedPlan(projection.sourcePlan, context)}}. 
> It goes on only when it gets an {{AdaptiveSparkPlanExec}} back.
> * {{InsertAdaptiveSparkPlan.apply}} passes {{isSubquery = false}}. So a plan 
> with no exchange, no distribution requirement and no subquery comes back 
> unwrapped, and {{projectedBroadcast}} returns {{None}}.
> The tests miss it:
> * {{EnableAdaptiveExecutionSuite}} sets 
> {{spark.sql.adaptive.forceApply=true}}, so the {{AEOn}} suites always wrap 
> the plan.
> * The V2 projection tests in 
> {{DataSourceV2EnhancedRuntimePartitionFilterSuite}} run with AQE off.
> With {{forceApply = false}} in {{EnableAdaptiveExecutionSuite}}, all 3 
> projection tests of {{DynamicPartitionPruningV1SuiteAEOn}} fail. The plan has 
> no {{ProjectedBroadcastValueSubqueryExec}}.
> Found while working on SPARK-60702.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to