[ 
https://issues.apache.org/jira/browse/SPARK-60758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60758:
-----------------------------------
    Labels: pull-request-available  (was: )

> Require the same broadcast mode when a DPP filter reuses a broadcast without 
> AQE
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-60758
>                 URL: https://issues.apache.org/jira/browse/SPARK-60758
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Peter Toth
>            Priority: Major
>              Labels: pull-request-available
>
> Without AQE, {{PlanDynamicPruningFilters.reusableBroadcast}} keeps an 
> {{onlyInBroadcast}} DPP filter when a broadcast hash join broadcasts a plan 
> with the same result as the filter's build plan. For a direct reuse it 
> compares only the plans. It then builds a {{BroadcastExchangeExec}} keyed by 
> the filter's own build keys. When the join broadcasts that plan with another 
> hash mode, {{ReuseExchangeAndSubquery}} cannot share the two exchanges. So 
> the build plan runs and is broadcast twice. This goes against 
> {{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}}, which 
> applies DPP only where a broadcast can be reused. Results are not affected.
> The hash mode differs when:
> * the join broadcasts the plan on other keys;
> * the join is a null-aware anti join, which broadcasts with {{isNullAware = 
> true}}.
> Under AQE, {{PlanAdaptiveDynamicPruningFilters}} compares the broadcast 
> exchanges, mode included, and skips null-aware anti joins. There such a 
> filter is planned as {{true}}. Without AQE, the broadcast value projection of 
> SPARK-58265 also requires the same mode.
> A scratch run with AQE off, where the same filtered plan is the build plan of 
> a DPP filter in one union branch and is broadcast by a join in the other:
> * the join on the DPP filter's key: one broadcast exchange, which the DPP 
> filter reuses;
> * the join on another key: two broadcast exchanges, the DPP filter builds its 
> own;
> * a null-aware anti join: two broadcast exchanges.
> With AQE on, the second case is planned as {{true}}.
> A possible fix is to require the same mode for a direct reuse too, as the AQE 
> rule does. SPARK-60702 counts such a filter as kept, so its key gets no 
> runtime bloom filter. With this fix, it can count only a broadcast with the 
> same mode.
> Found while working on SPARK-60702.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to