[
https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ulysses you updated SPARK-35282:
--------------------------------
Description:
Use AQE runtime statistics to decide if we can use shuffled hash join instead
of sort merge join. Currently, the formula of shuffled hash join selection does
not work due to the dymanic shuffle partition number.
Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to
decide if join can be converted to shuffled hash join safely.
was:
Use AQE runtime statistics to decide if we can use shuffled hash join instead
of sort merge join. Currently, the formula of shuffled hash join selection is
not worked due to the dymanic shuffle partition number.
Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to
decide if join can be converted to shuffled hash join safely.
> Support AQE side shuffled hash join formula
> -------------------------------------------
>
> Key: SPARK-35282
> URL: https://issues.apache.org/jira/browse/SPARK-35282
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: ulysses you
> Priority: Major
>
> Use AQE runtime statistics to decide if we can use shuffled hash join instead
> of sort merge join. Currently, the formula of shuffled hash join selection
> does not work due to the dymanic shuffle partition number.
>
> Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to
> decide if join can be converted to shuffled hash join safely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
