[jira] [Updated] (SPARK-35282) Support AQE side shuffled hash join formula
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-35282: -- Description: Use AQE runtime statistics to decide if we can use shuffled hash join instead of sort merge join. Currently, the formula of shuffled hash join selection does not work due to the dymanic shuffle partition number. Add a new config `spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold` to decide if join can be converted to shuffled hash join safely. was: Use AQE runtime statistics to decide if we can use shuffled hash join instead of sort merge join. Currently, the formula of shuffled hash join selection does not work due to the dymanic shuffle partition number. Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to decide if join can be converted to shuffled hash join safely. > Support AQE side shuffled hash join formula > --- > > Key: SPARK-35282 > URL: https://issues.apache.org/jira/browse/SPARK-35282 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.2.0 >Reporter: XiDuo You >Assignee: XiDuo You >Priority: Major > Fix For: 3.2.0 > > > Use AQE runtime statistics to decide if we can use shuffled hash join instead > of sort merge join. Currently, the formula of shuffled hash join selection > does not work due to the dymanic shuffle partition number. > > Add a new config `spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold` to > decide if join can be converted to shuffled hash join safely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-35282) Support AQE side shuffled hash join formula
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35282: Description: Use AQE runtime statistics to decide if we can use shuffled hash join instead of sort merge join. Currently, the formula of shuffled hash join selection does not work due to the dymanic shuffle partition number. Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to decide if join can be converted to shuffled hash join safely. was: Use AQE runtime statistics to decide if we can use shuffled hash join instead of sort merge join. Currently, the formula of shuffled hash join selection is not worked due to the dymanic shuffle partition number. Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to decide if join can be converted to shuffled hash join safely. > Support AQE side shuffled hash join formula > --- > > Key: SPARK-35282 > URL: https://issues.apache.org/jira/browse/SPARK-35282 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.2.0 >Reporter: ulysses you >Priority: Major > > Use AQE runtime statistics to decide if we can use shuffled hash join instead > of sort merge join. Currently, the formula of shuffled hash join selection > does not work due to the dymanic shuffle partition number. > > Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to > decide if join can be converted to shuffled hash join safely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-35282) Support AQE side shuffled hash join formula
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35282: Parent: SPARK-33828 Issue Type: Sub-task (was: New Feature) > Support AQE side shuffled hash join formula > --- > > Key: SPARK-35282 > URL: https://issues.apache.org/jira/browse/SPARK-35282 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.2.0 >Reporter: ulysses you >Priority: Major > > Use AQE runtime statistics to decide if we can use shuffled hash join instead > of sort merge join. Currently, the formula of shuffled hash join selection is > not worked due to the dymanic shuffle partition number. > > Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to > decide if join can be converted to shuffled hash join safely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
