[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-01-53-423.png! !image-2023-05-16-21-16-37-376.png! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-05-45-361.png! !image-2023-05-16-21-16-13-128.png! and When shuffledHashJoin is enabled, gc is very serious. !image-2023-05-16-21-12-24-618.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-15-21-047.png! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. >From 8.1min(shuffledHashJoin) to 3.9min(sortMergeJoin). enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-01-53-423.png! !image-2023-05-16-21-16-37-376.png! disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-05-45-361.png! !image-2023-05-16-21-16-13-128.png! And When shuffledHashJoin is enabled, gc is very serious !image-2023-05-16-21-12-24-618.png! But sortMergeJoin executes without this problem !image-2023-05-16-21-15-21-047.png! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-20-18-727.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-21-35-493.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-22-44-532.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-22-16-170.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-23-33-611.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-23-35-237.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png! !image-2023-05-16-21-22-16-170.png! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=1340,height=92! !image-2023-05-16-21-21-35-493.png! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png! !image-2023-05-16-21-22-16-170.png! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png|width=990,height=68! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png! > !image-2023-05-16-21-22-16-170.png! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=1340,height=92! !image-2023-05-16-21-21-35-493.png! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png! !image-2023-05-16-21-22-16-170.png! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png! !image-2023-05-16-21-21-35-493.png! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png! !image-2023-05-16-21-22-16-170.png! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png|width=1340,height=92! > !image-2023-05-16-21-21-35-493.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png! > !image-2023-05-16-21-22-16-170.png! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-24-09-182.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-01-53-423.png! > !image-2023-05-16-21-16-37-376.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-05-45-361.png! > !image-2023-05-16-21-16-13-128.png! > > and When shuffledHashJoin is enabled, gc is very serious. > !image-2023-05-16-21-12-24-618.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-15-21-047.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png! !image-2023-05-16-21-21-35-493.png! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png! !image-2023-05-16-21-22-16-170.png! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-01-53-423.png! !image-2023-05-16-21-16-37-376.png! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-05-45-361.png! !image-2023-05-16-21-16-13-128.png! and When shuffledHashJoin is enabled, gc is very serious. !image-2023-05-16-21-12-24-618.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-15-21-047.png! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png! > !image-2023-05-16-21-21-35-493.png! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png! > !image-2023-05-16-21-22-16-170.png! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png|width=1190,height=78! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png! !image-2023-05-16-21-22-16-170.png! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png|width=990,height=68! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png|width=1190,height=78! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png|width=1114,height=73! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png|width=1190,height=78! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png|width=990,height=68! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png|width=1114,height=73! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png|width=1114,height=73! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png|width=1114,height=73! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png|width=990,height=68! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png|width=1114,height=73! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-28-44-163.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png, > image-2023-05-16-21-28-11-514.png, image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png|width=990,height=68! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png|width=1114,height=73! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-16-21-28-11-514.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png, > image-2023-05-16-21-28-11-514.png, image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-20-18-727.png|width=990,height=68! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-22-44-532.png|width=1114,height=73! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-28-44-163.png|width=935,height=64! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-28-11-514.png|width=922,height=67! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-20-18-727.png|width=990,height=68! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-22-44-532.png|width=1114,height=73! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-20-18-727.png, > image-2023-05-16-21-21-35-493.png, image-2023-05-16-21-22-16-170.png, > image-2023-05-16-21-22-44-532.png, image-2023-05-16-21-23-33-611.png, > image-2023-05-16-21-23-35-237.png, image-2023-05-16-21-24-09-182.png, > image-2023-05-16-21-28-11-514.png, image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: application_1684208757063_0028_90.html > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: application_1684208757063_0028_90.html, > image-2023-05-16-21-20-18-727.png, image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-22-44-532.png, > image-2023-05-16-21-23-33-611.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Description: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-28-44-163.png|width=935,height=64! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-28-11-514.png|width=922,height=67! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! was: Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when shuffle hash join is enabled and the performance is better when sortMergeJoin is used. Performance difference: from 3.9min(sortMergeJoin) to 8.1min(shuffledHashJoin) 1. enable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-28-44-163.png|width=935,height=64! !image-2023-05-16-21-21-35-493.png|width=924,height=502! 2. disable shuffledHashJoin, the execution plan is as follows: !image-2023-05-16-21-28-11-514.png|width=922,height=67! !image-2023-05-16-21-22-16-170.png|width=934,height=477! And when shuffledHashJoin is enabled, gc is very serious, !image-2023-05-16-21-23-35-237.png|width=929,height=570! but sortMergeJoin executes without this problem. !image-2023-05-16-21-24-09-182.png|width=931,height=573! Any suggestions on how to solve it？Thanks! > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: application_1684208757063_0028_90.html, > image-2023-05-16-21-20-18-727.png, image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-22-44-532.png, > image-2023-05-16-21-23-33-611.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: (was: application_1684208757063_0028_90.html) > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-22-44-532.png, > image-2023-05-16-21-23-33-611.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: (was: image-2023-05-16-21-20-18-727.png) > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-22-44-532.png, > image-2023-05-16-21-23-33-611.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: (was: image-2023-05-16-21-22-44-532.png) > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: (was: image-2023-05-16-21-23-33-611.png) > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-17-16-53-42-302.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png, image-2023-05-17-16-53-42-302.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-17-16-54-59-053.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png, image-2023-05-17-16-53-42-302.png, > image-2023-05-17-16-54-59-053.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: sort1.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png, image-2023-05-17-16-53-42-302.png, > image-2023-05-17-16-54-59-053.png, shuffle1.png, sort1.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: shuffle1.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png, image-2023-05-17-16-53-42-302.png, > image-2023-05-17-16-54-59-053.png, shuffle1.png, sort1.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: image-2023-05-19-10-43-51-747.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png, image-2023-05-17-16-53-42-302.png, > image-2023-05-17-16-54-59-053.png, image-2023-05-19-10-43-51-747.png, > shuffle1.png, sort1.png, sort2.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Attachment: sort2.png > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png, image-2023-05-17-16-53-42-302.png, > image-2023-05-17-16-54-59-053.png, image-2023-05-19-10-43-51-747.png, > shuffle1.png, sort1.png, sort2.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43526) when shuffle hash join is enabled, q95 performance deteriorates
[ https://issues.apache.org/jira/browse/SPARK-43526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caican updated SPARK-43526: --- Affects Version/s: 3.3.2 > when shuffle hash join is enabled, q95 performance deteriorates > --- > > Key: SPARK-43526 > URL: https://issues.apache.org/jira/browse/SPARK-43526 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.2, 3.2.0, 3.3.2 >Reporter: caican >Priority: Major > Attachments: image-2023-05-16-21-21-35-493.png, > image-2023-05-16-21-22-16-170.png, image-2023-05-16-21-23-35-237.png, > image-2023-05-16-21-24-09-182.png, image-2023-05-16-21-28-11-514.png, > image-2023-05-16-21-28-44-163.png, image-2023-05-17-16-53-42-302.png, > image-2023-05-17-16-54-59-053.png, image-2023-05-19-10-43-51-747.png, > shuffle1.png, sort1.png, sort2.png > > > Testing with 5TB dataset, the performance of q95 in tpcds deteriorates when > shuffle hash join is enabled and the performance is better when sortMergeJoin > is used. > > Performance difference: from 3.9min(sortMergeJoin) to > 8.1min(shuffledHashJoin) > > 1. enable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-44-163.png|width=935,height=64! > !image-2023-05-16-21-21-35-493.png|width=924,height=502! > 2. disable shuffledHashJoin, the execution plan is as follows: > !image-2023-05-16-21-28-11-514.png|width=922,height=67! > !image-2023-05-16-21-22-16-170.png|width=934,height=477! > > And when shuffledHashJoin is enabled, gc is very serious, > !image-2023-05-16-21-23-35-237.png|width=929,height=570! > > but sortMergeJoin executes without this problem. > !image-2023-05-16-21-24-09-182.png|width=931,height=573! > > Any suggestions on how to solve it？Thanks! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
