[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235635#comment-15235635 ] JESSE CHEN commented on SPARK-13307: Performance back on track on Spark 2.0. Closing this. > TPCDS

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174501#comment-15174501 ] Xiao Li commented on SPARK-13307: - Really thank you for your response. > TPCDS query 66 degraded by 30%

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174473#comment-15174473 ] Davies Liu commented on SPARK-13307: In the plan, I saw that the column pruning does not work well,

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160260#comment-15160260 ] Xiao Li commented on SPARK-13307: - You need to check the plan and check the join type it is using. I

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160256#comment-15160256 ] Xiao Li commented on SPARK-13307: - First, I am not sure if usage of broadcastjoin makes sense in this

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159636#comment-15159636 ] JESSE CHEN commented on SPARK-13307: I tuned up the autoBroadcastJoinThreshold to 100MB, made no

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146765#comment-15146765 ] Xiao Li commented on SPARK-13307: - In the following PR: https://github.com/apache/spark/pull/9645,

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146740#comment-15146740 ] JESSE CHEN commented on SPARK-13307: Uploaded newly collected plans (logical, analyzed, optimized and

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146755#comment-15146755 ] Xiao Li commented on SPARK-13307: - Please tune “spark.sql.autoBroadcastJoinThreshold” to enable the

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146751#comment-15146751 ] Xiao Li commented on SPARK-13307: - 1.6.1 is using SortMergeJoin, but 1.4.1 is using ShuffleHashJoin. I

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146138#comment-15146138 ] JESSE CHEN commented on SPARK-13307: Have you looked at the plans? I provided at above links. >

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146140#comment-15146140 ] Xiao Li commented on SPARK-13307: - Could you provided logical plans, as suggested above? The attached

[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

[ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145829#comment-15145829 ] Xiao Li commented on SPARK-13307: - Please use explain(true). It will be much easier to analyze when