marymwu created SPARK-16093: ------------------------------- Summary: Spark2.0 take no effect after set spark.sql.autoBroadcastJoinThreshold = 1 Key: SPARK-16093 URL: https://issues.apache.org/jira/browse/SPARK-16093 Project: Spark Issue Type: Bug Affects Versions: 2.0.0 Reporter: marymwu
Spark2.0 take no effect after set spark.sql.autoBroadcastJoinThreshold = 1 Precondition: set spark.sql.autoBroadcastJoinThreshold = 1; Testcase: "INSERT OVERWRITE TABLE RPS__H_REPORT_MORE_DIMENSION_FIRST_CHANNEL_VISIT_CD_DAY PARTITION (p_event_date='2016-06-18') select a.app_key,a.app_channel,b.device_model,sum(b.visits) visitsNum from (select app_key,app_channel,lps_did from RPS__H_REPORT_MORE_DIMENSION_EARLIEST_NEWUSER_LIST_C ) a join (select app_key,lps_did,device_model, count(1) as visits from RPS__H_REPORT_MORE_DIMENSION_SMALL where p_event_date = '2016-06-18' and ( log_type=1 or log_type=2) group by app_key,lps_did,device_model) b on a.lps_did = b.lps_did and a.app_key=b.app_key group by a.app_key,a.app_channel,b.device_model; " == Physical Plan == InsertIntoHiveTable MetastoreRelation default, rps__h_report_more_dimension_first_channel_visit_cd_day, None, Map(p_event_date -> Some(2016-06-18)), true, false +- TungstenAggregate(key=[app_key#7,app_channel#9,device_model#20], functions=[(sum(visits#3L),mode=Final,isDistinct=false)], output=[app_key#7,app_channel#9,device_model#20,visitsNum#4L]) +- Exchange(coordinator id: 41547585) hashpartitioning(app_key#7, app_channel#9, device_model#20, 600), Some(coordinator[target post-shuffle partition size: 500000000]) +- TungstenAggregate(key=[app_key#7,app_channel#9,device_model#20], functions=[(sum(visits#3L),mode=Partial,isDistinct=false)], output=[app_key#7,app_channel#9,device_model#20,sum#41L]) +- Project [app_key#7,app_channel#9,device_model#20,visits#3L] +- BroadcastHashJoin [lps_did#8,app_key#7], [lps_did#13,app_key#12], Inner, BuildRight, None :- Filter (isnotnull(app_key#7) && isnotnull(lps_did#8)) : +- HiveTableScan [app_key#7,app_channel#9,lps_did#8], MetastoreRelation default, rps__h_report_more_dimension_earliest_newuser_list_c, None +- BroadcastExchange HashedRelationBroadcastMode(List(input[1, string], input[0, string])) +- TungstenAggregate(key=[app_key#12,lps_did#13,device_model#20], functions=[(count(1),mode=Final,isDistinct=false)], output=[app_key#12,lps_did#13,device_model#20,visits#3L]) +- Exchange(coordinator id: 733045095) hashpartitioning(app_key#12, lps_did#13, device_model#20, 600), Some(coordinator[target post-shuffle partition size: 500000000]) +- TungstenAggregate(key=[app_key#12,lps_did#13,device_model#20], functions=[(count(1),mode=Partial,isDistinct=false)], output=[app_key#12,lps_did#13,device_model#20,count#39L]) +- Project [app_key#12,lps_did#13,device_model#20] +- Filter ((isnotnull(app_key#12) && isnotnull(lps_did#13)) && ((cast(log_type#11 as int) = 1) || (cast(log_type#11 as int) = 2))) +- HiveTableScan [app_key#12,lps_did#13,device_model#20,log_type#11], MetastoreRelation default, rps__h_report_more_dimension_small, None, [isnotnull(p_event_date#10),(p_event_date#10 = 2016-06-18)] Time taken: 4.775 seconds, Fetched 1 row(s) 16/06/20 16:55:16 INFO CliDriver: Time taken: 4.775 seconds, Fetched 1 row(s) Note: +- BroadcastHashJoin [lps_did#8,app_key#7], [lps_did#13,app_key#12], Inner, BuildRight, None Result: 1. Execution failed, spark service is unavailable. 2. Even though set spark.sql.autoBroadcastJoinThreshold = 1, BroadcastHashJoin has been used when join two large tables. Error log is as attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org