Hi Hao: I tried broadcastjoin with following steps, and found that my
query is still running slow ,not very sure if I'm doing right with
broadcastjoin:1.add spark.sql.autoBroadcastJoinThreshold 104857600(100MB)
in conf/spark-default.conf. 100MB is larger than any of my 2 tables.2.start
, May 5, 2015 10:16 AM
To: luohui20...@sina.com; Cheng, Hao; Olivier Girardot; user
Subject: RE: 回复：RE:
回复：Re: sparksql running slow while joining_2_tables.
You can use
Explain extended select ….
From:
luohui20...@sina.com [mailto:luohui20...@sina.com]
Sent: Tuesday, May 05, 2015 9:52 AM
To: Cheng
...@intel.com;
ssab...@gmail.com; user@spark.apache.org
Subject: 回复：回复：RE: 回复：Re: sparksql running slow while joining_2_tables.
update status after i did some tests. I modified some other parameters, found 2
parameters maybe relative.
spark_worker_instance and spark.sql.shuffle.partitions
before Today I
You can use
Explain extended select ….
From: luohui20...@sina.com [mailto:luohui20...@sina.com]
Sent: Tuesday, May 05, 2015 9:52 AM
To: Cheng, Hao; Olivier Girardot; user
Subject: 回复：RE: 回复：Re: sparksql running slow while joining_2_tables.
As I know broadcastjoin is automatically enabled
As I know broadcastjoin is automatically enabled by
spark.sql.autoBroadcastJoinThreshold.refer to
http://spark.apache.org/docs/latest/sql-programming-guide.html#other-configuration-options
and how to check my app's physical plan,and others things like optimized
plan,executable plan.etc
thanks
