Could you post the result of explain `c.explain`? If it is broadcast join, you will see it in explain.
On Sat, Nov 26, 2016 at 10:51 AM, Swapnil Shinde <swapnilushi...@gmail.com> wrote: > Hello > I am trying a broadcast join on dataframes but it is still doing > SortMergeJoin. I even try setting spark.sql.autoBroadcastJoinThreshold > higher but still no luck. > > Related piece of code- > val c = a.join(braodcast(b), "id") > > On a side note, if I do SizeEstimator.estimate(b) and it is really > high(460956584 bytes) compared to data it contains. b has just 85 rows and > around 4964 bytes. > Help is very much appreciated!! > > Thanks > Swapnil > > >
