Hi, Which version of spark you are using.
Less than 10Mb automatically converted as broadcast join in spark. \Thanks, selvam R On Sat, Nov 26, 2016 at 6:51 PM, Swapnil Shinde <swapnilushi...@gmail.com> wrote: > Hello > I am trying a broadcast join on dataframes but it is still doing > SortMergeJoin. I even try setting spark.sql.autoBroadcastJoinThreshold > higher but still no luck. > > Related piece of code- > val c = a.join(braodcast(b), "id") > > On a side note, if I do SizeEstimator.estimate(b) and it is really > high(460956584 bytes) compared to data it contains. b has just 85 rows and > around 4964 bytes. > Help is very much appreciated!! > > Thanks > Swapnil > > > -- Selvam Raman "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
