Hello I am trying a broadcast join on dataframes but it is still doing SortMergeJoin. I even try setting spark.sql.autoBroadcastJoinThreshold higher but still no luck.
Related piece of code- val c = a.join(braodcast(b), "id") On a side note, if I do SizeEstimator.estimate(b) and it is really high(460956584 bytes) compared to data it contains. b has just 85 rows and around 4964 bytes. Help is very much appreciated!! Thanks Swapnil