Right, you can use SparkContext and SQLContext in multiple threads. They are thread safe.
Best Regards, Shixiong Zhu 2015-10-01 4:57 GMT+08:00 <saif.a.ell...@wellsfargo.com>: > Hi all, > > I have a process where I do some calculations on each one of the columns > of a dataframe. > Intrinsecally, I run across each column with a for loop. On the other > hand, each process itself is non-entirely-distributable. > > To speed up the process, I would like to submit a spark program for each > column, any suggestions? I was thinking on primitive threads sharing a > spark context. > > Thank you, > Saif > >
