hi,all : I'm using spark2.4, I try to use multi thread to use sparkcontext , I found a example : https://hadoopist.wordpress.com/2017/02/03/how-to-use-threads-in-spark-job-to-achieve-parallel-read-and-writes/ some code like this : for (a <- 0 until 4) { val thread = new Thread { override def run() { sc.parallelize(Array("ddd", "eee", "fff")).toDF().write.format("parquet").mode("overWrite").save("/tmp/vgiri/file2") ..... when I run the code in local or client model , it will be work. But In cluster model I meet the ERROR: SPARK-29046 https://issues.apache.org/jira/browse/SPARK-29046 finally , I use java.util.concurrent.Semaphore to wait all the sub thread to be finished before the main thread stop the sparkcontext , then the code worked in cluster model.
But I don't understand why In local or client model ,even I didn't use a Semaphore, the code can also work ?
