I see this happens when there is a deadlock situation. The RDD test1 has a Couchbase call and it seems to be having threads hanging there. Eventhough all the connections are closed I see the threads related to Couchbase causing the job to hang for sometime before it gets cleared up.
Would the driver not wait till all the stuff related to test1 is completed before calling test2 as test2 is dependent on test1? val test1 =RDD1.mapPartitions.() val test2 = test1.mapPartititions() On Sat, Dec 19, 2015 at 12:24 AM, Jeff Zhang <zjf...@gmail.com> wrote: > First you need to know where the hang happens (driver or executor), > checking log would be helpful > > On Sat, Dec 19, 2015 at 12:25 AM, SRK <swethakasire...@gmail.com> wrote: > >> Hi, >> >> My Spark Batch job seems to hung up sometimes for a long time before it >> starts the next stage/exits. Basically it happens when it has >> mapPartition/foreachPartition in a stage. Any idea as to why this is >> happening? >> >> Thanks, >> Swetha >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-batch-getting-hung-up-tp25735.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > > -- > Best Regards > > Jeff Zhang >
