>>> Would the driver not wait till all the stuff related to test1 is completed before calling test2 as test2 is dependent on test1? >>> val test1 =RDD1.mapPartitions.() >>> val test2 = test1.mapPartititions()
On the driver side, actually these 2 lines of code will be executed but the real computation on the executor won't get executed until you call action on the rdd (I suppose you call that after these code) You might need to check on the executor side why the hang issue happens. On Mon, Dec 21, 2015 at 11:04 AM, swetha kasireddy < swethakasire...@gmail.com> wrote: > I see this happens when there is a deadlock situation. The RDD test1 has a > Couchbase call and it seems to be having threads hanging there. Eventhough > all the connections are closed I see the threads related to Couchbase > causing the job to hang for sometime before it gets cleared up. > > Would the driver not wait till all the stuff related to test1 is completed > before calling test2 as test2 is dependent on test1? > > val test1 =RDD1.mapPartitions.() > > val test2 = test1.mapPartititions() > > On Sat, Dec 19, 2015 at 12:24 AM, Jeff Zhang <zjf...@gmail.com> wrote: > >> First you need to know where the hang happens (driver or executor), >> checking log would be helpful >> >> On Sat, Dec 19, 2015 at 12:25 AM, SRK <swethakasire...@gmail.com> wrote: >> >>> Hi, >>> >>> My Spark Batch job seems to hung up sometimes for a long time before it >>> starts the next stage/exits. Basically it happens when it has >>> mapPartition/foreachPartition in a stage. Any idea as to why this is >>> happening? >>> >>> Thanks, >>> Swetha >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-batch-getting-hung-up-tp25735.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > -- Best Regards Jeff Zhang
