Hey Conrad, has it started happening recently?
We recently started having some sporadic problems with drivers on EMR when it gets stuck, up until two weeks ago everything was fine. We're trying to figure out with the EMR team where the issue is coming from. On Tue, Nov 27, 2018 at 6:29 AM Conrad Lee <con...@parsely.com> wrote: > > Dear spark community, > > I'm running spark 2.3.2 on EMR 5.19.0. I've got a job that's hanging in the > final stage--the job usually works, but I see this hanging behavior in about > one out of 50 runs. > > The second-to-last stage sorts the dataframe, and the final stage writes the > dataframe to HDFS. > > Here you can see the executor logs, which indicate that it has finished > processing the task. > > Here you can see the thread dump from the executor that's hanging. Here's > the text of the blocked thread. > > I tried to work around this problem by enabling speculation, but speculative > execution never takes place. I don't know why. > > Can anyone here help me? > > Thanks, > Conrad -- Sent from my iPhone --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
