Local mode: Stages hang for minutes

Hi, I'm doing some testing of workloads using local mode on a server. I see weird behavior where a job is submitted to the application and it just hangs for several minutes doing nothing. The stages are submitted as pending and in the application UI the stage view claims no tasks have been

Re: Local mode: Stages hang for minutes

I should add that the pauses are not from GC and also in tracing the CPU call tree in the JVM it seems like nothing is doing any work, just seems to be idling or blocking. On Thu, Dec 3, 2015 at 11:24 AM, Richard Marscher wrote: > Hi, > > I'm doing some testing of

Re: Local mode: Stages hang for minutes

Ended up realizing I was only looking at the call tree for running threads. After looking at blocking threads I saw that it was spending hundreds of compute hours blocking on jets3t calls to S3. Realized it was looking over likely thousands if not hundreds of thousands of S3 files accumulated over

Re: Local mode: Stages hang for minutes

You can try to run "jstack" a couple of times while the app is hung to look for patterns for where the app is hung. -- Ali On Dec 3, 2015, at 8:27 AM, Richard Marscher wrote: > I should add that the pauses are not from GC and also in tracing the CPU call > tree in