Hey Tobias, Can you try using the YARN Fair Scheduler and set yarn.scheduler.fair.continuous-scheduling-enabled to true?
-Sandy On Sun, Dec 7, 2014 at 5:39 PM, Tobias Pfeiffer <t...@preferred.jp> wrote: > Hi, > > thanks for your responses! > > On Sat, Dec 6, 2014 at 4:22 AM, Sandy Ryza <sandy.r...@cloudera.com> > wrote: >> >> What version are you using? In some recent versions, we had a couple of >> large hardcoded sleeps on the Spark side. >> > > I am using Spark 1.1.1. > > As Andrew mentioned, I guess most of the 10 seconds waiting time probably > comes from YARN itself. (Other YARN applications also take a while to start > up.) I'm just really puzzled about what exactly takes so long there... for > a job that runs an hour or so, that is of course negligible, but I am > starting up an instance per client to do interactive job processing *for > this client*, and it feels like "yeah, thanks for logging in, now please > wait a while until you can actually use the program", that's a bit > suboptimal. > > Tobias > > >
