You would have needed to configure it by
setting yarn.scheduler.capacity.resource-calculator to something ending in
DominantResourceCalculator. If you haven't configured it, there's a high
probability that the recently committed
https://issues.apache.org/jira/browse/SPARK-6050 will fix your
We're using the capacity scheduler, to the best of my knowledge. Unsure if
multi resource scheduling is used, but if you know of an easy way to figure
that out, then let me know.
Thanks,
Anders
On Sat, Feb 21, 2015 at 12:05 AM, Sandy Ryza sandy.r...@cloudera.com
wrote:
Are you using the
Are you using the capacity scheduler or fifo scheduler without multi
resource scheduling by any chance?
On Thu, Feb 12, 2015 at 1:51 PM, Anders Arpteg arp...@spotify.com wrote:
The nm logs only seems to contain similar to the following. Nothing else
in the same time range. Any help?
The nm logs only seems to contain similar to the following. Nothing else in
the same time range. Any help?
2015-02-12 20:47:31,245 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: KILL_CONTAINER sent to absent container
It seems unlikely to me that it would be a 2.2 issue, though not entirely
impossible. Are you able to find any of the container logs? Is the
NodeManager launching containers and reporting some exit code?
-Sandy
On Thu, Feb 12, 2015 at 1:21 PM, Anders Arpteg arp...@spotify.com wrote:
No, not
Interesting to hear that it works for you. Are you using Yarn 2.2 as well?
No strange log message during startup, and can't see any other log messages
since no executer gets launched. Does not seems to work in yarn-client mode
either, failing with the exception below.
Exception in thread main
No, not submitting from windows, from a debian distribution. Had a quick
look at the rm logs, and it seems some containers are allocated but then
released again for some reason. Not easy to make sense of the logs, but
here is a snippet from the logs (from a test in our small test cluster) if
you'd
This is tricky to debug. Check logs of node and resource manager of YARN to
see if you can trace the error. In the past I have to closely look at
arguments getting passed to YARN container (they get logged before
attempting to launch containers). If I still don't get a clue, I had to
check the
Hi,
Compiled the latest master of Spark yesterday (2015-02-10) for Hadoop 2.2
and failed executing jobs in yarn-cluster mode for that build. Works
successfully with spark 1.2 (and also master from 2015-01-16), so something
has changed since then that prevents the job from receiving any executors
Hi Anders,
I just tried this out and was able to successfully acquire executors. Any
strange log messages or additional color you can provide on your setup?
Does yarn-client mode work?
-Sandy
On Wed, Feb 11, 2015 at 1:28 PM, Anders Arpteg arp...@spotify.com wrote:
Hi,
Compiled the latest
10 matches
Mail list logo
