@yohann Looks like something is wrong with my environment which I am yet to
figure out but the theory so far makes sense and I had also tried it in
another environments with very minimal configuration like my environment
and it works fine so clearly something is wrong with my env I don't know
why
When you run on Yarn, you don’t even need to start a spark cluster (spark
master and slaves). Yarn receives a job and then allocate resources for the
application master and then its workers.
Check the resources available in the node section of the resource manager UI
(and is your node actually
@yohann sorry I am assuming you meant application master if so I believe
spark is the one that provides application master. Is there anyway to look
for how much resources are being requested and how much yarn is allowed to
provide? I would assume this is a common case if so I am not sure why these
yarn.scheduler.capacity.maximum-am-resource-percent by default is set to
0.1 and I tried changing it to 1.0 and still no luck. same problem
persists. The master here is yarn and I just trying to spawn spark-shell
--master yarn --deploy-mode client and run a simple world count so I am not
sure why
Following the logs from the resource manager:
2018-07-08 07:23:23,382 WARN
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in
queue, it is likely set too low. skipping enforcement to allow at
Are you able to run a simple Map Reduce job on yarn without any issues?
If you have any issues: I had this problem on Mac. Use CSRUTIL in Mac, to
disable it. Then add a softlink
sudo ln –s /usr/bin/java/bin/java
The new versions of Mac from EL Captain does not allow softlinks in
Hi,
It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu!
I am not running any code since I can't even spawn spark-shell with yarn as
master as described in my previous email. I just want to run simple word
count using yarn as master.
Thanks!
Below is the resource manager
You running on emr? You checked the emr logs?
Was in similar situation where job was stuck in accepted and then it
died..turned out to be an issue w. My code when running g with huge
data.perhaps try to reduce gradually the load til it works and then start
from there?
Not a huge help but I
Hi All,
I am trying to run a simple word count using YARN as a cluster manager. I
am currently using Spark 2.3.1 and Apache hadoop 2.7.3. When I spawn
spark-shell like below it gets stuck in ACCEPTED stated forever.
./bin/spark-shell --master yarn --deploy-mode client
I set my
