[
https://issues.apache.org/jira/browse/SAMZA-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654617#comment-14654617
]
Luis Casillas commented on SAMZA-601:
-------------------------------------
I just experienced the same symptoms with hello-samza as well, with 0.9.1. I
found the cause in my environment and managed to work around it.
The problem as I experienced it is that the YARN setup managed by the
{{bin/grid}} script gets confused over the fact that I have a Hadoop
installation somewhere else, and my shell has environment variables pointing at
that. Note below the message saying that resourcemanager is logging to an
extraneous Hadoop 2.4.1 installation:
{code}
% ./bin/grid start all
EXECUTING: start all
EXECUTING: start zookeeper
JMX enabled by default
Using config:
/Users/luis.casillas/GitHub/hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
EXECUTING: start yarn
starting resourcemanager, logging to
/Users/luis.casillas/Software/hadoop-2.4.1/logs/yarn-luis.casillas-resourcemanager-Luis-Macbook.local.out
2015-08-04 17:39:55.971 java[8208:433058] Unable to load realm mapping info
from SCDynamicStore
starting nodemanager, logging to
/Users/luis.casillas/Software/hadoop-2.4.1/logs/yarn-luis.casillas-nodemanager-Luis-Macbook.local.out
2015-08-04 17:39:57.001 java[8253:433135] Unable to load realm mapping info
from SCDynamicStore
EXECUTING: start kafka
{code}
So to work around this problem, I had to unset two environment variables:
* {{HADOOP_PREFIX}} (controls the logging destination)
* {{HADOOP_CONF_DIR}} (controls which {{yarn-site.xml}} the resource manager
will use)
If I may make a suggestion, it sounds like perhaps the {{bin/grid}} script in
the example ought to do something like this:
{code}
export HADOOP_PREFIX="`dirname $0`/../deploy/yarn"
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc
{code}
> hello-samza runs out of resources when all 3 jobs are run
> ----------------------------------------------------------
>
> Key: SAMZA-601
> URL: https://issues.apache.org/jira/browse/SAMZA-601
> Project: Samza
> Issue Type: Bug
> Components: hello-samza
> Affects Versions: 0.9.0
> Reporter: Navina Ramesh
> Assignee: Navina Ramesh
> Attachments: hello-samza-with-3-jobs.png
>
>
> There seems to be some change in how YARN chooses the resources. When we run
> all 3 jobs on a machine with 8GB RAM, the jobs don't proceed because
> application master itself claims 3GB of the total 4G memory.
> Need to understand how YARN assigns the vmem and vcore allocations on the
> nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
