Hey All,
I am running a Hadoop 2.0(cdh4.2.1) cluster on a single node with 1
NodeManager.
We have an Map only job that launches a pig job on the cluster(similar to
what oozie does)
We are seeing that the map only job launches the pig script but the pig job
is stuck in ACCEPTED state with no trackingUI assigned.
I dont see any error in the nodemanager logs or the resource manager logs
as such.
On the nodemanager i see this logs
2013-06-21 15:05:13,084 INFO capacity.ParentQueue - assignedContainer
queue=root usedCapacity=0.4 absoluteUsedCapacity=0.4 used=memory: 2048
cluster=memory: 5120
2013-06-21 15:05:38,898 INFO capacity.CapacityScheduler - Application
Submission: appattempt_1371850881510_0003_000001, user: smehta queue:
default: capacity=1.0, absoluteCapacity=1.0, usedResources=2048MB,
usedCapacity=0.4, absoluteUsedCapacity=0.4, numApps=2, numContainers=2,
currently active: 2
Which suggests that the cluster has capacity but still no application
master is assigned to it.
What am I missing?Any help is appreciated.
I keep seeing this logs on the node manager
2013-06-21 16:19:37,675 INFO monitor.ContainersMonitorImpl - Memory usage
of ProcessTree 12484 for container-id
container_1371850881510_0002_01_000002: 157.1mb of 1.0gb physical memory
used; 590.1mb of 2.1gb virtual memory used
2013-06-21 16:19:37,696 INFO monitor.ContainersMonitorImpl - Memory usage
of ProcessTree 12009 for container-id
container_1371850881510_0002_01_000001: 181.0mb of 1.0gb physical memory
used; 1.4gb of 2.1gb virtual memory used
2013-06-21 16:19:37,946 INFO nodemanager.NodeStatusUpdaterImpl - Sending
out status for container: container_id {, app_attempt_id {, application_id
{, id: 2, cluster_timestamp: 1371850881510, }, attemptId: 1, }, id: 1, },
state: C_RUNNING, diagnostics: "", exit_status: -1000,
2013-06-21 16:19:37,946 INFO nodemanager.NodeStatusUpdaterImpl - Sending
out status for container: container_id {, app_attempt_id {, application_id
{, id: 2, cluster_timestamp: 1371850881510, }, attemptId: 1, }, id: 2, },
state: C_RUNNING, diagnostics: "", exit_status: -1000,
2013-06-21 16:19:38,948 INFO nodemanager.NodeStatusUpdaterImpl - Sending
out status for container: container_id {, app_attempt_id {, application_id
{, id: 2, cluster_timestamp: 1371850881510, }, attemptId: 1, }, id: 1, },
state: C_RUNNING, diagnostics: "", exit_status: -1000,
2013-06-21 16:19:38,948 INFO nodemanager.NodeStatusUpdaterImpl - Sending
out status for container: container_id {, app_attempt_id {, application_id
{, id: 2, cluster_timestamp: 1371850881510, }, attemptId: 1, }, id: 2, },
state: C_RUNNING, diagnostics: "", exit_status: -1000,
2013-06-21 16:19:39,950 INFO nodemanager.NodeStatusUpdaterImpl - Sending
out status for container: container_id {, app_attempt_id {, application_id
{, id: 2, cluster_timestamp: 1371850881510, }, attemptId: 1, }, id: 1, },
state: C_RUNNING, diagnostics: "", exit_status: -1000,
2013-06-21 16:19:39,950 INFO nodemanager.NodeStatusUpdaterImpl - Sending
out status for container: container_id {, app_attempt_id {, application_id
{, id: 2, cluster_timestamp: 1371850881510, }, attemptId: 1, }, id: 2, },
state: C_RUNNING, diagnostics: "", exit_status: -1000,
Here are my memory configurations
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>5120</value>
<source>yarn-site.xml</source>
</property>
property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
<source>mapred-site.xml</source>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>512</value>
<source>mapred-site.xml</source>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>
-Xmx512m -Djava.net.preferIPv4Stack=true -XX:+UseCompressedOops
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/home/sfdc/logs/hadoop/userlogs/@taskid@/
</value>
<source>mapred-site.xml</source>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>1024</value>
<source>mapred-site.xml</source>
</property>
Regards,
Siddhi
