Hi,
I got another problem when running the Twitter Trend Example in cluster
mode.
I can deploy the twitter-counter and twitter-adapter.
For adapter, there are 2 containers running, one is the AppMaster and
the other is the adapter application.
For counter, there are also 2 containers running, one is the AppMaster
and the other is the counter application.
There should be one more container running for counter application, yes?
I check the logs of the AppMaster of twitter-counter.
Two containers for counter application were set up, but one of them was
completed. I don't know why.
Is this problem related to memory allocation?
10:10:55.925 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Initializing
ApplicationMaster with args [--container_memory, 10, --num_containers, 2,
--priority, 0, -c, counter, -zk, testing.machine1:2181]
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Application
master for app, appId=1, clustertimestamp=1353031752144, attemptId=1
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Application
master args = []
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - arg / Amount
of memory in MB to be requested to host the S4 node / container_memory / 10
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - arg / No. of
containers on which the S4 node needs to be hosted / num_containers / 2
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - arg /
Application Priority. Default 0 / priority / 0
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - arg / The
cluster (or application) name / c / counter
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - arg /
Zookeeper connection string / zk / testing.machine1:2181
10:10:56.062 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Application
master args = [[ option: container_memory [ARG] :: Amount of memory in MB to
be requested to host the S4 node ], [ option: num_containers [ARG] :: No. of
containers on which the S4 node needs to be hosted ], [ option: priority [ARG]
:: Application Priority. Default 0 ], [ option: c cluster [ARG] :: The cluster
(or application) name ], [ option: zk [ARG] :: Zookeeper connection string ]]
10:10:56.111 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Starting
ApplicationMaster
10:10:56.209 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Connecting to
ResourceManager at testing.machine1/192.168.0.100:8030
<http://192.168.0.100:8030>
10:10:56.672 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Min mem
capability of resources in this cluster 512
10:10:56.673 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Max mem
capability of resources in this cluster 3072
10:10:56.673 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Container
memory specified below min threshold of cluster. Using min value.,
specified=10, min=512
10:10:56.673 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=0, appDone=false, total=2, requested=0, completed=0,
failed=0, currentAllocated=0
10:10:57.682 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=2
10:10:57.684 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=1, releasedSet=0, progress=0.0
10:10:57.684 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Requested
container ask: {Priority: {Priority: 0}, Capability: memory: 512}
10:10:57.698 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:10:57.698 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 5632
10:10:57.699 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:10:57.699 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=0, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=0
10:10:57.699 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=1, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=0
10:10:58.699 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:10:58.699 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:10:58.715 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=2
10:10:58.717 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Launching S4
node on a new container., containerId=container_1353031752144_0001_01_000002,
containerNode=testing.machine1:40558, containerNodeURI=testing.machine1:8042,
containerStateNEW, containerResourceMemory512
10:10:58.719 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Launching S4
node on a new container., containerId=container_1353031752144_0001_01_000003,
containerNode=testing.machine2:44677, containerNodeURI=testing.machine2:8042,
containerStateNEW, containerResourceMemory512
10:10:58.719 [Thread-2] DEBUG o.a.s.tools.yarn.S4ApplicationMaster - Connecting
to ContainerManager for containerid=container_1353031752144_0001_01_000002
10:10:58.719 [Thread-2] INFO o.a.s.tools.yarn.S4ApplicationMaster - Connecting
to ContainerManager at testing.machine1:40558
10:10:58.726 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:10:58.726 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:10:58.726 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=1, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:10:58.726 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=2, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:10:58.726 [Thread-3] DEBUG o.a.s.tools.yarn.S4ApplicationMaster - Connecting
to ContainerManager for containerid=container_1353031752144_0001_01_000003
10:10:58.726 [Thread-3] INFO o.a.s.tools.yarn.S4ApplicationMaster - Connecting
to ContainerManager at testing.machine2:44677
10:10:58.735 [Thread-3] INFO o.a.s.tools.yarn.S4ApplicationMaster - Setting up
container launch container for
containerid=container_1353031752144_0001_01_000003
10:10:58.735 [Thread-2] INFO o.a.s.tools.yarn.S4ApplicationMaster - Setting up
container launch container for
containerid=container_1353031752144_0001_01_000002
10:10:58.740 [Thread-2] INFO o.a.s.tools.yarn.S4ApplicationMaster - Using
default user name root
10:10:58.740 [Thread-3] INFO o.a.s.tools.yarn.S4ApplicationMaster - Using
default user name root
10:10:59.726 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:10:59.726 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:10:59.728 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:10:59.728 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:10:59.728 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:10:59.728 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=2, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:10:59.728 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=3, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:00.729 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:00.729 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:00.731 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:00.731 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:00.731 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:00.731 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=3, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:00.731 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=4, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:01.731 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:01.731 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:01.733 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:01.733 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:01.733 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:01.733 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=4, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:01.733 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=5, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:02.733 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:02.733 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:02.736 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:02.736 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:02.736 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:02.736 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=5, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:02.736 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=6, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:03.736 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:03.736 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:03.738 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:03.739 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:03.739 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:03.739 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=6, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:03.739 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=7, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:04.739 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:04.739 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:04.742 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:04.742 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:04.742 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:04.742 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=7, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:04.742 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=8, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:05.742 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:05.743 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:05.745 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:05.745 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:05.745 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:05.745 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=8, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:05.745 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=9, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:06.745 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:06.746 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:06.748 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:06.748 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:06.748 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:06.748 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=9, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:06.748 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=10, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:07.748 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:07.749 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:07.753 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:07.754 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:07.754 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:07.754 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=10, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:07.754 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=11, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:08.754 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:08.754 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:08.756 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:08.757 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:08.757 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:08.757 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=11, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:08.757 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=12, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:09.757 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:09.757 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:09.764 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:09.765 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:09.765 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:09.765 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=12, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:09.765 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=13, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:10.765 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:10.765 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:10.768 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:10.768 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:10.768 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:10.768 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=13, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:10.768 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=14, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:11.768 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:11.769 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:11.772 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:11.772 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:11.772 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:11.772 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=14, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:11.773 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=15, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:12.773 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:12.774 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:12.776 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:12.777 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:12.777 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:12.777 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=15, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:12.777 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=16, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:13.777 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:13.777 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:13.779 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:13.780 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:13.780 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:13.780 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=16, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:13.780 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=17, appDone=false, total=2, requested=2, completed=0,
failed=0, currentAllocated=2
10:11:14.780 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:14.780 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.0
10:11:14.782 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:14.782 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:14.783 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=1
10:11:14.783 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got container
status for containerID= container_1353031752144_0001_01_000003, state=COMPLETE,
exitStatus=0, diagnostics=
10:11:14.783 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Container
completed successfully., containerId=container_1353031752144_0001_01_000003
10:11:14.783 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=17, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:14.783 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=18, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:15.783 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:15.784 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.5
10:11:15.786 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:15.786 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:15.786 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:15.786 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=18, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:15.786 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=19, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:16.786 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:16.787 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.5
10:11:16.789 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:16.790 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:16.790 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:16.790 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=19, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:16.790 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=20, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:17.790 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:17.790 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.5
10:11:17.793 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:17.793 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:17.793 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:17.793 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=20, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:17.793 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=21, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:18.793 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for
containers, askCount=0
10:11:18.793 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Sending
request to RM for containers, requestedSet=0, releasedSet=0, progress=0.5
10:11:18.796 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, allocatedCnt=0
10:11:18.796 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
available resources in the cluster memory: 4608
10:11:18.796 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Got response
from RM for container ask, completedCnt=0
10:11:18.796 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=21, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
10:11:18.796 [main] INFO o.a.s.tools.yarn.S4ApplicationMaster - Current
application state: loop=22, appDone=false, total=2, requested=2, completed=1,
failed=0, currentAllocated=2
On Fri, Nov 16, 2012 at 2:32 AM, Matthieu Morel
<[email protected] <mailto:[email protected]>> wrote:
Hi,
What specific parameter are you trying to set in the yarn
configuration? It looks like you have a max of 128MB per container.
What seems to be failing is the application master. It uses (by
default) 128MB for heap + default stack size, but that would be over
128MB.
Of course, we still have to properly handle all parameters available in
the yarn command, including memory settings, but from what I see the
issue you are facing is related to the configuration of your Yarn
cluster. You can get more information from other log files (apart from
the client console logs), maybe that can help as well. But somewhere
you seem to be capping the container memory to an insufficient value.
Regards,
Matthieu
On Thu, Nov 15, 2012 at 9:12 AM, Frank Zheng
<[email protected] <mailto:[email protected]>> wrote:
> Hi All,
>
> I am confused about the memory configuration.
> In the S4ApplicationMaster.java, it says:
> // A resource ask has to be at least the minimum of the
capability
> of the cluster, the value has to be
> // a multiple of the min value and cannot exceed the max.
> // If it is not an exact multiple of min, the RM will
allocate to
> the nearest multiple of min
> So I set the minimum memory as 128 in the yarn-site.xml
> But when I deployed the Twitter Counter application, I got this
error.
>
> 17:04:25.558 [main] INFO o.apache.s4.tools.yarn.S4YarnClient - Got
> application report from ASM for, appId=1, clientToken=null,
appDiagnostics=,
> appMasterHost=, appQueue=default, appMasterRpcPort=0,
> appStartTime=1352970249485, yarnAppState=RUNNING,
> distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
> 17:04:26.560 [main] INFO o.apache.s4.tools.yarn.S4YarnClient - Got
> application report from ASM for, appId=1, clientToken=null,
appDiagnostics=,
> appMasterHost=, appQueue=default, appMasterRpcPort=0,
> appStartTime=1352970249485, yarnAppState=RUNNING,
> distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
> 17:04:27.563 [main] INFO o.apache.s4.tools.yarn.S4YarnClient - Got
> application report from ASM for, appId=1, clientToken=null,
appDiagnostics=,
> appMasterHost=, appQueue=default, appMasterRpcPort=0,
> appStartTime=1352970249485, yarnAppState=RUNNING,
> distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
> 17:04:28.569 [main] INFO o.apache.s4.tools.yarn.S4YarnClient - Got
> application report from ASM for, appId=1, clientToken=null,
> appDiagnostics=Application application_1352970173321_0001 failed
1 times due
> to AM Container for appattempt_1352970173321_0001_000001 exited with
> exitCode: 143 due to: Container
> [pid=12119,containerID=container_1352970173321_0001_01_000001] is
running
> beyond physical memory limits. Current usage: 138.2mb of 128.0mb
physical
> memory used; 1.1gb of 5.2gb virtual memory used. Killing container.
> Dump of the process-tree for container_1352970173321_0001_01_000001 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
FULL_CMD_LINE
> |- 12213 12119 12119 12119 (java) 285 11 1085763584 35085
java -Xmx128m
> org.apache.s4.tools.yarn.S4ApplicationMaster --container_memory 10
> --num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
> |- 12119 9509 12119 12119 (bash) 4 3 108703744 304 /bin/bash
-c java
> -Xmx128m org.apache.s4.tools.yarn.S4ApplicationMaster
--container_memory 10
> --num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
>
1>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stdout
>
2>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stderr
>
>
> .Failing this attempt.. Failing the application., appMasterHost=,
> appQueue=default, appMasterRpcPort=0, appStartTime=1352970249485,
> yarnAppState=FAILED, distributedFinalState=FAILED, appTrackingUrl=,
> appUser=root
> 17:04:28.569 [main] INFO o.apache.s4.tools.yarn.S4YarnClient -
Application
> did not finish. YarnState=FAILED, DSFinalStatus=FAILED. Breaking
monitoring
> loop
> 17:04:28.569 [main] ERROR o.apache.s4.tools.yarn.S4YarnClient -
Application
> failed to complete successfully
>
>
> Should the S4AppMaster use the multiple of minimum memory
automatically? Why
> the memory of container is only 128 MB?
>
>
> Sincerely,
> Zheng Yu
> Mobile: (852) 60670059
> Email: [email protected] <mailto:[email protected]>
>
>
>
--
Sincerely,
Zheng Yu
Mobile: (852) 60670059
Email: [email protected] <mailto:[email protected]>
