Hi team,
Trying to access live data from couchbase using input Operators.
It is running fine on my local. The tuples are emitting as Pojo through the
output port of inputOperator.
But when I upload the .apa file on data torrent, it gets launched but soon it
goes into Failed State just after been accepted.
Can you please help me to resolve the issue ? I am sending logs as an attached
file.
Regards,
Hitesh Goyal
Simpli5d Technologies
Cont No.: 9996588220
2016-09-22 12:40:30,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Updating application attempt appattempt_1474540741175_0001_000001 with final
state: FAILED, and exit status: 1
2016-09-22 12:40:30,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000001 State change from LAUNCHED to FINAL_SAVING
2016-09-22 12:40:30,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Unregistering app attempt : appattempt_1474540741175_0001_000001
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Application finished, removing password for appattempt_1474540741175_0001_000001
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000001 State change from FINAL_SAVING to FAILED
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of
failed attempts is 1. The max attempts is 2
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Application Attempt appattempt_1474540741175_0001_000001 is done.
finalState=FAILED
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering app attempt : appattempt_1474540741175_0001_000002
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from NEW to SUBMITTED
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo:
Application application_1474540741175_0001 requests cleared
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application removed - appId: application_1474540741175_0001 user: dtadmin
queue: default #user-pending-applications: 0 #user-active-applications: 0
#queue-pending-applications: 0 #queue-active-applications: 0
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application application_1474540741175_0001 from user: dtadmin activated in
queue: default
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application added - appId: application_1474540741175_0001 user:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@77d0922b,
leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1
#queue-pending-applications: 0 #queue-active-applications: 1
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Added Application Attempt appattempt_1474540741175_0001_000002 to scheduler
from user dtadmin in queue default
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from SUBMITTED to SCHEDULED
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Null container completed...
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1474540741175_0001_02_000001 Container Transitioned from NEW to
ALLOCATED
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin
OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS
APPID=application_1474540741175_0001
CONTAINERID=container_1474540741175_0001_02_000001
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned
container container_1474540741175_0001_02_000001 of capacity <memory:1024,
vCores:1> on host localhost.localdomain:35189, which has 1 containers,
<memory:1024, vCores:1> used and <memory:28976, vCores:7> available after
allocation
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
assignedContainer application attempt=appattempt_1474540741175_0001_000002
container=Container: [ContainerId: container_1474540741175_0001_02_000001,
NodeId: localhost.localdomain:35189, NodeHttpAddress:
localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0,
Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0,
usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0,
numApps=1, numContainers=0 clusterResource=<memory:30000, vCores:8>
2016-09-22 12:40:31,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Re-sorting assigned queue: root.default stats: default: capacity=1.0,
absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>,
usedCapacity=0.034133334, absoluteUsedCapacity=0.034133334, numApps=1,
numContainers=1
2016-09-22 12:40:31,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.034133334
absoluteUsedCapacity=0.034133334 used=<memory:1024, vCores:1>
cluster=<memory:30000, vCores:8>
2016-09-22 12:40:31,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM:
Sending NMToken for nodeId : localhost.localdomain:35189 for container :
container_1474540741175_0001_02_000001
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1474540741175_0001_02_000001 Container Transitioned from ALLOCATED to
ACQUIRED
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM:
Clear node set for appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Storing attempt: AppId: application_1474540741175_0001 AttemptId:
appattempt_1474540741175_0001_000002 MasterContainer: Container: [ContainerId:
container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189,
NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>,
Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189
}, ]
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from SCHEDULED to
ALLOCATED_SAVING
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from ALLOCATED_SAVING to
ALLOCATED
2016-09-22 12:40:31,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching
masterappattempt_1474540741175_0001_000002
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up
container Container: [ContainerId: container_1474540741175_0001_02_000001,
NodeId: localhost.localdomain:35189, NodeHttpAddress:
localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0,
Token: Token { kind: ContainerToken, service: 192.168.0.41:35189 }, ] for AM
appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to
launch container container_1474540741175_0001_02_000001 : ${JAVA_HOME}/bin/java
-Djava.io.tmpdir=$PWD/tmp -Xmx768m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/dt-heap-1.bin -Dhadoop.root.logger=INFO,RFA
-Dhadoop.log.dir=<LOG_DIR>
-Ddt.attr.APPLICATION_PATH=hdfs://localhost:54310/var/datatorrent/apps/application_1474540741175_0001
com.datatorrent.stram.StreamingAppMaster 1><LOG_DIR>/AppMaster.stdout
2><LOG_DIR>/AppMaster.stderr
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Create AMRMToken for ApplicationAttempt: appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Creating password for appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,321 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done
launching container Container: [ContainerId:
container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189,
NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>,
Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189
}, ] for AM appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,321 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from ALLOCATED to LAUNCHED
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1474540741175_0001_02_000001 Container Transitioned from ACQUIRED to
COMPLETED
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
Completed container: container_1474540741175_0001_02_000001 in state:
COMPLETED event:FINISHED
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin
OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS
APPID=application_1474540741175_0001
CONTAINERID=container_1474540741175_0001_02_000001
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released
container container_1474540741175_0001_02_000001 of capacity <memory:1024,
vCores:1> on host localhost.localdomain:35189, which currently has 0
containers, <memory:0, vCores:0> used and <memory:30000, vCores:8> available,
release resources=true
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Updating application attempt appattempt_1474540741175_0001_000002 with final
state: FAILED, and exit status: 1
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from LAUNCHED to FINAL_SAVING
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
default used=<memory:0, vCores:0> numContainers=0 user=dtadmin
user-resources=<memory:0, vCores:0>
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Unregistering app attempt : appattempt_1474540741175_0001_000002
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
completedContainer container=Container: [ContainerId:
container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189,
NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>,
Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189
}, ] queue=default: capacity=1.0, absoluteCapacity=1.0,
usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0,
numApps=1, numContainers=0 cluster=<memory:30000, vCores:8>
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0
used=<memory:0, vCores:0> cluster=<memory:30000, vCores:8>
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Re-sorting completed queue: root.default stats: default: capacity=1.0,
absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0,
absoluteUsedCapacity=0.0, numApps=1, numContainers=0
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Application attempt appattempt_1474540741175_0001_000002 released container
container_1474540741175_0001_02_000001 on node: host:
localhost.localdomain:35189 #containers=0 available=30000 used=0 with event:
FINISHED
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Application finished, removing password for appattempt_1474540741175_0001_000002
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from FINAL_SAVING to FAILED
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of
failed attempts is 2. The max attempts is 2
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating
application application_1474540741175_0001 with final state: FAILED
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1474540741175_0001 State change from ACCEPTED to FINAL_SAVING
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Application Attempt appattempt_1474540741175_0001_000002 is done.
finalState=FAILED
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo:
Application application_1474540741175_0001 requests cleared
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating
info for app: application_1474540741175_0001
2016-09-22 12:40:32,313 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application removed - appId: application_1474540741175_0001 user: dtadmin
queue: default #user-pending-applications: 0 #user-active-applications: 0
#queue-pending-applications: 0 #queue-active-applications: 0
2016-09-22 12:40:32,313 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application
application_1474540741175_0001 failed 2 times due to AM Container for
appattempt_1474540741175_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking
page:http://data1:8088/proxy/application_1474540741175_0001/Then, click on
links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1474540741175_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
2016-09-22 12:40:32,314 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1474540741175_0001 State change from FINAL_SAVING to FAILED
2016-09-22 12:40:32,314 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Application removed - appId: application_1474540741175_0001 user: dtadmin
leaf-queue of parent: root #applications: 0
2016-09-22 12:40:32,315 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin
OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE
DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application
application_1474540741175_0001 failed 2 times due to AM Container for
appattempt_1474540741175_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking
page:http://data1:8088/proxy/application_1474540741175_0001/Then, click on
links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1474540741175_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
APPID=application_1474540741175_0001
2016-09-22 12:40:32,317 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:
appId=application_1474540741175_0001,name=MyFirstApplication,user=dtadmin,queue=default,state=FAILED,trackingUrl=http://data1:8088/cluster/app/application_1474540741175_0001,appMasterHost=N/A,startTime=1474540826765,finishTime=1474540832312,finalStatus=FAILED
16-09-22 12:40:33,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Null container completed...
