Hi team,
I missed an attachment. Please find it now.
-----Original Message-----
From: Hitesh Goyal [mailto:[email protected]]
Sent: Friday, September 23, 2016 1:10 PM
To: [email protected]
Subject: RE: Trying to acces live data
Hi team,
Please find application logs as attached files.
-----Original Message-----
From: Priyanka Gugale [mailto:[email protected]]
Sent: Friday, September 23, 2016 11:00 AM
To: dev
Subject: Re: Trying to acces live data
Hi Hitesh,
Can you collect all logs including app master logs and send it? Run command
"yarn logs -applicationId <you app id>" on cluster to get logs for an
application.
Also please collect stderr output logs of appMaster. You can find them under
"<your hadoop location>/logs/userlogs/<app id>/<containerId>/"
-Priyanka
On Fri, Sep 23, 2016 at 10:55 AM, Hitesh Goyal <[email protected]>
wrote:
> Hi team,
>
> Trying to access live data from couchbase using input Operators.
>
> It is running fine on my local. The tuples are emitting as Pojo
> through the output port of inputOperator.
>
> But when I upload the .apa file on data torrent, it gets launched but
> soon it goes into Failed State just after been accepted.
>
> Can you please help me to resolve the issue ? I am sending logs as an
> attached file.
>
> Note:- I am not using built in operators. I have made my own operators
> which are quite similar to CouchBasePojoInputOperator and CouchBaseStore.
>
>
>
> Regards,
>
> *Hitesh Goyal*
>
> Simpli5d Technologies
>
> Cont No.: 9996588220
>
>
>
2016-09-22 12:40:30,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Updating application attempt appattempt_1474540741175_0001_000001 with final
state: FAILED, and exit status: 1
2016-09-22 12:40:30,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000001 State change from LAUNCHED to FINAL_SAVING
2016-09-22 12:40:30,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Unregistering app attempt : appattempt_1474540741175_0001_000001
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Application finished, removing password for appattempt_1474540741175_0001_000001
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000001 State change from FINAL_SAVING to FAILED
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of
failed attempts is 1. The max attempts is 2
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Application Attempt appattempt_1474540741175_0001_000001 is done.
finalState=FAILED
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering app attempt : appattempt_1474540741175_0001_000002
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from NEW to SUBMITTED
2016-09-22 12:40:30,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo:
Application application_1474540741175_0001 requests cleared
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application removed - appId: application_1474540741175_0001 user: dtadmin
queue: default #user-pending-applications: 0 #user-active-applications: 0
#queue-pending-applications: 0 #queue-active-applications: 0
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application application_1474540741175_0001 from user: dtadmin activated in
queue: default
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application added - appId: application_1474540741175_0001 user:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@77d0922b,
leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1
#queue-pending-applications: 0 #queue-active-applications: 1
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Added Application Attempt appattempt_1474540741175_0001_000002 to scheduler
from user dtadmin in queue default
2016-09-22 12:40:30,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from SUBMITTED to SCHEDULED
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Null container completed...
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1474540741175_0001_02_000001 Container Transitioned from NEW to
ALLOCATED
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin
OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS
APPID=application_1474540741175_0001
CONTAINERID=container_1474540741175_0001_02_000001
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned
container container_1474540741175_0001_02_000001 of capacity <memory:1024,
vCores:1> on host localhost.localdomain:35189, which has 1 containers,
<memory:1024, vCores:1> used and <memory:28976, vCores:7> available after
allocation
2016-09-22 12:40:31,308 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
assignedContainer application attempt=appattempt_1474540741175_0001_000002
container=Container: [ContainerId: container_1474540741175_0001_02_000001,
NodeId: localhost.localdomain:35189, NodeHttpAddress:
localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0,
Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0,
usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0,
numApps=1, numContainers=0 clusterResource=<memory:30000, vCores:8>
2016-09-22 12:40:31,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Re-sorting assigned queue: root.default stats: default: capacity=1.0,
absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>,
usedCapacity=0.034133334, absoluteUsedCapacity=0.034133334, numApps=1,
numContainers=1
2016-09-22 12:40:31,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.034133334
absoluteUsedCapacity=0.034133334 used=<memory:1024, vCores:1>
cluster=<memory:30000, vCores:8>
2016-09-22 12:40:31,309 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM:
Sending NMToken for nodeId : localhost.localdomain:35189 for container :
container_1474540741175_0001_02_000001
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1474540741175_0001_02_000001 Container Transitioned from ALLOCATED to
ACQUIRED
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM:
Clear node set for appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Storing attempt: AppId: application_1474540741175_0001 AttemptId:
appattempt_1474540741175_0001_000002 MasterContainer: Container: [ContainerId:
container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189,
NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>,
Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189
}, ]
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from SCHEDULED to
ALLOCATED_SAVING
2016-09-22 12:40:31,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from ALLOCATED_SAVING to
ALLOCATED
2016-09-22 12:40:31,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching
masterappattempt_1474540741175_0001_000002
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up
container Container: [ContainerId: container_1474540741175_0001_02_000001,
NodeId: localhost.localdomain:35189, NodeHttpAddress:
localhost.localdomain:8042, Resource: <memory:1024, vCores:1>, Priority: 0,
Token: Token { kind: ContainerToken, service: 192.168.0.41:35189 }, ] for AM
appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to
launch container container_1474540741175_0001_02_000001 : ${JAVA_HOME}/bin/java
-Djava.io.tmpdir=$PWD/tmp -Xmx768m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/dt-heap-1.bin -Dhadoop.root.logger=INFO,RFA
-Dhadoop.log.dir=<LOG_DIR>
-Ddt.attr.APPLICATION_PATH=hdfs://localhost:54310/var/datatorrent/apps/application_1474540741175_0001
com.datatorrent.stram.StreamingAppMaster 1><LOG_DIR>/AppMaster.stdout
2><LOG_DIR>/AppMaster.stderr
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Create AMRMToken for ApplicationAttempt: appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Creating password for appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,321 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done
launching container Container: [ContainerId:
container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189,
NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>,
Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189
}, ] for AM appattempt_1474540741175_0001_000002
2016-09-22 12:40:31,321 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from ALLOCATED to LAUNCHED
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1474540741175_0001_02_000001 Container Transitioned from ACQUIRED to
COMPLETED
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
Completed container: container_1474540741175_0001_02_000001 in state:
COMPLETED event:FINISHED
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin
OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS
APPID=application_1474540741175_0001
CONTAINERID=container_1474540741175_0001_02_000001
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released
container container_1474540741175_0001_02_000001 of capacity <memory:1024,
vCores:1> on host localhost.localdomain:35189, which currently has 0
containers, <memory:0, vCores:0> used and <memory:30000, vCores:8> available,
release resources=true
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Updating application attempt appattempt_1474540741175_0001_000002 with final
state: FAILED, and exit status: 1
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from LAUNCHED to FINAL_SAVING
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
default used=<memory:0, vCores:0> numContainers=0 user=dtadmin
user-resources=<memory:0, vCores:0>
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Unregistering app attempt : appattempt_1474540741175_0001_000002
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
completedContainer container=Container: [ContainerId:
container_1474540741175_0001_02_000001, NodeId: localhost.localdomain:35189,
NodeHttpAddress: localhost.localdomain:8042, Resource: <memory:1024, vCores:1>,
Priority: 0, Token: Token { kind: ContainerToken, service: 192.168.0.41:35189
}, ] queue=default: capacity=1.0, absoluteCapacity=1.0,
usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0,
numApps=1, numContainers=0 cluster=<memory:30000, vCores:8>
2016-09-22 12:40:32,310 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0
used=<memory:0, vCores:0> cluster=<memory:30000, vCores:8>
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Re-sorting completed queue: root.default stats: default: capacity=1.0,
absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0,
absoluteUsedCapacity=0.0, numApps=1, numContainers=0
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Application attempt appattempt_1474540741175_0001_000002 released container
container_1474540741175_0001_02_000001 on node: host:
localhost.localdomain:35189 #containers=0 available=30000 used=0 with event:
FINISHED
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
Application finished, removing password for appattempt_1474540741175_0001_000002
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1474540741175_0001_000002 State change from FINAL_SAVING to FAILED
2016-09-22 12:40:32,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of
failed attempts is 2. The max attempts is 2
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating
application application_1474540741175_0001 with final state: FAILED
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1474540741175_0001 State change from ACCEPTED to FINAL_SAVING
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Application Attempt appattempt_1474540741175_0001_000002 is done.
finalState=FAILED
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo:
Application application_1474540741175_0001 requests cleared
2016-09-22 12:40:32,312 INFO
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating
info for app: application_1474540741175_0001
2016-09-22 12:40:32,313 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application removed - appId: application_1474540741175_0001 user: dtadmin
queue: default #user-pending-applications: 0 #user-active-applications: 0
#queue-pending-applications: 0 #queue-active-applications: 0
2016-09-22 12:40:32,313 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application
application_1474540741175_0001 failed 2 times due to AM Container for
appattempt_1474540741175_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking
page:http://data1:8088/proxy/application_1474540741175_0001/Then, click on
links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1474540741175_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
2016-09-22 12:40:32,314 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1474540741175_0001 State change from FINAL_SAVING to FAILED
2016-09-22 12:40:32,314 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Application removed - appId: application_1474540741175_0001 user: dtadmin
leaf-queue of parent: root #applications: 0
2016-09-22 12:40:32,315 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dtadmin
OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE
DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application
application_1474540741175_0001 failed 2 times due to AM Container for
appattempt_1474540741175_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking
page:http://data1:8088/proxy/application_1474540741175_0001/Then, click on
links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1474540741175_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
APPID=application_1474540741175_0001
2016-09-22 12:40:32,317 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:
appId=application_1474540741175_0001,name=MyFirstApplication,user=dtadmin,queue=default,state=FAILED,trackingUrl=http://data1:8088/cluster/app/application_1474540741175_0001,appMasterHost=N/A,startTime=1474540826765,finishTime=1474540832312,finalStatus=FAILED
16-09-22 12:40:33,311 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Null container completed...
