Hi All, As druid uses Hadoop MapReduce to ingest batch data but I am trying spark for ingesting data into druid taking reference from https://github.com/metamx/druid-spark-batch But we are stuck at the following error. Application Log:—>2018-03-20T07:54:28,782 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Will allocate AM container, with 896 MB memory including 384 MB overhead 2018-03-20T07:54:28,782 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Setting up container launch context for our AM 2018-03-20T07:54:28,785 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Setting up the launch environment for our AM container 2018-03-20T07:54:28,793 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Preparing resources for our AM container 2018-03-20T07:54:29,364 WARN [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 2018-03-20T07:54:29,371 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Uploading resource file:/hdfs1/druid-0.11.0/var/tmp/spark-49af67df-1a21-4790-a02b-c737c7a44946/__spark_libs__8247917347016008883.zip -> hdfs://n2pl-pa-hdn220.xxx.xxx:8020/user/yarn/.sparkStaging/application_1521457397747_0013/__spark_libs__8247917347016008883.zip 2018-03-20T07:54:29,607 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Uploading resource file:/hdfs1/druid-0.11.0/var/tmp/spark-49af67df-1a21-4790-a02b-c737c7a44946/__spark_conf__2240950972346324291.zip -> hdfs://n2pl-pa-hdn220.xxx.xxx:8020/user/yarn/.sparkStaging/application_1521457397747_0013/__spark_conf__.zip 2018-03-20T07:54:29,673 INFO [task-runner-0-priority-0] org.apache.spark.SecurityManager - Changing view acls to: yarn 2018-03-20T07:54:29,673 INFO [task-runner-0-priority-0] org.apache.spark.SecurityManager - Changing modify acls to: yarn 2018-03-20T07:54:29,673 INFO [task-runner-0-priority-0] org.apache.spark.SecurityManager - Changing view acls groups to: 2018-03-20T07:54:29,673 INFO [task-runner-0-priority-0] org.apache.spark.SecurityManager - Changing modify acls groups to: 2018-03-20T07:54:29,673 INFO [task-runner-0-priority-0] org.apache.spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn); groups with view permissions: Set(); users with modify permissions: Set(yarn); groups with modify permissions: Set() 2018-03-20T07:54:29,679 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Submitting application application_1521457397747_0013 to ResourceManager 2018-03-20T07:54:29,709 INFO [task-runner-0-priority-0] org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1521457397747_0013 2018-03-20T07:54:29,713 INFO [task-runner-0-priority-0] org.apache.spark.scheduler.cluster.SchedulerExtensionServices - Starting Yarn extension services with app application_1521457397747_0013 and attemptId None 2018-03-20T07:54:30,722 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - Application report for application_1521457397747_0013 (state: FAILED) 2018-03-20T07:54:30,729 INFO [task-runner-0-priority-0] org.apache.spark.deploy.yarn.Client - client token: N/A diagnostics: Application application_1521457397747_0013 failed 2 times due to AM Container for appattempt_1521457397747_0013_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://n-pa-hdn220.xxx.xxxx:8088/cluster/app/application_1521457397747_0013 Then click on links to logs of each attempt. Diagnostics: No such file or directory ENOENT: No such file or directory at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:756) at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:211) at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:252) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:1003) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:999) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:1006) at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:421) at org.apache.hadoop.yarn.util.FSDownload$3.run(FSDownload.java:419) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:419) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:365) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) As far as I can understand there is something wrong with the job submission through Yarn. On local machine it is running but HDP cluster it is giving error. |
Binary file nodemanager/recovery-state/yarn-nm-state/000552.log matches
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:23,713 INFO
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:startContainerInternal(850)) - Creating a new
application reference for app application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:23,718 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type INIT_APPLICATION
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:23,720 INFO
application.ApplicationImpl(ApplicationImpl.java:handle(464)) - Application
application_1521457397747_0002 transitioned from NEW to INITING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,026 DEBUG
hdfs.DFSClient (DFSClient.j ava:primitiveMkdir(3069)) -
/app-logs/druid/logs-ifile/application_1521457397747_0002: masked=rwxr-x---
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,036 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type INIT_CONTAINER
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,037 INFO
application.ApplicationImpl(ApplicationImpl.java:transition(304)) - Adding
container_e11_1521457397747_0002_01_000001 to application
application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,037 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_LOG_HANDLING_INITED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,039 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_INITED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,040 INFO
application.ApplicationImpl(ApplicationImpl.java:handle(464)) - Application
application_1521457397747_0002 transitioned from INITING to RUNNING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,042 INFO
containermanager.AuxServices (AuxServices.java:handle(215)) - Got event
CONTAINER_INIT for appId application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 DEBUG
localizer.LocalizedResource(LocalizedResource.java:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_libs__6487569756535459002.zip
of type REQUEST
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 INFO
localizer.LocalizedResource(LocalizedResource.java:handle(203)) - Resource
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_libs__6487569756535459002.zip
transitioned from INIT to DOWNLOADING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 DEBUG
localizer.LocalizedResource(LocalizedResource.java:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_conf__.zip
of type REQUEST
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 INFO
localizer.LocalizedResource(LocalizedResource.java:handle(203)) - Resource
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_conf__.zip
transitioned from INIT to DOWNLOADING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,345 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:startLocalizer(126)) - Copying from
/hdfs1/hadoop/yarn/local/nmPrivate/container_e11_1521457397747_0002_01_000001.tokens
to
/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002/container_e11_1521457397747_0002_01_000001.tokens
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,345 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:startLocalizer(133)) - Localizer CWD set to
/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
=
file:/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,511 WARN
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:processHeartbeat(1017)) - {
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_libs__6487569756535459002.zip,
1521457763387, ARCHIVE, null} failed: No such file or directory
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,512 DEBUG
localizer.LocalizedResource(LocalizedResource.java:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_libs__6487569756535459002.zip
of type LOCALIZATION_FAILED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,513 INFO
localizer.LocalizedResource(LocalizedResource.java:handle(203)) - Resource
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_libs__6487569756535459002.zip(->/hdfs1/hadoop/yarn/local/usercache/druid/filecache/10/__spark_libs__6487569756535459002.zip)
transitioned from DOWNLOADING to FAILED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,516 INFO
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(165)) - Container
container_e11_1521457397747_0002_01_000001 sent RELEASE event on a resource
request {
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_libs__6487569756535459002.zip,
1521457763387, ARCHIVE, null } not present in cache.
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,516 DEBUG
localizer.LocalizedResource(LocalizedResource.java:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_conf__.zip
of type RELEASE
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,521 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_CONTAINER_FINISHED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,522 INFO
application.ApplicationImpl(ApplicationImpl.java:transition(347)) - Removing
container_e11_1521457397747_0002_01_000001 from application
application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,522 INFO
containermanager.AuxServices (AuxServices.java:handle(215)) - Got event
CONTAINER_STOP for appId application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,536 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type FINISH_APPLICATION
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,536 INFO
application.ApplicationImpl(ApplicationImpl.java:handle(464)) - Application
application_1521457397747_0002 transitioned from RUNNING to
APPLICATION_RESOURCES_CLEANINGUP
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs1/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,543 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs1/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs10/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,544 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs10/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs11/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,544 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs11/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs12/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,545 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs12/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs11/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs10/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs1/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs12/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs13/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs14/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs14/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs16/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs16/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs13/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs15/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs15/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs13/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs16/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs14/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,549 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs15/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs2/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,549 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs2/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,550 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs2/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs3/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs5/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,551 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs5/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,551 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs5/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,551 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs3/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs3/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs7/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs7/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs6/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs6/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs7/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs8/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs8/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 INFO
containermanager.AuxServices (AuxServices.java:handle(215)) - Got event
APPLICATION_STOP for appId application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs8/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs9/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 DEBUG
nodemanager.DeletionService(DeletionService.java:run(292)) - Deleting path:
[/hdfs9/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs6/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs9/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,556 INFO
yarn.YarnShuffleService (YarnShuffleService.java:stopApplication(179)) -
Stopping application application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,556 INFO
shuffle.ExternalShuffleBlockResolver
(ExternalShuffleBlockResolver.java:applicationRemoved(206)) - Application
application_1521457397747_0002 removed, cleanupLocalDirs = false
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,560 INFO
shuffle.ExternalShuffleBlockResolver
(ExternalShuffleBlockResolver.java:applicationRemoved(186)) - Application
application_1521457397747_0002 removed, cleanupLocalDirs = false
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,560 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_RESOURCES_CLEANEDUP
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,560 DEBUG
security.NMTokenSecretManagerInNM
(NMTokenSecretManagerInNM.java:appFinished(191)) - Removing application
attempts NMToken keys for application application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,561 INFO
application.ApplicationImpl(ApplicationImpl.java:handle(464)) - Application
application_1521457397747_0002 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,561 INFO
logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:finishLogAggregation(413)) - Application just
finished : application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,589 DEBUG
hdfs.DFSClient (DFSOutputStream.java:computePacketChunkSize(1924)) -
computePacketChunkSize:
src=/app-logs/druid/logs-ifile/application_1521457397747_0002/n2pl-pa-xxxxxx_45454,
chunkSize=516, chunksPerPacket=126, packetSize=65016
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,614 DEBUG
hdfs.DFSClient (DFSOutputStream.java:writeChunkImpl(2019)) - DFSClient
writeChunk allocating new packet seqno=0,
src=/app-logs/druid/logs-ifile/application_1521457397747_0002/n2pl-pa-xxxxxx_45454,
packetSize=65016, chunksPerPacket=126, bytesCurBlock=0
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir : null baseDir:
/hdfs1/hadoop/yarn/log/application_1521457397747_0002,/hdfs10/hadoop/yarn/log/application_1521457397747_0002,/hdfs11/hadoop/yarn/log/application_1521457397747_0002,/hdfs12/hadoop/yarn/log/application_1521457397747_0002,/hdfs13/hadoop/yarn/log/application_1521457397747_0002,/hdfs14/hadoop/yarn/log/application_1521457397747_0002,/hdfs15/hadoop/yarn/log/application_1521457397747_0002,/hdfs16/hadoop/yarn/log/application_1521457397747_0002,/hdfs2/hadoop/yarn/log/application_1521457397747_0002,/hdfs3/hadoop/yarn/log/application_1521457397747_0002,/hdfs4/hadoop/yarn/log/application_1521457397747_0002,/hdfs5/hadoop/yarn/log/application_1521457397747_0002,/hdfs6/hadoop/yarn/log/application_1521457397747_0002,/hdfs7/hadoop/yarn/log/application_1521457397747_0002,/hdfs8/hadoop/yarn/log/application_1521457397747_0002,/hdfs9/hadoop/yarn/log/application_1521457397747_0002,
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,706 DEBUG
application.ApplicationImpl(ApplicationImpl.java:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_LOG_HANDLING_FINISHED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,706 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs1/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs10/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs11/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs12/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs13/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs14/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs15/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs16/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs2/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs3/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs4/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs5/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs6/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs7/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs8/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs9/hadoop/yarn/log/application_1521457397747_0002
yarn/nm-audit.log:2018-03-19 16:39:23,726 INFO nodemanager.NMAuditLogger:
USER=druid IP=10.5.233.220 OPERATION=StartContainer Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1521457397747_0002
CONTAINERID=container_e11_1521457397747_0002_01_000001
yarn/nm-audit.log:2018-03-19 16:39:24,519 WARN nodemanager.NMAuditLogger:
USER=druid OPERATION=Container Finished - Failed TARGET=ContainerImpl
RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED
APPID=application_1521457397747_0002
CONTAINERID=container_e11_1521457397747_0002_01_000001
[abinitio@n2pl-pa-hdd230 hadoop-yarn]$
[abinitio@n2pl-pa-hdd230 hadoop-yarn]$ sudo grep -r
application_1521457397747_0002
Binary file nodemanager/recovery-state/yarn-nm-state/000552.log matches
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:23,713 INFO
containermanager.ContainerManagerImpl
(ContainrManagerImpl.java:startContainerInternal(850)) - Creating a new
application reference for app application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:23,718 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type INIT_APPLICATION
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:23,720 INFO
application.ApplicationImpl (ApplicationImpl.jva:handle(464)) - Application
application_1521457397747_0002 transitioned from NEW to INITING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,026 DEBUG
hdfs.DFSClient (DFSClient.java:primitiveMkdir(069)) -
/app-logs/druid/logs-ifile/application_1521457397747_0002: masked=rwxr-x---
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,036 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type INIT_CONTAINER
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,037 INFO
application.ApplicationImpl (ApplicationImpl.jva:transition(304)) - Adding
container_e11_1521457397747_0002_01_000001 to application
application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,037 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_LOG_HANDLING_INITED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,039 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_INITED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,040 INFO
application.ApplicationImpl (ApplicationImpl.jva:handle(464)) - Application
application_1521457397747_0002 transitioned from INITING to RUNNING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,042 INFO
containermanager.AuxServices (AuxServices.javahandle(215)) - Got event
CONTAINER_INIT for appId application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 DEBUG
localizer.LocalizedResource (LocalizedResourcejava:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_ibs__6487569756535459002.zip
of type REQUEST
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 INFO
localizer.LocalizedResource (LocalizedResourcejava:handle(203)) - Resource
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_lis__6487569756535459002.zip
transitioned from INIT to DOWNLOADING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 DEBUG
localizer.LocalizedResource (LocalizedResourcejava:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_onf__.zip
of type REQUEST
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,046 INFO
localizer.LocalizedResource (LocalizedResourcejava:handle(203)) - Resource
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_cof__.zip
transitioned from INIT to DOWNLOADING
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,345 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:startLocalizer(126)) - Copying from
/hdfs1/hadoop/yarn/local/nmPrivate/container_e11_1521457397747_0002_01_000001.tokes
to
/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002/container_e11_1521457397747_0002_01_000001.tokens
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,345 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:startLocalizer(133)) - Localizer CWD set to
/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_152145739774_0002
=
file:/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,511 WARN
localizer.ResourceLocalizationService
(ResourcLocalizationService.java:processHeartbeat(1017)) - {
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_152147397747_0002/__spark_libs__6487569756535459002.zip,
1521457763387, ARCHIVE, null } failed: No such file or directory
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,512 DEBUG
localizer.LocalizedResource (LocalizedResourcejava:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_ibs__6487569756535459002.zip
of type LOCALIZATION_FAILED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,513 INFO
localizer.LocalizedResource (LocalizedResourcejava:handle(203)) - Resource
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_lis__6487569756535459002.zip(->/hdfs1/hadoop/yarn/local/usercache/druid/filecache/10/__spark_libs__6487569756535459002.zip)
transitioned fromDOWNLOADING to FAILED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,516 INFO
localizer.LocalResourcesTrackerImpl (LocalResorcesTrackerImpl.java:handle(165))
- Container container_e11_1521457397747_0002_01_000001 sent RELEASE event on a
resource request {
hdfs://2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_libs__6487569756535459002.zip,
152145773387, ARCHIVE, null } not present in cache.
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,516 DEBUG
localizer.LocalizedResource (LocalizedResourcejava:handle(193)) - Processing
hdfs://n2pl-pa-hdn220.xxxx.xx:8020/user/druid/.sparkStaging/application_1521457397747_0002/__spark_onf__.zip
of type RELEASE
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,521 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_CONTAINER_FINISHED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,522 INFO
application.ApplicationImpl (ApplicationImpl.jva:transition(347)) - Removing
container_e11_1521457397747_0002_01_000001 from application
application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:24,522 INFO
containermanager.AuxServices (AuxServices.javahandle(215)) - Got event
CONTAINER_STOP for appId application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,536 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type FINISH_APPLICATION
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,536 INFO
application.ApplicationImpl (ApplicationImpl.jva:handle(464)) - Application
application_1521457397747_0002 transitioned from RUNNING to
APPLICATION_RESOURCES_CLEANINGUP
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs1/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,543 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs1/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs10/hadoop/yarn/local/usercahe/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,544 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs10/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs11/hadoop/yarn/local/usercahe/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,544 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs11/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs12/hadoop/yarn/local/usercahe/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,545 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs12/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs11/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs10/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs1/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,546 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs12/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs13/hadoop/yarn/local/usercahe/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs14/hadoop/yarn/local/usercahe/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs14/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs16/hadoop/yarn/local/usercahe/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs16/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs13/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs15/hadoop/yarn/local/usercahe/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs15/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs13/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs16/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,548 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs14/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,549 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs15/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs2/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,549 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs2/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,550 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs2/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs3/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs5/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,551 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs5/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,551 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs5/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,551 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs3/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs3/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs4/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs4/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs7/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs7/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs6/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs6/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs7/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs8/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs8/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,552 INFO
containermanager.AuxServices (AuxServices.javahandle(215)) - Got event
APPLICATION_STOP for appId application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs8/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir :
/hdfs9/hadoop/yarn/local/usercace/druid/appcache/application_1521457397747_0002
baseDir : null
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 DEBUG
nodemanager.DeletionService (DeletionService.jva:run(292)) - Deleting path:
[/hdfs9/hadoop/yarn/local/usercache/druid/appcache/application_1521457397747_0002]
as user: [druid]
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,553 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs6/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,554 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(492)) - Deleting absolute path :
/hdfs9/hadoop/yarn/local/usercache/druid/appcache/application_152145739747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,556 INFO
yarn.YarnShuffleService (YarnShuffleService.jaa:stopApplication(179)) -
Stopping application application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,556 INFO
shuffle.ExternalShuffleBlockResolver
(ExternalhuffleBlockResolver.java:applicationRemoved(206)) - Application
application_1521457397747_0002 removed, cleanupLocalDirs = false
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,560 INFO
shuffle.ExternalShuffleBlockResolver
(ExternalhuffleBlockResolver.java:applicationRemoved(186)) - Application
application_1521457397747_0002 removed, cleanupLocalDirs = false
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,560 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_RESOURCES_CLEANEDUP
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,560 DEBUG
security.NMTokenSecretManagerInNM
(NMTokenSecrtManagerInNM.java:appFinished(191)) - Removing application attempts
NMToken keys for application application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,561 INFO
application.ApplicationImpl (ApplicationImpl.jva:handle(464)) - Application
application_1521457397747_0002 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,561 INFO
logaggregation.AppLogAggregatorImpl
(AppLogAggegatorImpl.java:finishLogAggregation(413)) - Application just
finished : application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,589 DEBUG
hdfs.DFSClient (DFSOutputStream.java:computePaketChunkSize(1924)) -
computePacketChunkSize:
src=/app-logs/druid/logs-ifile/application_1521457397747_0002/n2pl-pa-xxxxxx45454,
chunkSize=516, chunksPerPacket=126, packetSize=65016
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,614 DEBUG
hdfs.DFSClient (DFSOutputStream.java:writeChunImpl(2019)) - DFSClient
writeChunk allocating new packet seqno=0,
src=/app-logs/druid/logs-ifile/application_1521457397747_0002/n2pl-pa-hdd30.xxxx.xx_45454,
packetSize=65016, chunksPerPacket=126, bytesCurBlock=0
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:FileDeletionTask : user : druid
subDir : null baseDir :
/hdfs1/hadoop/yan/log/application_1521457397747_0002,/hdfs10/hadoop/yarn/log/application_1521457397747_0002,/hdfs11/hadoop/yarn/log/application_152145739777_0002,/hdfs12/hadoop/yarn/log/application_1521457397747_0002,/hdfs13/hadoop/yarn/log/application_1521457397747_0002,/hdfs14/hadoop/yarn/lo/application_1521457397747_0002,/hdfs15/hadoop/yarn/log/application_1521457397747_0002,/hdfs16/hadoop/yarn/log/application_1521457397747_002,/hdfs2/hadoop/yarn/log/application_1521457397747_0002,/hdfs3/hadoop/yarn/log/application_1521457397747_0002,/hdfs4/hadoop/yarn/log/appliction_1521457397747_0002,/hdfs5/hadoop/yarn/log/application_1521457397747_0002,/hdfs6/hadoop/yarn/log/application_1521457397747_0002,/hdfs7/adoop/yarn/log/application_1521457397747_0002,/hdfs8/hadoop/yarn/log/application_1521457397747_0002,/hdfs9/hadoop/yarn/log/application_152157397747_0002,
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,706 DEBUG
application.ApplicationImpl (ApplicationImpl.jva:handle(453)) - Processing
application_1521457397747_0002 of type APPLICATION_LOG_HANDLING_FINISHED
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,706 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs1/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs10/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs11/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs12/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs13/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs14/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs15/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs16/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,707 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs2/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs3/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs4/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs5/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs6/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs7/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs8/hadoop/yarn/log/application_1521457397747_0002
yarn/yarn-yarn-nodemanager-n2pl-pa-xxxxxx.log:2018-03-19 16:39:25,708 INFO
nodemanager.DefaultContainerExecutor
(DefaultCntainerExecutor.java:deleteAsUser(501)) - Deleting path :
/hdfs9/hadoop/yarn/log/application_1521457397747_0002
yarn/nm-audit.log:2018-03-19 16:39:23,726 INFO nodemanager.NMAuditLogger:
USER=druid IP=10.5.233.220 OPERATION=Start Container Request
ARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1521457397747_0002
CONTAINERID=container_e11_1521457397747_0002_01_00001
yarn/nm-audit.log:2018-03-19 16:39:24,519 WARN nodemanager.NMAuditLogger:
USER=druid OPERATION=Container Finished - Failed TARGET=ContinerImpl
RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED
APPID=application_1521457397747_0002
CONAINERID=container_e11_1521457397747_0002_01_000001
Thanks, Nayan |
