[
https://issues.apache.org/jira/browse/YARN-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563446#comment-16563446
]
Bibin A Chundatt commented on YARN-8418:
----------------------------------------
Thanks [~leftnoteasy] and [~rohithsharma]
*Node manager restart exception*
{code}
2018-07-31 18:18:48,624 ERROR
org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController:
Failed to setup application log directory for application_1533029486266_0004
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (token for mapred: HDFS_DELEGATION_TOKEN owner=mapred/had...@hadoop.com,
renewer=yarn, realUser=, issueDate=1533031437960, maxDate=1533033237960,
sequenceNumber=6, masterKeyId=9) is expired, current time: 2018-07-31
18:20:37,772+0800 expected renewal time: 2018-07-31 18:08:16,749+0800
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1499)
at org.apache.hadoop.ipc.Client.call(Client.java:1445)
at org.apache.hadoop.ipc.Client.call(Client.java:1355)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
{code}
*LogAggregationService enabling*
{code}
2018-07-31 18:18:51,009 ERROR
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: NM node labels
{} were not accepted by RM and message from RM : null
2018-07-31 18:18:51,046 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
LogAggregation enabled for application application_1533029486266_0004
2018-07-31 18:18:52,027 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1533029486266_0004_01_000005 transitioned from SCHEDULED
to KILLING
{code}
Aggregated files for vm2
{code}
Found 4 items
-rw-r----- 2 mapred hadoop 122450 2018-07-31 18:23
/nmremotelog/mapred/logs/application_1533029486266_0004/vm1_64318
-rw-r----- 2 mapred hadoop 19351 2018-07-31 18:23
/nmremotelog/mapred/logs/application_1533029486266_0004/vm2_64318
-rw-r----- 2 mapred hadoop 17929 2018-07-31 18:23
/nmremotelog/mapred/logs/application_1533029486266_0004/vm3_64318
-rw-r----- 2 mapred hadoop 14082 2018-07-31 18:23
/nmremotelog/mapred/logs/application_1533029486266_0004/vm4_64318
{code}
> App local logs could leaked if log aggregation fails to initialize for the app
> ------------------------------------------------------------------------------
>
> Key: YARN-8418
> URL: https://issues.apache.org/jira/browse/YARN-8418
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.8.0, 3.0.0-alpha1
> Reporter: Bibin A Chundatt
> Assignee: Bibin A Chundatt
> Priority: Critical
> Attachments: YARN-8418.001.patch, YARN-8418.002.patch,
> YARN-8418.003.patch, YARN-8418.004.patch, YARN-8418.005.patch,
> YARN-8418.006.patch, YARN-8418.007.patch, YARN-8418.008.patch,
> YARN-8418.009.patch
>
>
> If log aggregation fails init createApp directory container logs could get
> leaked in NM directory
> For log running application restart of NM after token renewal this case is
> possible/ Application submission with invalid delegation token
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
