[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436999#comment-13436999 ] Robert Joseph Evans commented on MAPREDUCE-4296: Will do. I'll remove the 30 min check, I don't think we really need it, but I'll put in a comment with it warning people. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437028#comment-13437028 ] Thomas Graves commented on MAPREDUCE-4296: -- +1 Thanks Bobby! remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437030#comment-13437030 ] Hadoop QA commented on MAPREDUCE-4296: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541413/MR-4296.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2748//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2748//console This message is automatically generated. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436167#comment-13436167 ] Thomas Graves commented on MAPREDUCE-4296: -- Mostly looks good. couple minor things: * we should add new configs to the release notes. * YarnConfiguration - the comment is wrong (+ /** Whether to enable log aggregation */) - should be to enable log aggregation deletion service. * I'm on the wall about the 30 minute check. In one way its nice so users don't shoot themselves but at the same time it makes it harder to test and is less flexible. I think in the minimum we should document that we have the 30 minute config, otherwise perhaps just remove the check and add a comment that you shouldn't set it to low. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435122#comment-13435122 ] Robert Joseph Evans commented on MAPREDUCE-4296: Thanks for the review Jason I am fixing my fat fingered mistakes :) remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435180#comment-13435180 ] Hadoop QA commented on MAPREDUCE-4296: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541061/MR-4296.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2735//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2735//console This message is automatically generated. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435262#comment-13435262 ] Jason Lowe commented on MAPREDUCE-4296: --- Getting close, a few more points: * We're checking the permission of each app log directory, but the permission of the user log directory above them could prevent us from deleting the app directory. * Checking for group write doesn't imply we'd still be able to delete. I'm thinking in particular of the setup issues discussed in MAPREDUCE-4557. Rather than trying to avoid the errant deletes and potentially screwing up those checks, we could just proceed with the delete attempt and not log a giant exception if the error is a permission problem. We could still end up logging a line per application in a misconfigured user log directory, but that's better than logging dozens of lines for a full exception trace that isn't helpful for permission issues anyway. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435279#comment-13435279 ] Robert Joseph Evans commented on MAPREDUCE-4296: Yes I agree, it was ugly when I put that check in, and after thinking about it I saw the check as well. I will update it. The issue with outputting only a small message is with HADOOP-8661, so I will have to do something a bit ugly to make the message small, at least until HADOOP-8661 goes in. If this one goes in first I will probably just undo the uglyness in the that patch. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435326#comment-13435326 ] Hadoop QA commented on MAPREDUCE-4296: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541087/MR-4296.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2737//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2737//console This message is automatically generated. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435367#comment-13435367 ] Jason Lowe commented on MAPREDUCE-4296: --- +1 (non-binding), lgtm. Thanks Bobby! remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[
https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434587#comment-13434587
]
Jason Lowe commented on MAPREDUCE-4296:
---
Patch looks good overall, some cleanup comments:
* Probably need a release note stating the necessary chmod operations on user
directories under the top-level aggregated log directory
* Speaking of which, is it worth checking the permissions of the user directory
before we try to delete all the application logs underneath it? I'm thinking
of all the exceptions that will be logged periodically if one user's log
directory doesn't have the proper permissions to allow deletion but there are
hundreds of application logs underneath it.
* With all the nested try/catch blocks, it might be easier to read if one or
more of the {{for}} loops were factored out into private methods
* Spelling nits (many are end-user visible and should be cleaned up):
** retain is misspelled in yarn.log-aggregation.retian-seconds
** being is misspelled as beign
** desabled because retiontion should be disabled because retention
** aggregated misspelled as aggregatied
remove old aggregated logs
--
Key: MAPREDUCE-4296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.3
Reporter: Thomas Graves
Assignee: Robert Joseph Evans
Attachments: MR-4296.txt
Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never
removed. We should have mechanism to remove them after certain period.
It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434659#comment-13434659 ] Hadoop QA commented on MAPREDUCE-4296: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540952/MR-4296.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2730//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2730//console This message is automatically generated. remove old aggregated logs -- Key: MAPREDUCE-4296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Robert Joseph Evans Attachments: MR-4296.txt Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
