[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534581#comment-14534581 ] Kihwal Lee commented on HDFS-6888: -- bq. One of the key points of the HDFS audit log is to show accesses to files, including for security purposes. If a user can legitimately use getFileInfo(), then it needs to get logged. The patch allows users to exclude certain ops from audit log. {{getFileIno}} needs to be removed from {{hdfs-default.xml}}, so that it is logged by default. I will update the title. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: BB2015-05-TBR, log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, > HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512878#comment-14512878 ] Hadoop QA commented on HDFS-6888: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 35s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 3 new or modified test files. | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | javac | 7m 28s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 7m 54s | The applied patch generated 2 additional checkstyle issues. | | {color:green}+1{color} | install | 1m 33s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 3m 3s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. | | {color:green}+1{color} | native | 3m 11s | Pre-build of native portion | | {color:red}-1{color} | hdfs tests | 163m 59s | Tests failed in hadoop-hdfs. | | | | 212m 16s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12676436/HDFS-6888-6.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / a00e001 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/10395/artifact/patchprocess/checkstyle-result-diff.txt | | hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/10395/artifact/patchprocess/testrun_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/10395/testReport/ | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/10395/console | This message was automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, > HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512817#comment-14512817 ] Allen Wittenauer commented on HDFS-6888: One of the key points of the HDFS audit log is to show accesses to files, including for security purposes. If a user can legitimately use getFileInfo(), then it needs to get logged. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, > HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180823#comment-14180823 ] Hadoop QA commented on HDFS-6888: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676436/HDFS-6888-6.patch against trunk revision a36399e. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8483//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8483//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, > HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157037#comment-14157037 ] Hadoop QA commented on HDFS-6888: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12672571/HDFS-6888-5.patch against trunk revision c2fa5d1. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8299//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8299//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8299//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, > HDFS-6888-5.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156658#comment-14156658 ] Eric Payne commented on HDFS-6888: -- [~airbots], thank you for fixing this issue. +1 (non-binding) on the patch. I have one very tiny nit and one question: - In the description of {{hdfs-default.xml}}, I would have used the phrase "writes to" instead of "outputs" - Is there someplace that documents all of the configuration properties? Is that generated automatically from the descriptions in the *xml files? > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, > HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145683#comment-14145683 ] Hadoop QA commented on HDFS-6888: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670807/HDFS-6888-4.patch against trunk revision f48686a. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8171//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8171//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, > HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132981#comment-14132981 ] Gera Shegalov commented on HDFS-6888: - Thanks, [~airbots] for updating the patch. +1 (non-binding) Two nits: debugCmdSet can be declared as a general Set {code} private Set debugCmdSet = new HashSet(); {code} Its initialization can be more brief: {code} debugCmdSet.addAll(Arrays.asList(conf.getTrimmedStrings( DFSConfigKeys.DFS_AUDIT_LOG_DEBUG_CMDLIST))); {code} > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129431#comment-14129431 ] Chen He commented on HDFS-6888: --- unit test failure is because of HDFS-6694 > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129417#comment-14129417 ] Hadoop QA commented on HDFS-6888: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667859/HDFS-6888-3.patch against trunk revision b02a4b4. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7988//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7988//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114200#comment-14114200 ] Chen He commented on HDFS-6888: --- Test error is because of HADOOP-11020 > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-3.patch, > HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114141#comment-14114141 ] Hadoop QA commented on HDFS-6888: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664998/HDFS-6888-3.patch against trunk revision c4c9a78. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.security.TestRefreshUserMappings The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDecommission {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7829//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7829//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-3.patch, > HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113797#comment-14113797 ] Chen He commented on HDFS-6888: --- HEAD is now at c4c9a78 HADOOP-11001. Fix test-patch to work with the git repo. (kasha) Removing patchprocess/ Switched to branch 'trunk' Your branch is up-to-date with 'origin/trunk'. Current branch trunk is up to date. /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory /bin/grep: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory /bin/grep: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory HDFS-6888 patch is being downloaded at Thu Aug 28 14:11:35 UTC 2014 from http://issues.apache.org /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 44: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 100: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 102: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory The patch does not appear to apply with p0 to p2 rm: cannot remove '/tmp/tmp.paths.7370': No such file or directory PATCH APPLICATION FAILED > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113795#comment-14113795 ] Chen He commented on HDFS-6888: --- It works on my mac. Is this because Hadoop repository move to git? > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113786#comment-14113786 ] Hadoop QA commented on HDFS-6888: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org against trunk revision c4c9a78. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7820//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107394#comment-14107394 ] Gera Shegalov commented on HDFS-6888: - Hi [~airbots], sorry for being unclear. [~kihwal] suggests: bq. We could have logAuditEvent() check cmd against "getfileinfo" or a *collection of such commands* and log at debug level. Picking this idea up can you introduce some conf like dfs.audit..cmdlist=getfileinfo,anotherLogFloodingCmd,... In {{o.a.h.hdfs.server.namenode.FSNamesystem.DefaultAuditLogger#initialize}} you could read the list using {{auditDebugCmds conf.getTrimmedStrings("dfs.audit.debug.cmdlist"}} and use it for filtering. Currenly v2 hardcodes "getfileinfo". {code} --- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java +++ hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java @@ -359,6 +359,9 @@ private void logAuditEvent(boolean succeeded, UserGroupInformation ugi, InetAddress addr, String cmd, String src, String dst, HdfsFileStatus stat) { FileStatus status = null; +if(cmd.equals("getfileinfo") && !auditLog.isDebugEnabled()) { + return; +} if (stat != null) { Path symlink = stat.isSymlink() ? new Path(stat.getSymlink()) : null; Path path = dst != null ? new Path(dst) : new Path(src); {code} Also auditLog.isDebugEnabled() is a cheaper check, and should be done before {{equals}} > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107249#comment-14107249 ] Chen He commented on HDFS-6888: --- Hi [~jira.shegalov], you lost me there. We need to let the getfileinfo cmd only log when the auditlog is in debug level, right? > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106362#comment-14106362 ] Gera Shegalov commented on HDFS-6888: - [~kihwal], I am +1 for making some commands debug level, so we have an option to capture them in the logs and remove them. [~airbots], how about making the list of DEBUG-level commands configurable via a csv list for conf.getTrimmedStrings instead of hardcoding it as in v2 of the patch. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106334#comment-14106334 ] Hadoop QA commented on HDFS-6888: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663508/HDFS-6888-2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 3 release audit warnings. {color:red}-1 core tests{color}. The test build failed in hadoop-hdfs-project/hadoop-hdfs {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7710//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7710//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7710//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106295#comment-14106295 ] Hadoop QA commented on HDFS-6888: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663508/HDFS-6888-2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 3 release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.namenode.TestAuditLogger org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.TestDecommission {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7703//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7703//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7703//console This message is automatically generated. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105722#comment-14105722 ] Kihwal Lee commented on HDFS-6888: -- We could have {{logAuditEvent()}} check {{cmd}} against "getfileinfo" or a collection of such commands and log at debug level. Otherwise we will have to change HdfsAuditLogger, which will introduce incompatibility. Users who want to see this in the log can change the log level at runtime. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105698#comment-14105698 ] Kihwal Lee commented on HDFS-6888: -- What if we log it at debug level? > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105623#comment-14105623 ] Gera Shegalov commented on HDFS-6888: - getfileinfo a common RPC but we have identified many times using the audit log an app that needs some "help" in optimizing their FileSystem API usage. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105604#comment-14105604 ] Kihwal Lee commented on HDFS-6888: -- See the patch in HDFS-3733. When this logging was added, a number of test cases were added. These test cases need to be modified. "open" (i.e. getBlockLocations()) is logged for webhdfs. For example, {{testAuditWebHdfsOpen()} generates 3 audit log entries: getfileinfo/webhdfs, open/webhdfs and open/rpc. After this change, it will have the last two entries. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)