[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[ https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136582#comment-15136582 ] Vinayakumar B commented on HDFS-5730: - bq. Vinayakumar B, May we close this JIRA and continue the proposed work as part of HDFS-9395? Hi [~umamahesh], Are you okay to close this? > Inconsistent Audit logging for HDFS APIs > > > Key: HDFS-5730 > URL: https://issues.apache.org/jira/browse/HDFS-5730 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Labels: BB2015-05-TBR > Attachments: HDFS-5730.patch, HDFS-5730.patch > > > When looking at the audit loggs in HDFS, I am seeing some inconsistencies > what was logged with audit and what is added recently. > For more details please check the comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135406#comment-15135406
]
Hadoop QA commented on HDFS-5730:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HDFS-5730 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12623677/HDFS-5730.patch |
| JIRA Issue | HDFS-5730 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14416/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Labels: BB2015-05-TBR
> Attachments: HDFS-5730.patch, HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[ https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135362#comment-15135362 ] Kuhu Shukla commented on HDFS-5730: --- [~vinayrpet], May we close this JIRA and continue the proposed work as part of HDFS-9395? > Inconsistent Audit logging for HDFS APIs > > > Key: HDFS-5730 > URL: https://issues.apache.org/jira/browse/HDFS-5730 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Labels: BB2015-05-TBR > Attachments: HDFS-5730.patch, HDFS-5730.patch > > > When looking at the audit loggs in HDFS, I am seeing some inconsistencies > what was logged with audit and what is added recently. > For more details please check the comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524891#comment-14524891
]
Hadoop QA commented on HDFS-5730:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12623677/HDFS-5730.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10646/console |
This message was automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-5730.patch, HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524865#comment-14524865
]
Hadoop QA commented on HDFS-5730:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12623677/HDFS-5730.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10635/console |
This message was automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-5730.patch, HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072115#comment-14072115
]
Hadoop QA commented on HDFS-5730:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623677/HDFS-5730.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7443//console
This message is automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-5730.patch, HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[ https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13884103#comment-13884103 ] Uma Maheswara Rao G commented on HDFS-5730: --- Thanks a lot, Colin for taking a look. More reviews are welcomed. > Inconsistent Audit logging for HDFS APIs > > > Key: HDFS-5730 > URL: https://issues.apache.org/jira/browse/HDFS-5730 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-5730.patch, HDFS-5730.patch > > > When looking at the audit loggs in HDFS, I am seeing some inconsistencies > what was logged with audit and what is added recently. > For more details please check the comments. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[ https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13883669#comment-13883669 ] Colin Patrick McCabe commented on HDFS-5730: Does anyone have a strong opinion about this approach? If not, I will review this in detail later this week > Inconsistent Audit logging for HDFS APIs > > > Key: HDFS-5730 > URL: https://issues.apache.org/jira/browse/HDFS-5730 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-5730.patch, HDFS-5730.patch > > > When looking at the audit loggs in HDFS, I am seeing some inconsistencies > what was logged with audit and what is added recently. > For more details please check the comments. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875070#comment-13875070
]
Hadoop QA commented on HDFS-5730:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623677/HDFS-5730.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5912//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5912//console
This message is automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-5730.patch, HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873994#comment-13873994
]
Hadoop QA commented on HDFS-5730:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623446/HDFS-5730.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.fs.TestResolveHdfsSymlink
org.apache.hadoop.fs.TestSymlinkHdfsDisable
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5901//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5901//console
This message is automatically generated.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-5730.patch
>
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873109#comment-13873109
]
Uma Maheswara Rao G commented on HDFS-5730:
---
Also there are many redundant checks. We need to remove them. This check
already done in LogAuditEvent method
{code}
if (auditLog.isInfoEnabled() && isExternalInvocation()) {
...
..
private void logAuditEvent(boolean success, String cmd, String src,
String dst, HdfsFileStatus stat) throws IOException {
if (isAuditEnabled() && isExternalInvocation()) {
logAuditEvent(allowed, getRemoteUser(), getRemoteIp(),
cmd, src, dst, stat);
}
}
{code}
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[ https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871122#comment-13871122 ] Uma Maheswara Rao G commented on HDFS-5730: --- Thanks Colin, for your feedback on this JIRA. Actually Yes, we should have operation status codes should be logged as one parameter. At the same time indicating Auth failures is also important. I will get a patch shortly which will include one new parameter for logging status codes and suceeded(existing) parameter will be used for Auth failures (if needed I will change parameter name). > Inconsistent Audit logging for HDFS APIs > > > Key: HDFS-5730 > URL: https://issues.apache.org/jira/browse/HDFS-5730 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > When looking at the audit loggs in HDFS, I am seeing some inconsistencies > what was logged with audit and what is added recently. > For more details please check the comments. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[ https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871004#comment-13871004 ] Colin Patrick McCabe commented on HDFS-5730: Thanks for filing this, Uma. I agree that we should make the logs consistent. Personally, I think there is a case to be made for including the failed attempts for all these operations in the audit log. If there are a lot of people making requests that fail because of authorization problems, that could be a red flag that a sysadmin would want to know about. What do you think? > Inconsistent Audit logging for HDFS APIs > > > Key: HDFS-5730 > URL: https://issues.apache.org/jira/browse/HDFS-5730 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > When looking at the audit loggs in HDFS, I am seeing some inconsistencies > what was logged with audit and what is added recently. > For more details please check the comments. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs
[
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13865273#comment-13865273
]
Uma Maheswara Rao G commented on HDFS-5730:
---
HDFS audit logging interface:
{code}
/**
* Same as
* {@link #logAuditEvent(boolean, String, InetAddress, String, String,
String, FileStatus)}
* with additional parameters related to logging delegation token tracking
* IDs.
*
* @param succeeded Whether authorization succeeded.
* @param userName Name of the user executing the request.
* @param addr Remote address of the request.
* @param cmd The requested command.
* @param src Path of affected source file.
* @param dst Path of affected destination file (if any).
* @param stat File information for operations that change the file's metadata
* (permissions, owner, times, etc).
* @param ugi UserGroupInformation of the current user, or null if not logging
* token tracking information
* @param dtSecretManager The token secret manager, or null if not logging
* token tracking information
*/
public abstract void logAuditEvent(boolean succeeded, String userName,
InetAddress addr, String cmd, String src, String dst,
FileStatus stat, UserGroupInformation ugi,
DelegationTokenSecretManager dtSecretManager);
{code}
Here succeeded parameter indicates whether Authorization check succeeded.
Recent APIs like addCacheDirective, modifyCacheDirective,
removeCacheDirective..etc are used that parameter to indicate whether op
succeeded or not.
{code}
boolean success = false;
...
writeLock();
try {
checkOperation(OperationCategory.WRITE);
if (isInSafeMode()) {
throw new SafeModeException(
"Cannot add cache directive", safeMode);
}
cacheManager.modifyDirective(directive, pc, flags);
getEditLog().logModifyCacheDirectiveInfo(directive,
cacheEntry != null);
success = true;
} finally {
writeUnlock();
if (success) {
getEditLog().logSync();
}
if (isAuditEnabled() && isExternalInvocation()) {
logAuditEvent(success, "modifyCacheDirective", null, null, null);
}
RetryCache.setState(cacheEntry, success);
}
{code}
But all the older APIs like startFile..etc handled the AccessControlException
explicitly and passed the first parameter value as false if failure. No log for
other IOE.
Also snapShot related APIs followed other pattern. Here we just logged only on
success.
{code}
String createSnapshot(String snapshotRoot, String snapshotName)
throws SafeModeException, IOException {
..
.
getEditLog().logSync();
if (auditLog.isInfoEnabled() && isExternalInvocation()) {
logAuditEvent(true, "createSnapshot", snapshotRoot, snapshotPath, null);
}
return snapshotPath;
}
{code}
So, we have to unify the audit logging here in all APIs.
> Inconsistent Audit logging for HDFS APIs
>
>
> Key: HDFS-5730
> URL: https://issues.apache.org/jira/browse/HDFS-5730
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> When looking at the audit loggs in HDFS, I am seeing some inconsistencies
> what was logged with audit and what is added recently.
> For more details please check the comments.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
