[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3733:
--
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
Target Version/s: (was: 2.2.0-alpha)
Status: Resolved (was: Patch Available)
Test failure is unrelated.
I've committed this. Thanks Andy!
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt,
> hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Isaacson updated HDFS-3733:
Attachment: hdfs-3733-7.txt
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt,
> hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Isaacson updated HDFS-3733:
Attachment: hdfs-3733-6.txt
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt,
> hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-3733:
-
Hadoop Flags: Reviewed
+1
Andy, thanks for the update. The new patch looks good.
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt,
> hdfs-3733-4.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Isaacson updated HDFS-3733:
Attachment: hdfs-3733-4.txt
Attaching latest version of patch.
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt,
> hdfs-3733-4.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Isaacson updated HDFS-3733:
Attachment: hdfs-3733-3.txt
TestHftpDelegationToken is known flaky. TestFsck broke because Fsck uses
getFileInfo under the covers (see NamenodeFsck#fsck) and we now audit log those
accesses correctly.
So I've updated TestFsck's verifyAuditLogs to handle this case correctly. New
patch attached.
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt,
> hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Isaacson updated HDFS-3733:
Attachment: hdfs-3733-2.txt
Sync patch with trunk and revert a gratuitous NL-at-EOF change that might be
messing up patch application.
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Isaacson updated HDFS-3733:
Attachment: hdfs-3733-1.txt
Attaching new patch, removing the need for the ThreadLocal curClient by simply
using NamenodeWebHdfsMethods#REMOTE_ADDRESS. Various other cleanups, and add
test showing that Hftp is also audited.
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS access
[
https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3733:
--
Target Version/s: 2.2.0-alpha (was: 2.1.0-alpha)
Summary: Audit logs should include WebHDFS access (was: audit
logging must cover WebHdfs access)
> Audit logs should include WebHDFS access
>
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Andy Isaacson
>Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS";
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN
> generates the redirect, but do generate audit log entries when the second
> phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location:
> http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v
> 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}}
> thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
