[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497039#comment-14497039 ] Jakob Homan commented on HDFS-8155: --- After HDFS-8154, it will be much easier for other backends than Hadoop to offer access via the WebHDFS specification. In this environment, it would be good to support more types of authentication, even if Hadoop itself does not immediately support it. OAuth2 would be a good candidate. We should amend the WebHDFS spec to support OAuth tokens, specifically by providing either bearer/refresh tokens in the config ([RFC 4.1|https://tools.ietf.org/html/rfc6749#section-4.1], with the allowance that the tokens have already been obtained to obviate the need for user interaction), or via a credential that can be exchanged for those tokens ([RFC 4.3|https://tools.ietf.org/html/rfc6749#section-4.3]). This would allow a WebHDFS backed to support either OAuth2 or SPENGO. WebHDFS backends (including Hadoop) would only be expected to support one type of authentication per system and would be able to reject calls made using another type. Under this proposal, post HDFS-8154, the WebHDFSFileSystem will need to be updated to support presenting OAuth credentials, but it is not necessary to modify the Namenode or Datanodes to accept them. That can be done as part of HADOOP-11744. > Support OAuth2 authentication in WebHDFS > > > Key: HDFS-8155 > URL: https://issues.apache.org/jira/browse/HDFS-8155 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Reporter: Jakob Homan > > WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497105#comment-14497105 ] Kai Zheng commented on HDFS-8155: - Hello [~jakobhoman], Thanks for having this and the good thought. We're working on HADOOP-11817, where both JWT token and OAuth2 token are to be supported for Hadoop web thru a generic token representation and API by pluggable approach. We use [CloudFoundry|https://github.com/cloudfoundry/uaa] for the OAuth2 test. We'll post our initial patch in this week and I hope our work can meet with your need. We would be glad to help with the web HDFS case, would you mind our side working on this issue as well? We would definitely welcome your thoughts, ideas and reviews, considering your concrete OAuth2 token provider and cases. Thanks. > Support OAuth2 authentication in WebHDFS > > > Key: HDFS-8155 > URL: https://issues.apache.org/jira/browse/HDFS-8155 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Reporter: Jakob Homan > > WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497688#comment-14497688 ] Kai Zheng commented on HDFS-8155: - Thanks a lot for reporting the JIRA. I have plans for working on OAuth2 and implementing the WebHDFS case, so I am taking this JIRA. Initial patches and design draft will be uploaded to HADOOP-11766 in this week or early next week, please help review and comment then. Thanks. > Support OAuth2 authentication in WebHDFS > > > Key: HDFS-8155 > URL: https://issues.apache.org/jira/browse/HDFS-8155 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Reporter: Jakob Homan > > WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497700#comment-14497700 ] Kai Zheng commented on HDFS-8155: - [~jghoman], I noticed this issue was linked to HDFS-8154 as depended. Would you provide your rational? I thought the OAuth2 support for Web HDFS can be done separately like we would do for Hadoop Web UI, or you mean more than that? Thanks. > Support OAuth2 authentication in WebHDFS > > > Key: HDFS-8155 > URL: https://issues.apache.org/jira/browse/HDFS-8155 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Reporter: Jakob Homan >Assignee: Kai Zheng > > WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498281#comment-14498281 ] Jakob Homan commented on HDFS-8155: --- Hey Kai- This JIRA is part of the larger effort of 8154 to make the WebHDFS REST specification more general and accessible to other clients and back-end implementations. It will likely build on your work to add OAuth2 throughout the system. Effectively, this JIRA is for two items: a) add OAuth2 as a possible [authentication method|https://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Authentication] (along with SPENGO, simple and delegation tokens) and b) add support in the WebHDFSFileSystem for passing OAuth tokens (or obtaining those tokens via configuration-supplied credentials or user/name password) to the WebHDFS backend. I'm interested in the client and non-Namenode WebHDFS backends, while you're focusing on the Namenode and other current components. I would like to get the change to the WebHDFS spec and support on the client in soon. Happy to use your code, or to commit it if it's ready. > Support OAuth2 authentication in WebHDFS > > > Key: HDFS-8155 > URL: https://issues.apache.org/jira/browse/HDFS-8155 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Reporter: Jakob Homan >Assignee: Kai Zheng > > WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498444#comment-14498444
]
Haohui Mai commented on HDFS-8155:
--
I think that there are two use cases here:
* Using WebHDFS in UI
* Using WebHDFS programmatically (e.g., through {{WebHdfsFileSystem}})
For the first use case -- WebHDFS now recognizes the auth cookie of the UI
therefore the UI works as long as any third-party filter behaves correctly
w.r.t. the UI pages.
For the second use case -- WebHDFS is designed to use DT as the authentication
method. To authenticate, the third-party filter (OAuth2 filter included) should
control when to issue a DT when getting the {{GETDELEGATIONTOKEN}} call. The DT
needs to be presented to the server in all subsequent usages.
I don't think injecting any third-party payload (e.g., OAuth tokens) into
WebHdfsFileSystem make sense.
> Support OAuth2 authentication in WebHDFS
>
>
> Key: HDFS-8155
> URL: https://issues.apache.org/jira/browse/HDFS-8155
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: webhdfs
>Reporter: Jakob Homan
>Assignee: Kai Zheng
>
> WebHDFS should be able to accept OAuth2 credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499092#comment-14499092
]
Jakob Homan commented on HDFS-8155:
---
bq. For the first use case – WebHDFS now recognizes the auth cookie of the UI
therefore the UI works as long as any third-party filter behaves correctly
w.r.t. the UI pages.
I agree. I'm not considering UI right now.
bq. For the second use case – WebHDFS is designed to use DT as the
authentication method.
WebHDFS supports [three distinct types of
authentication|https://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Authentication]:
SPENGO, simple, delegation token.
Please consider JIRA in light of the linked JIRA, HDFS-8154, which is going to
extract WebHDFS as a separate interface that other backing stores will support.
Currently the only way for some backing store to gain access to the Hadoop
ecosystem is to implement oah.FileSystem, which would give it access to JVM
based frameworks (Pig, Hive, Spark, etc.). Additionally, such a store may wish
to expose a REST interface to itself or provide easy access to non-JVM systems.
Such a system could go about defining a REST specification into the
oah.FileSystem, but that definition would look exactly (or pretty much) like
what WebHDFS already defines. Instead of such duplication, HDFS-8154 looks to
make what we already have (WebHDFS) more general and useful. As part of that,
we need to add support for a more widely used authorization system, OAuth2.
An important point is that
[WebHDFS|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L91]
is misnamed:
{code:title=WebHDFSFileSystem.java}
public class WebHdfsFileSystem extends FileSystem
implements DelegationTokenRenewer.Renewable,
TokenAspect.TokenManagementDelegator {
{code}
WebHDFS extends FileSystem, not DistributedFileSystem and so should properly be
called WebFileSystem. As such, the general purpose methods that it implements
(and its REST endpoints expose) are suitable for implementation for lots of
backing stores. HDFS-8154 and this JIRA are about making that extensibility
explicit and easy.
bq. To authenticate, the third-party filter (OAuth2 filter included) should
control when to issue a DT when getting the GETDELEGATIONTOKEN call. The DT
needs to be presented to the server in all subsequent usages.
Not all file systems issue delegation tokens, so it should not be a requirement
for WebHDFS-backed systems to either. Instead, OAuth2 credentials (generic
credentials per RFC spec section 4.3, explicit bearer/refresh tokens, or even
maybe plaintext password/usernames) should be able to be provided and passed
into whatever framework is actually handling the negotiation (ie, the filters).
bq. I don't think injecting any third-party payload (e.g., OAuth tokens) into
WebHdfsFileSystem make sense.
SPNEGO is already a third-party payload. This JIRA only adds OAuth as another
option.
> Support OAuth2 authentication in WebHDFS
>
>
> Key: HDFS-8155
> URL: https://issues.apache.org/jira/browse/HDFS-8155
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: webhdfs
>Reporter: Jakob Homan
>Assignee: Kai Zheng
>
> WebHDFS should be able to accept OAuth2 credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 authentication in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500828#comment-14500828
]
Kai Zheng commented on HDFS-8155:
-
Hi [~jghoman],
bq.We should amend the WebHDFS spec to support OAuth tokens, specifically by
providing either bearer/refresh tokens in the config (RFC 4.1, with the
allowance that the tokens have already been obtained to obviate the need for
user interaction), or via a credential that can be exchanged for those tokens
(RFC 4.3).
I understand you're externalizing and defining a generic WebHDFS interface and
spec for more backend stores other than just HDFS. That looks great to me. As
you may use Swagger or RAML to define the REST interface and generate the spec
doc accordingly, I'm not yet sure if we need to couple with OAuth2 stuff with
it, or how tightly if we have to. We have already support Simple, SPNEGO and
DT, how they existing methods would be defined in your spec? I would take a
look. In HADOOP-11766 we're working on a generic token support for Hadoop,
based on it and a general token representation or API {{AuthToken}} would have
the OAuth2 token support. We're going this way because there may different
OAuth2 token providers and corresponding specifics. Simply saying, in places in
Hadoop codes that uses a token (say OAuth2 token), it would use the AuthToken
type, and the real AuthToken implementations, corresponding token decoders and
validators are pluggable and configurable. In this way it would be possible to
support more tokens (like JWT token), more OAuth2 providers, avoiding to change
the basic thing. How would you think this approach? I thought we should avoid
coupling with OAuth2 credentials tightly. Will it work in your side to add
another method like {{TokenAuth}} for the general token support in your case
and plugin the OAuth2 specific things? We're working on the design and maybe
you could review it then have the confirm. Thanks.
> Support OAuth2 authentication in WebHDFS
>
>
> Key: HDFS-8155
> URL: https://issues.apache.org/jira/browse/HDFS-8155
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: webhdfs
>Reporter: Jakob Homan
>Assignee: Kai Zheng
>
> WebHDFS should be able to accept OAuth2 credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
