[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-8155: --- Fix Version/s: (was: 2.8) 2.8.0 > Support OAuth2 in WebHDFS > - > > Key: HDFS-8155 > URL: https://issues.apache.org/jira/browse/HDFS-8155 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 2.8.0 > > Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, > HDFS-8155.003.patch, HDFS-8155.004.patch, HDFS-8155.005.patch, > HDFS-8155.006.patch > > > WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8 Status: Resolved (was: Patch Available) The test failures, numerous though they are, are unrelated and happening to other JIRAS as well (HDFS-8983). Having received the rare double-Chris(D|N) +1, I've committed this to trunk and branch-2. Resolving. Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 2.8 Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch, HDFS-8155.005.patch, HDFS-8155.006.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Attachment: HDFS-8155.006.patch Fixed ChrisN's final coment. Yeah, Jenkins is being weird for me. I ran through all the HDFS tests manually and except for a couple non-repeatable, unrelated failures, everything passed. I'll let Jenkins run again, but unless it's something real, I'll go ahead and commit this later today. Thanks. Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch, HDFS-8155.005.patch, HDFS-8155.006.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Open (was: Patch Available) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch, HDFS-8155.005.patch, HDFS-8155.006.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Patch Available (was: Open) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch, HDFS-8155.005.patch, HDFS-8155.006.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Attachment: HDFS-8155.003.patch New patch addressing checkstyle and ChrisN's review. bq. ConfRefreshTokenBasedAccessTokenProvider, ConfRefreshTokenBasedAccessTokenProvider: There are no timeouts specified in the calls to the refresh URL. Timeouts can be controlled by calling client.setConnectTimeout and client.setReadTimeout. Done. bq. AccessTokenProvider: Optional - consider extending Configured so that it inherits the implementations of getConf and setConf for free. Configured sets the conf as part of the constructor, which breaks the way ATP implementations sets its values. I kept it as Configurable just to avoid code churn. bq. WebHDFS.md: Typo: toekns instead of tokens Done. bq. Please address the javac and checkstyle warnings. Done. my local test patch is happy (at last) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Patch Available (was: Open) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Open (was: Patch Available) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Patch Available (was: Open) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Attachment: HDFS-8155.004.patch Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Open (was: Patch Available) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Open (was: Patch Available) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Patch Available (was: Open) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch, HDFS-8155.005.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Attachment: HDFS-8155.005.patch Fixed ChrisD's comment. Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch, HDFS-8155.003.patch, HDFS-8155.004.patch, HDFS-8155.005.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Attachment: HDFS-8155.002.patch Revised patch based on ChrisD's comments. Still applies to trunk and branch-2. bq. Instead of adding its own Time classes under utils, it may make sense to move Clock from YARN to Common. Or use the Timer class in Common, if that's sufficient. Done. The Clock in YARN is public stable so can't be moved. There are about six other clocks or timers floating around. I picked the most useful and public in common. bq. If the timing classes could use existing code, the Utils class could be package-private in o.a.h.hdfs.web.oauth2 Done. They did and disappeared. I ended up moving the AccessTokenTimer out of utils package, which caused that package to vanish in a puff of smoke. Utils is now pp. bq. Please add scope/visibility annotations to classes Done. bq. The protected, mutable fields in the AccessTokenProvider classes can be private. nextRefreshMSSinceEpoch in Timer, also (constructor should explicitly init to 0 instead of self-ref) Done. bq. Instead of requiring initialize, would it make sense for AccessTokenProvider implementations to implement Configurable as appropriate? Done. There's quite a lot going on in setConf, but oh well. bq. What is the expectation for multithreaded access for these classes? Individual implementations should do the appropriate needful. For the two provided ones I've synchronized on the refresh method, which should work. bq. If AccessTokenProvider were an abstract class, could the impls share more of the code in refresh? Superficially, they look very similar... I've gone back and forth on how much to share. I'd like to leave them a bit separate for now and see how many other implementations are provided and common they are. Since the code is public/evolving, it will work to update things as necessary. As part of the Configurable change, the class did became abstract rather than an interface, which will make changes in the future easier. bq. Should refresh throw IOException instead of IllegalArgumentException, since AccessTokenProvider::getAccessToken supports it? Done. Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Status: Patch Available (was: Open) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-8155: -- Attachment: HDFS-8155-1.patch First patch for review. We've been testing a version of this code for a few months and it's working well. Two types of OAuth code grants (client credentials and refresh/access tokens provided by the conf) are supported by default and other code grants are user implementable. I had planned on using Apache Oltu for this, but that project doesn't seem very active and it's main benefit - special case support for oauth2 providers like github/twitter/fb, etc. - is of marginal benefit for WebHDFS and could easily be implemented by the user if necessary. I didn't end up using the Authenticator client class because it's too closely tied to the spnego implementation, but after this goes in it will be a good idea to make that class more generic and use it for the oauth stuff as well. Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-8155-1.patch WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8155) Support OAuth2 in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-8155: Summary: Support OAuth2 in WebHDFS (was: Support OAuth2 authentication in WebHDFS) Support OAuth2 in WebHDFS - Key: HDFS-8155 URL: https://issues.apache.org/jira/browse/HDFS-8155 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Reporter: Jakob Homan Assignee: Kai Zheng WebHDFS should be able to accept OAuth2 credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
