[jira] [Commented] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206063#comment-16206063
]
Santhosh G Nayak commented on HADOOP-14935:
---
Thanks [~ste...@apache.org] for reviewing and committing the patch.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.1.0
>
> Attachments: HADOOP-14935-003.patch, HADOOP-14935-004.patch,
> HADOOP-14935-005.patch, HADOOP-14935.1.patch, HADOOP-14935.2.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203207#comment-16203207
]
Santhosh G Nayak commented on HADOOP-14935:
---
[~tmarquardt], according to HDFS permissions guide, traversal check is required
for all the operations. But, wasb authorization permission spec does not have
traversal check. The main reason to use READ as a substitute to traversal as a
*compromise* was to have a protection against wrongly diagnosed security issue
with hive. Now that it is clearly understood that the same security issue can
be resolved using the implementation of {{access()}} method in
{{NativeAzureFileSystem}}, authorization check from {{getFileStatus()}} alone
can be removed.
However, it is not understood, if {{getFileStatus()}} is used instead of
{{access()}} in other applications. So, the main motivation to add
{{fs.azure.enable.authorization.getfilestatus}} is to be able to *reintroduce
authorization check for {{getFileStatus()}}* easily, if security related issues
are exposed in other applications in authorization enabled secure cluster
without having to change the code which involves longer turn around time. This
cannot be achieved using {{azure Authorization}} configuration alone, as it is
meant for all the file system operations. I hope this clarifies the confusion.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935-003.patch, HADOOP-14935-004.patch,
> HADOOP-14935.1.patch, HADOOP-14935.2.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202309#comment-16202309
]
Santhosh G Nayak commented on HADOOP-14935:
---
Thanks for reviewing the patch [~steve_l] and [~tmarquardt]. The reason for
adding this new config option to enable/disable authorization feature in
{{getFileStatus()}} is described
[here|https://issues.apache.org/jira/browse/HADOOP-14935?focusedCommentId=16200244=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16200244].
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935-003.patch, HADOOP-14935-004.patch,
> HADOOP-14935.1.patch, HADOOP-14935.2.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201894#comment-16201894
]
Santhosh G Nayak edited comment on HADOOP-14935 at 10/12/17 12:48 PM:
--
Thanks [~steve_l] for reviewing the patch and tuning few tests.
Addressed comments from TODO section in v4 patch,
- Introduced a new test class
{{TestNativeAzureFileSystemAuthorizationGetFileStatus}} which sets
{{fs.azure.enable.authorization.getfilestatus}} to {{true}} and runs the same
authorization test, that way all the code paths are covered.
- Separated the tests for different permission combinations.
- New sticky bit related rename tests may have passed because of the additional
policies configured for {{getFileStatus()}} to work.
Testing: All the tests passed with Azure South India storage endpoint.
was (Author: snayak):
Thanks [~steve_l] for reviewing the patch and tuning few tests.
Addressed comments from TODO section in v4 patch,
- Introduced a new test class
{{TestNativeAzureFileSystemAuthorizationGetFileStatus}} which sets
{{fs.azure.enable.authorization.getfilestatus}} to {{true}} and runs the same
authorization test, that way all the code paths are covered.
- Separated the tests for different permission combinations.
- New sticky bit related rename tests may have passed because of the additional
policies configured for {{getFileStatus()}} to work.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935-003.patch, HADOOP-14935-004.patch,
> HADOOP-14935.1.patch, HADOOP-14935.2.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14935:
--
Attachment: HADOOP-14935-004.patch
Thanks [~steve_l] for reviewing the patch and tuning few tests.
Addressed comments from TODO section in v4 patch,
- Introduced a new test class
{{TestNativeAzureFileSystemAuthorizationGetFileStatus}} which sets
{{fs.azure.enable.authorization.getfilestatus}} to {{true}} and runs the same
authorization test, that way all the code paths are covered.
- Separated the tests for different permission combinations.
- New sticky bit related rename tests may have passed because of the additional
policies configured for {{getFileStatus()}} to work.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935-003.patch, HADOOP-14935-004.patch,
> HADOOP-14935.1.patch, HADOOP-14935.2.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201828#comment-16201828
]
Santhosh G Nayak commented on HADOOP-14935:
---
{quote}
L2665 - Instead of introducing a new config option, please use the
azureAuthorization config option. The code
in the patch on L2669 thru L2679 was added recently in commit 928820
specifically for authorization, so you can use azureAuthorization.
{quote}
{{azureAuthorization}} is added to enable/disable authorization for all the
FileSystem apis, whereas configuration property
{{fs.azure.enable.authorization.getfilestatus}} is introduced to enable/disable
the authorization on {{getFileStatus()}}.
{quote}
L3739 & L3740 - Shouldn't we have WasbAuthorizationOperations.ALL to avoid
multiple network calls for READ and WRITE? I suppose it may be too late for
this change.
{quote}
I agree that it affects the overall latency of {{access()}} operation. However,
we are having this requirement only in {{access()}}, whose usage is not high
compared to other apis. So, I would prefer to keep this as is.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935-003.patch, HADOOP-14935.1.patch,
> HADOOP-14935.2.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14935:
--
Attachment: HADOOP-14935.2.patch
Thanks [~ste...@apache.org] for initial review.
Attaching v2 patch addressing the review comments. I have tested the patch
against {{Azure South India}} storage endpoint.
{quote}
This patch includes a new option for
getFileStatus"fs.azure.enable.authorization.getfilestatus". What is this, why
is it needed, and why isn't it is own patch?
And, given that HADOOP-14845 only added this code last week, how stable is all
of this?
{quote}
The main motivation for HADOOP-14845 is to workaround the security issue (that
anyone can load up any other user's data in hive). Thinking was that {{READ}}
permission can be used as a replacement to traverse {{EXECUTE}} permission at
one level which protects from security issue. It was just a *compromise* as it
is precisely stated in the JIRA.
Recently, it is discovered that hive has been using {{FileSystem.access()}}
method for checking the access permissions instead of directly using
{{getFileStatus()}} on the path. So, implementing
{{NativeAzureFileSyste.access()}} should fix this security issue. Also, using
{{READ}} permission for {{getFileStatus()}} is unintuitive and the number of
policies to configure increases without having any additional benefit.
Ideally, we should remove the authorization check from {{getFileStatus()}}
altogether. But, we do not know, if any applications use {{getFileStatus()}}
instead of {{access()}} method which could lead to security issue (if any). So,
adding {{fs.azure.enable.authorization.getfilestatus}} configuration property
to enable/disable this feature, so that we can quickly fallback to the
compromise proposed in HADOOP-14845.
*NativeAzureFileSystems*
{quote}
javadocs to Access to explictly declare that AccessControlException is raised
on access control, FNFE if the file is not present; same as the superclass.
Maybe just use the \{@inheritDoc\}
tag to do this.
{quote}
Removed the javadoc from the derived {{access()}} method, so that it can
inherit it from the base class method.
{quote}
if it's just formatting changes, let's leave the changes to getFileStatus out:
keeps the merge complexity down.
{quote}
Configuration property {{fs.azure.enable.authorization.getfilestatus}} to
enable/disable the authorization on {{getFileStatus()}} is added.
{{testAccessFileDoesNotExist}} doesn't reset permissions. It does not matter as
{{authorizer}} is initialized in every test.
Added tests for following scenarios :-
- To verify that execute isn't validated in
{{TestNativeAzureFileSystemAuthorization.testAccessWhenPermissionsMatch()}}.
- To verify that permission on the intermediate directory, when a file is
created under a directory which does not exist and owner policy is enabled.
- To verify if FNFE is raised when {{access()}} is called on file having no
permission and does not exist.
- For all the permission combinations.
Fixed checkstyle related issues as well.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935.1.patch, HADOOP-14935.2.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198649#comment-16198649
]
Santhosh G Nayak commented on HADOOP-14935:
---
Tested that patch against {{Azure South India}} storage endpoint.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935.1.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
[
https://issues.apache.org/jira/browse/HADOOP-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14935:
--
Attachment: HADOOP-14935.1.patch
Attaching a patch containing proposed changes.
> Azure: POSIX permissions are taking effect in access() method even when
> authorization is enabled
>
>
> Key: HADOOP-14935
> URL: https://issues.apache.org/jira/browse/HADOOP-14935
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14935.1.patch
>
>
> FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
> not override {{access(path,mode)}} method and uses the default implementation
> from the base class. This base implementaion uses the POSIX permissions to
> check if the requested user has access to given path or not even when
> authorization is enabled, which is incorrect.
> {{NativeAzureFileSystem.access()}} in authorization enabled mode should use
> the authorization mechanism provided instead of relying on the POSIX
> permission ons. So the proposal is to override {{FileSystem.access()}} method
> in {{NativeAzureFileSystem}} such that it honors the authorization mechanism
> configured in authorization enabled mode and falls back to POSIX permissions
> otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14935) Azure: POSIX permissions are taking effect in access() method even when authorization is enabled
Santhosh G Nayak created HADOOP-14935:
-
Summary: Azure: POSIX permissions are taking effect in access()
method even when authorization is enabled
Key: HADOOP-14935
URL: https://issues.apache.org/jira/browse/HADOOP-14935
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 2.9.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
FileSystem implementation class for azure i.e. {{NativeAzureFileSystem}} does
not override {{access(path,mode)}} method and uses the default implementation
from the base class. This base implementaion uses the POSIX permissions to
check if the requested user has access to given path or not even when
authorization is enabled, which is incorrect.
{{NativeAzureFileSystem.access()}} in authorization enabled mode should use the
authorization mechanism provided instead of relying on the POSIX permission
ons. So the proposal is to override {{FileSystem.access()}} method in
{{NativeAzureFileSystem}} such that it honors the authorization mechanism
configured in authorization enabled mode and falls back to POSIX permissions
otherwise.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14640:
--
Attachment: HADOOP-14640.4.patch
Thanks [~jnp].
Attached v4 version of the patch containing findbugs, checkstyle and licence
header fixes.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch, HADOOP-14640.4.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087200#comment-16087200
]
Santhosh G Nayak edited comment on HADOOP-14640 at 7/14/17 11:29 AM:
-
Thanks [~jnp].
I agree that token parsing makes code tightly coupled with the token format,
which could result in parsing errors when token format changes. It is better to
treat token as opaque as you suggested.
Attaching v3 version of the patch containing following changes -
- Token is treated as opaque and client sets the expiry time to be 60 mins.
- If request fails because of the expired or invalid token, client retries the
request after re-fetching the token.
was (Author: snayak):
[~jnp], I agree that token parsing makes code tightly coupled with the token
format, which could result in parsing errors when token format changes. It is
better to treat token as opaque as you suggested.
Attaching v3 version of the patch containing following changes -
- Token is treated as opaque and client sets the expiry time to be 60 mins.
- If request fails because of the expired or invalid token, client retries the
request after re-fetching the token.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14640:
--
Attachment: HADOOP-14640.3.patch
[~jnp], I agree that token parsing makes code tightly coupled with the token
format, which could result in parsing errors when token format changes. It is
better to treat token as opaque as you suggested.
Attaching v3 version of the patch containing following changes -
- Token is treated as opaque and client sets the expiry time to be 60 mins.
- If request fails because of the expired or invalid token, client retries the
request after re-fetching the token.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085909#comment-16085909
]
Santhosh G Nayak edited comment on HADOOP-14640 at 7/13/17 4:12 PM:
Thanks [~jnp] for reviewing the patch.
Yes. {{AuthenticatedURL.Token}} does not provide the expiry time of the token.
So, introduced {{SpnegoToken}} class to represent {{SPNEGO}} token, which
provides information about the validity of the token along with the token
itself.
Token used in {{hadoop.auth}} cookie follows SPNEGO standard and will have
expiry time (represented as {{e}}) in it. So, I think it will be better, if we
can reuse that information, instead of setting different expiry time on our end
and treating it opaque.
Regarding {{expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L}}, adding
5 minutes buffer to proactively get new token to save additional network calls
of requests which can definitely fail because of the token expiry and to
eliminate any issues which can arise because of time difference between
machines (if any).
I have attached another patch addressing the checkstyle related comments and a
bug fix where impersonation context is getting lost in
{{SecureWasbRemoteCallHelper.getHttpRequest()}} as it is being invoked inside
{{ugi.doAs()}}.
Tested with {{Azure South India}} Storage endpoint.
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn clean package
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure (includes =
[dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure
---
[INFO] Compiling 57 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java
uses unchecked or unsafe operations.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-azure ---
[INFO] Compiling 58 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
[INFO] Surefire report directory:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
---
T E S T S
---
---
T E S T S
---
Running
[jira] [Comment Edited] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085909#comment-16085909
]
Santhosh G Nayak edited comment on HADOOP-14640 at 7/13/17 4:11 PM:
Thanks [~jnp] for reviewing the patch.
Yes. {{AuthenticatedURL.Token}} does not provide the expiry time of the token.
So, introduced {{SpnegoToken}} class to represent {{SPNEGO}} token, which
provides information about the validity of the token along with the token
itself.
Token used in {{hadoop.auth}} cookie follows SPNEGO standard and will have
expiry time (represented as {{e}}) in it. So, I think it will be better, if we
can reuse that information, instead of setting different expiry time on our end
and treating it opaque.
Regarding {{expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L}}, adding
5 minutes buffer to proactively get new token to save additional network calls
of requests which can definitely fail because of the token expiry and to
eliminate any issues which can arise because of time difference between
machines (if any).
I have attached another patch addressing the checkstyle related comments and a
bug fix where impersonation context is getting lost in
{{SecureWasbRemoteCallHelper.getHttpRequest()}} as it is being invoked inside
{{ugi.doAs()}}.
Tested against {{Azure South India}} Storage endpoint.
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn clean package
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure (includes =
[dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure
---
[INFO] Compiling 57 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java
uses unchecked or unsafe operations.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-azure ---
[INFO] Compiling 58 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
[INFO] Surefire report directory:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
---
T E S T S
---
---
T E S T S
---
Running
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14640:
--
Attachment: HADOOP-14640.2.patch
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14640:
--
Attachment: (was: HADOOP-14640.2.patch)
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14640:
--
Attachment: HADOOP-14640.2.patch
Thanks [~jnp] for reviewing the patch.
Yes. {{AuthenticatedURL.Token}} does not provide the expiry time of the token.
So, introduced {{SpnegoToken}} class to represent {{SPNEGO}} token, which
provides information about the validity of the token along with the token
itself.
Token used in {{hadoop.auth}} cookie follows SPNEGO standard and will have
expiry time (represented as {{e}}) in it. So, I think it will be better, if we
can reuse that information, instead of setting different expiry time on our end
and treating it opaque.
Regarding {{expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L}}, adding
5 minutes buffer to proactively get new token to save additional network calls
of requests which can definitely fail because of the token expiry and to
eliminate any issues which can arise because of time difference between
machines (if any).
I have attached another patch addressing the checkstyle related comments and a
bug fix where impersonation context is getting lost in
{{SecureWasbRemoteCallHelper.getHttpRequest()}} as it is being invoked inside
{{ugi.doAs()}}.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14640:
--
Attachment: HADOOP-14640.1.patch
Attaching initial version of the patch containing proposed changes.
- Introduced a new configuration {{fs.azure.enable.spnego.token.cache}} to
enable/disable SPNEGO token cache per {{WasbRemoteCallHelper}} object.
- If {{localhost}} URL is specified in the list of configured URLs, converting
that into actual fully qualified actual hostname for Kerberos to work.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
Santhosh G Nayak created HADOOP-14640:
-
Summary: Azure: Support affinity for service running on localhost
and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation
token generation
Key: HADOOP-14640
URL: https://issues.apache.org/jira/browse/HADOOP-14640
Project: Hadoop Common
Issue Type: Improvement
Components: fs/azure
Affects Versions: 2.9.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
separated list of URLs for authorization, SASKey generation and delegation
token generation.
To improve the performance, if service runs on the local machine, give it first
preference over the other configured list of URLs.
Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for every
request by talking to the remote service, before making actual rest requests.
The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14443) Azure: Support retry and client side failover for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443-branch2-2.patch
Thanks [~liuml07], I missed to run it in secure mode. Attached a new patch with
the fix.
> Azure: Support retry and client side failover for authorization, SASKey and
> delegation token generation
> ---
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch, HADOOP-14443.4.patch, HADOOP-14443.5.patch,
> HADOOP-14443.6.patch, HADOOP-14443.7.patch, HADOOP-14443-branch2-1.patch,
> HADOOP-14443-branch2-2.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> - Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14443) Azure: Support retry and client side failover for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443-branch2-1.patch
Thanks [~liuml07] for committing the patch to {{trunk}}. Attached a patch for
{{branch-2}} with {{ObjectMapper}} optimization and resolved conflicts. Could
you please review and commit it?
> Azure: Support retry and client side failover for authorization, SASKey and
> delegation token generation
> ---
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch, HADOOP-14443.4.patch, HADOOP-14443.5.patch,
> HADOOP-14443.6.patch, HADOOP-14443.7.patch, HADOOP-14443-branch2-1.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> - Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070387#comment-16070387
]
Santhosh G Nayak commented on HADOOP-14443:
---
[~liuml07] v7 Patch looks good. Tested against {{Azure Storage South India}}
endpoint. Following is the test report,
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.451 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.47 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.432 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.42 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.524 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.378 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.448 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.589 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.732 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.403 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.731 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.885 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.62 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.685 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.24 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.167 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.588 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running
[jira] [Comment Edited] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070387#comment-16070387
]
Santhosh G Nayak edited comment on HADOOP-14443 at 6/30/17 4:45 PM:
Thanks [~liuml07]. v7 version of the patch looks good. Tested against {{Azure
Storage South India}} endpoint. Following is the test report,
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.451 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.47 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.432 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.42 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.524 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.378 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.448 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.589 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.732 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.403 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.731 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.885 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.62 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.685 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.24 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.167 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.588 sec - in
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443.5.patch
Thanks [~liuml07] for reviewing v4 version of the patch. Attaching v5 version
of the patch with following addressed,
1. Updated {{index.md}} with recent changes.
2. Made changes to reuse {{ObjectMapper}} in {{JsonUtils#parse()}}.
3. Tested it with {{South India}} region of {{Azure storage}} endpoint.
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.508 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.432 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.593 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.402 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.546 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.409 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.417 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.564 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.205 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.659 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.288 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.246 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.164 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.045 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.34 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.019 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.242 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.301 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443.4.patch
Attaching {{HADOOP-14443.4.patch}} to fix {{javac}}, {{Javadoc}} subsystem
related failures.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch, HADOOP-14443.4.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443.3.patch
[~liuml07] Thanks for the comments. I have fixed few of those and attached a
new patch,
1. {{@Override}} annotation to new line.
2. Marked the parameters such as {{op}}, {{renewer}}, {{service}} and
{{token}} as constants?
3. There is a default retry spec which will be used, if user wants to override,
he can do so. If this is not recommended in Hadoop codebase, we can remove the
configuration.
4. Added {{Thread.currentThread().interrupt()}} in
{{WasbRemoteCallHelper::shouldRetry()}} when {{InterruptedException}} occurs.
5. This patch consolidates various http helpers which make common SPNEGO
connection into {{WasbRemoteCallHelper}} and {{SecureWasbRemoteCallHelper}}. It
makes easier to apply the same retry/failover logic to various remote HTTP
calls. I am afraid separating it may be difficult.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14461) Azure: handle failure gracefully in case of missing account access key
[
https://issues.apache.org/jira/browse/HADOOP-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036652#comment-16036652
]
Santhosh G Nayak commented on HADOOP-14461:
---
Thanks [~liuml07], I could not think of better way. May be we can just reduce
the scope of this JIRA to fix the error message in
{{AzureBlobStorageTestAccount}}.
> Azure: handle failure gracefully in case of missing account access key
> --
>
> Key: HADOOP-14461
> URL: https://issues.apache.org/jira/browse/HADOOP-14461
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14461.000.patch, HADOOP-14461.001.patch
>
>
> Currently if the {{fs.azure.account.key.youraccount}} is missing, we will get
> error stack like this:
> {code}
> java.lang.IllegalArgumentException: The String is not a valid Base64-encoded
> string.
> at com.microsoft.azure.storage.core.Base64.decode(Base64.java:63)
> at
> com.microsoft.azure.storage.StorageCredentialsAccountAndKey.(StorageCredentialsAccountAndKey.java:81)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createStorageAccount(AzureBlobStorageTestAccount.java:464)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createTestAccount(AzureBlobStorageTestAccount.java:501)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:522)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:451)
> at
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization.createTestAccount(TestNativeAzureFileSystemAuthorization.java:50)
> at
> org.apache.hadoop.fs.azure.AbstractWasbTestBase.setUp(AbstractWasbTestBase.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
> Actually, this error message is not very helpful. I'd admire you if you can
> immediately find the root cause of this failure.
> Currently if the test *account* is missing, we simply skip the test with
> meaningful error message. Here if the *account key* is missing, we should do
> the same thing: skip the test, and tell the user why we skip it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14461) Azure: handle failure gracefully in case of missing account access key
[
https://issues.apache.org/jira/browse/HADOOP-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032464#comment-16032464
]
Santhosh G Nayak commented on HADOOP-14461:
---
Thanks [~liuml07], this is a good initiative. This will surely help developers
who work on {{fs/azure}}.
Do we need to handle this in
{{org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration()}}
as well?
The only difference I see is that, in case of
{{AzureNativeFileSystemStore.getAccountKeyFromConfiguration()}}, if the storage
account key is not set, it calls
{{AzureNativeFileSystemStore.connectUsingAnonymousCredentials()}} by default,
{code}
// Check whether the account is configured with an account key.
propertyValue = getAccountKeyFromConfiguration(accountName,
sessionConfiguration);
if (propertyValue != null) {
// Account key was found.
// Create the Azure storage session using the account key and container.
connectUsingConnectionStringCredentials(
getAccountFromAuthority(sessionUri),
getContainerFromAuthority(sessionUri), propertyValue);
// Return to caller
return;
}
// The account access is not configured for this cluster. Try anonymous
// access.
connectUsingAnonymousCredentials(sessionUri);
{code}
but, {{anonymous}} access is turned off in
{{AzureBlobStorageTestAccount.createTestAccount()}}.
{code}
static CloudStorageAccount createTestAccount(Configuration conf)
throws URISyntaxException, KeyProviderException {
String testAccountName = conf.get(TEST_ACCOUNT_NAME_PROPERTY_NAME);
if (testAccountName == null) {
System.out
.println("Skipping live Azure test because of missing test account." +
" Please see RunningLiveWasbTests.txt for guidance.");
return null;
}
return createStorageAccount(testAccountName, conf, false);
}
{code}
> Azure: handle failure gracefully in case of missing account access key
> --
>
> Key: HADOOP-14461
> URL: https://issues.apache.org/jira/browse/HADOOP-14461
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14461.000.patch
>
>
> Currently if the {{fs.azure.account.key.youraccount}} is missing, we will get
> error stack like this:
> {code}
> java.lang.IllegalArgumentException: The String is not a valid Base64-encoded
> string.
> at com.microsoft.azure.storage.core.Base64.decode(Base64.java:63)
> at
> com.microsoft.azure.storage.StorageCredentialsAccountAndKey.(StorageCredentialsAccountAndKey.java:81)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createStorageAccount(AzureBlobStorageTestAccount.java:464)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createTestAccount(AzureBlobStorageTestAccount.java:501)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:522)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:451)
> at
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization.createTestAccount(TestNativeAzureFileSystemAuthorization.java:50)
> at
> org.apache.hadoop.fs.azure.AbstractWasbTestBase.setUp(AbstractWasbTestBase.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at
[jira] [Commented] (HADOOP-14460) Azure: update doc for live and contract tests
[
https://issues.apache.org/jira/browse/HADOOP-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032418#comment-16032418
]
Santhosh G Nayak commented on HADOOP-14460:
---
+1, the patch
[HADOOP-14460.002.patch|https://issues.apache.org/jira/secure/attachment/12870668/HADOOP-14460.002.patch]
looks good.
> Azure: update doc for live and contract tests
> -
>
> Key: HADOOP-14460
> URL: https://issues.apache.org/jira/browse/HADOOP-14460
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, fs/azure
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14460.000.patch, HADOOP-14460.001.patch,
> HADOOP-14460.002.patch
>
>
> In {{SimpleKeyProvider}}, we have following code for getting the key:
> {code}
> protected static final String KEY_ACCOUNT_KEY_PREFIX =
> "fs.azure.account.key.";
> ...
> protected String getStorageAccountKeyName(String accountName) {
> return KEY_ACCOUNT_KEY_PREFIX + accountName;
> }
> {code}
> While in documentation {{index.md}}, we have:
> {code}
>
> fs.azure.account.key.youraccount.blob.core.windows.net
> YOUR ACCESS KEY
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Status: Patch Available (was: Open)
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443.2.patch
Thanks [~ste...@apache.org] for reviewing this. Addressed the code review
comments.
This patch is test against {{Azure West US}} endpoint.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14443:
--
Attachment: HADOOP-14443.1.patch
Attaching initial version of the patch containing proposed changes.
> Azure: Add retry and client side failover for authorization, SASKey
> generation and delegation token generation requests to remote service
> -
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14443.1.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14443) Azure: Add retry and client side failover for authorization, SASKey generation and delegation token generation requests to remote service
Santhosh G Nayak created HADOOP-14443:
-
Summary: Azure: Add retry and client side failover for
authorization, SASKey generation and delegation token generation requests to
remote service
Key: HADOOP-14443
URL: https://issues.apache.org/jira/browse/HADOOP-14443
Project: Hadoop Common
Issue Type: Improvement
Components: fs/azure
Affects Versions: 2.9.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
Fix For: 2.9.0, 3.0.0-alpha3
Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
for authorization, SASKey generation and delegation token generation. If for
some reason the service is down, all the requests will fail.
So proposal is to,
- Add support to configure multiple URLs, so that if communication to one URL
fails, client can retry on another instance of the service running on different
node for authorization, SASKey generation and delegation token generation.
- Rename the configurations {{fs.azure.authorization.remote.service.url}} to
{{fs.azure.authorization.remote.service.urls}} and
{{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support the
comma separated list of URLs.
Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
configure the comma separated list of service URLs to get the delegation token.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14361) Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently
[
https://issues.apache.org/jira/browse/HADOOP-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14361:
--
Attachment: HADOOP-14361.1.patch
Currently, {{NativeAzureFileSystem.getDelegationToken()}} method uses
{{DelegationTokenAuthenticatedURL.getDelegationToken()}} to get the delegation
token from the remote delegation token service protected by Kerberos over HTTP
and it is not thread safe.
Attached a patch containing the fix where
{{NatieAzureFileSystem.getDelegationToken()}} method is marked as
{{synchronized}}.
> Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when
> invoked concurrently
>
>
> Key: HADOOP-14361
> URL: https://issues.apache.org/jira/browse/HADOOP-14361
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14361.1.patch
>
>
> Sometimes {{NativeAzureFileSystem.getDelegationToken()}} method fails with
> below exception when invoked concurrently,
> {code}Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Authentication failed, URL:
> http://delegationtokenmanger/?op=GETDELEGATIONTOKEN=rm%2Fhostname%40realm,
> status: 401, message: Authentication required
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:195)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$2.run(NativeAzureFileSystem.java:2993)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$2.run(NativeAzureFileSystem.java:2990)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> ... 29 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14361) Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently
Santhosh G Nayak created HADOOP-14361:
-
Summary: Azure: NativeAzureFileSystem.getDelegationToken() call
fails sometimes when invoked concurrently
Key: HADOOP-14361
URL: https://issues.apache.org/jira/browse/HADOOP-14361
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 2.9.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
Sometimes {{NativeAzureFileSystem.getDelegationToken()}} method fails with
below exception when invoked concurrently,
{code}Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
Authentication failed, URL:
http://delegationtokenmanger/?op=GETDELEGATIONTOKEN=rm%2Fhostname%40realm,
status: 401, message: Authentication required
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:195)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$2.run(NativeAzureFileSystem.java:2993)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$2.run(NativeAzureFileSystem.java:2990)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
... 29 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14351) Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl use Kerberos interactive user cache
[
https://issues.apache.org/jira/browse/HADOOP-14351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984612#comment-15984612
]
Santhosh G Nayak edited comment on HADOOP-14351 at 4/26/17 11:28 AM:
-
Thanks [~liuml07] for reviewing the patch. Attached another patch containing
the checkstyle fixes excluding the fix for the {{FileLength}} related warning.
The below findbugs warning is unrelated to changes done in the patch which was
introduced in [Commit
#2217e2f8ff418b88eac6ad36cafe3a9795a11f40|https://github.com/apache/hadoop/commit/2217e2f8ff418b88eac6ad36cafe3a9795a11f40]
of HADOOP-10809
{quote}
Useless object stored in variable keysToUpdateAsFolder of method
org.apache.hadoop.fs.azure.NativeAzureFileSystem.mkdirs(Path, FsPermission,
boolean)
{quote}
was (Author: snayak):
Attaching a patch containing the checkstyle fixes excluding the fix for the
{{FileLength}} related warning.
The below findbugs warning is unrelated to changes done in the patch which was
introduced in [Commit
#2217e2f8ff418b88eac6ad36cafe3a9795a11f40|https://github.com/apache/hadoop/commit/2217e2f8ff418b88eac6ad36cafe3a9795a11f40]
of HADOOP-10809
{quote}
Useless object stored in variable keysToUpdateAsFolder of method
org.apache.hadoop.fs.azure.NativeAzureFileSystem.mkdirs(Path, FsPermission,
boolean)
{quote}
> Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl use Kerberos
> interactive user cache
> -
>
> Key: HADOOP-14351
> URL: https://issues.apache.org/jira/browse/HADOOP-14351
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14351.1.patch, HADOOP-14351.2.patch
>
>
> Currently, {{RemoteWasbAuthorizerImpl.getRelativeBlobSASUri()}},
> {{RemoteWasbAuthorizerImpl.getContainerSASUri()}} and
> {{RemoteSASKeyGeneratorImpl.authorize()}} use Kerberos interactive user's
> ticket cache if the kerberos credential is not available for
> {{UserGroupInformation.getCurrentUser()}} or
> {{UserGroupInformation.getRealUser()}}.
> It results in usage of interactive user's ticket for impersonation, whenever
> services try to do File System operations as another user, which is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14351) Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl use Kerberos interactive user cache
[
https://issues.apache.org/jira/browse/HADOOP-14351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14351:
--
Attachment: HADOOP-14351.2.patch
Attaching a patch containing the checkstyle fixes excluding the fix for the
{{FileLength}} related warning.
The below findbugs warning is unrelated to changes done in the patch which was
introduced in [Commit
#2217e2f8ff418b88eac6ad36cafe3a9795a11f40|https://github.com/apache/hadoop/commit/2217e2f8ff418b88eac6ad36cafe3a9795a11f40]
of HADOOP-10809
{quote}
Useless object stored in variable keysToUpdateAsFolder of method
org.apache.hadoop.fs.azure.NativeAzureFileSystem.mkdirs(Path, FsPermission,
boolean)
{quote}
> Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl use Kerberos
> interactive user cache
> -
>
> Key: HADOOP-14351
> URL: https://issues.apache.org/jira/browse/HADOOP-14351
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14351.1.patch, HADOOP-14351.2.patch
>
>
> Currently, {{RemoteWasbAuthorizerImpl.getRelativeBlobSASUri()}},
> {{RemoteWasbAuthorizerImpl.getContainerSASUri()}} and
> {{RemoteSASKeyGeneratorImpl.authorize()}} use Kerberos interactive user's
> ticket cache if the kerberos credential is not available for
> {{UserGroupInformation.getCurrentUser()}} or
> {{UserGroupInformation.getRealUser()}}.
> It results in usage of interactive user's ticket for impersonation, whenever
> services try to do File System operations as another user, which is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14351) Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl use Kerberos interactive user cache
[
https://issues.apache.org/jira/browse/HADOOP-14351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14351:
--
Attachment: HADOOP-14351.1.patch
Attaching the patch containing the following changes.
1. Removed the following lines when doing SPENGO connection remote service.
{code}
if (!connectUgi.hasKerberosCredentials()) {
connectUgi = UserGroupInformation.getLoginUser();
}
{code}
2. Getting the delegation token in
{{RemoteSASKeyGeneratorImpl.getRelativeBlobSASUri()}},
{{RemoteSASKeyGeneratorImpl.getContainerSASUri()}} and
{{RemoteWasbAuthorizerImpl.authorize()}} methods as well.
> Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl use Kerberos
> interactive user cache
> -
>
> Key: HADOOP-14351
> URL: https://issues.apache.org/jira/browse/HADOOP-14351
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14351.1.patch
>
>
> Currently, {{RemoteWasbAuthorizerImpl.getRelativeBlobSASUri()}},
> {{RemoteWasbAuthorizerImpl.getContainerSASUri()}} and
> {{RemoteSASKeyGeneratorImpl.authorize()}} use Kerberos interactive user's
> ticket cache if the kerberos credential is not available for
> {{UserGroupInformation.getCurrentUser()}} or
> {{UserGroupInformation.getRealUser()}}.
> It results in usage of interactive user's ticket for impersonation, whenever
> services try to do File System operations as another user, which is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14351) Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl use Kerberos interactive user cache
Santhosh G Nayak created HADOOP-14351:
-
Summary: Azure: RemoteWasbAuthorizerImpl and
RemoteSASKeyGeneratorImpl use Kerberos interactive user cache
Key: HADOOP-14351
URL: https://issues.apache.org/jira/browse/HADOOP-14351
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 2.9.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
Currently, {{RemoteWasbAuthorizerImpl.getRelativeBlobSASUri()}},
{{RemoteWasbAuthorizerImpl.getContainerSASUri()}} and
{{RemoteSASKeyGeneratorImpl.authorize()}} use Kerberos interactive user's
ticket cache if the kerberos credential is not available for
{{UserGroupInformation.getCurrentUser()}} or
{{UserGroupInformation.getRealUser()}}.
It results in usage of interactive user's ticket for impersonation, whenever
services try to do File System operations as another user, which is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14273) Azure: Initialize kerberosSupportEnabled in NativeAzureFileSystem using the value of fs.azure.enable.kerberos.support configuration property
[
https://issues.apache.org/jira/browse/HADOOP-14273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14273:
--
Attachment: HADOOP-14273.1.patch
Attaching the patch containing proposed changes.
> Azure: Initialize kerberosSupportEnabled in NativeAzureFileSystem using the
> value of fs.azure.enable.kerberos.support configuration property
>
>
> Key: HADOOP-14273
> URL: https://issues.apache.org/jira/browse/HADOOP-14273
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14273.1.patch
>
>
> Currently, WASB driver uses the default value {{false}} for
> {{kerberosSupportEnabled}} in
> {{org.apache.hadoop.fs.azure.NativeAzureFileSystem}} which results in
> skipping the logic of fetching the delegation token from the remote service
> in secure mode (introduced in HADOOP-13945), which is incorrect.
> Fix is to initialize {{kerberosSupportEnabled}} instance member with value of
> the configuration property {{fs.azure.enable.kerberos.support}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14273) Azure: Initialize kerberosSupportEnabled in NativeAzureFileSystem using the value of fs.azure.enable.kerberos.support configuration property
Santhosh G Nayak created HADOOP-14273:
-
Summary: Azure: Initialize kerberosSupportEnabled in
NativeAzureFileSystem using the value of fs.azure.enable.kerberos.support
configuration property
Key: HADOOP-14273
URL: https://issues.apache.org/jira/browse/HADOOP-14273
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 2.8.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
Currently, WASB driver uses the default value {{false}} for
{{kerberosSupportEnabled}} in
{{org.apache.hadoop.fs.azure.NativeAzureFileSystem}} which results in skipping
the logic of fetching the delegation token from the remote service in secure
mode (introduced in HADOOP-13945), which is incorrect.
Fix is to initialize {{kerberosSupportEnabled}} instance member with value of
the configuration property {{fs.azure.enable.kerberos.support}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14272) Azure: WasbRemoteCallHelper should use String equals for comparison.
[
https://issues.apache.org/jira/browse/HADOOP-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14272:
--
Attachment: HADOOP-14272.1.patch
Attaching the patch containing proposed changes.
> Azure: WasbRemoteCallHelper should use String equals for comparison.
>
>
> Key: HADOOP-14272
> URL: https://issues.apache.org/jira/browse/HADOOP-14272
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14272.1.patch
>
>
> {{org.apache.hadoop.fs.azure.WasbRemoteCallHelper}} method is using the
> reference comparison to compare 2 strings, which is incorrect.
> {code}
> if (contentTypeHeader == null || contentTypeHeader.getValue() !=
> APPLICATION_JSON)}}
> {code}
> The fix is to use {{!APPLICATION_JSON.equals(contentTypeHeader.getValue())}}
> instead.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14272) Azure: WasbRemoteCallHelper should use String equals for comparison.
Santhosh G Nayak created HADOOP-14272:
-
Summary: Azure: WasbRemoteCallHelper should use String equals for
comparison.
Key: HADOOP-14272
URL: https://issues.apache.org/jira/browse/HADOOP-14272
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 2.8.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
{{org.apache.hadoop.fs.azure.WasbRemoteCallHelper}} method is using the
reference comparison to compare 2 strings, which is incorrect.
{code}
if (contentTypeHeader == null || contentTypeHeader.getValue() !=
APPLICATION_JSON)}}
{code}
The fix is to use {{!APPLICATION_JSON.equals(contentTypeHeader.getValue())}}
instead.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.11.patch
Thanks [~liuml07]. Patch #10 looks good. I have added similar exception
handling to {{RemoteWasbAuthorizerImpl.init()}} in patch #11.
[~ste...@apache.org] Could you please review?
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.10.patch, HADOOP-13945.11.patch,
> HADOOP-13945.1.patch, HADOOP-13945.2.patch, HADOOP-13945.3.patch,
> HADOOP-13945.4.patch, HADOOP-13945.5.patch, HADOOP-13945.6.patch,
> HADOOP-13945.7.patch, HADOOP-13945.8.patch, HADOOP-13945.9.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: (was: HADOOP-13945.10.patch)
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.10.patch, HADOOP-13945.1.patch,
> HADOOP-13945.2.patch, HADOOP-13945.3.patch, HADOOP-13945.4.patch,
> HADOOP-13945.5.patch, HADOOP-13945.6.patch, HADOOP-13945.7.patch,
> HADOOP-13945.8.patch, HADOOP-13945.9.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Comment: was deleted
(was: Thanks [~liuml07]. Patch #9 looks good to me. I have added similar
exception handling to {{RemoteWasbAuthorizerImpl.init()}} in patch #10.
[~ste...@apache.org] Could you please review?
)
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.10.patch, HADOOP-13945.1.patch,
> HADOOP-13945.2.patch, HADOOP-13945.3.patch, HADOOP-13945.4.patch,
> HADOOP-13945.5.patch, HADOOP-13945.6.patch, HADOOP-13945.7.patch,
> HADOOP-13945.8.patch, HADOOP-13945.9.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.10.patch
Thanks [~liuml07]. Patch #9 looks good to me. I have added similar exception
handling to {{RemoteWasbAuthorizerImpl.init()}} in patch #10.
[~ste...@apache.org] Could you please review?
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.10.patch, HADOOP-13945.10.patch,
> HADOOP-13945.1.patch, HADOOP-13945.2.patch, HADOOP-13945.3.patch,
> HADOOP-13945.4.patch, HADOOP-13945.5.patch, HADOOP-13945.6.patch,
> HADOOP-13945.7.patch, HADOOP-13945.8.patch, HADOOP-13945.9.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927681#comment-15927681
]
Santhosh G Nayak commented on HADOOP-13945:
---
[~liuml07], Fixed the javadoc and checkstyle warnings in patch #8.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch,
> HADOOP-13945.3.patch, HADOOP-13945.4.patch, HADOOP-13945.5.patch,
> HADOOP-13945.6.patch, HADOOP-13945.7.patch, HADOOP-13945.8.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.8.patch
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch,
> HADOOP-13945.3.patch, HADOOP-13945.4.patch, HADOOP-13945.5.patch,
> HADOOP-13945.6.patch, HADOOP-13945.7.patch, HADOOP-13945.8.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.7.patch
[~liuml07], Thanks for reviewing the patch.
I have addressed the review comments in patch #7.
(1) Added {{StringUtils.isNotEmpty(delegationToken)}} instead of explicit null
and empty checks.
(2) Wrapped {{InterruptedException}} inside {{WasbAuthorizationException}}, I
wanted to throw only meaningful exceptions to the above layer.
(3) Added javadoc for newly added methods.
(4) I have tested the patch in real secure Hadoop cluster with WASB storage by
running the basic fs operation shell commands, MR jobs and hive jobs.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch,
> HADOOP-13945.3.patch, HADOOP-13945.4.patch, HADOOP-13945.5.patch,
> HADOOP-13945.6.patch, HADOOP-13945.7.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.6.patch
Thanks [~liuml07] for reviewing the patch.
I have created a rebased patch excluding the changes in HADOOP-13930 (as it is
already committed) and addressed the following review comments,
(1) {{fs.azure.authorization.remote.service.url}} was introduced in
HADOOP-13930. Current patch does not have any reference to it.
(2) In code to log a message, keeping the exception as well.
(3) Unfortunately,
{{UserGroupInformation.getCurrentUser().getCredentials().getToken(WasbDelegationTokenIdentifier.TOKEN_KIND)}}
does not work, it takes only alias. Moving this logic to a util method.
(4) Fixed the nit by removing {{()}} for {{&&}} in {{if (isSecurityEnabled &&
(delegationToken != null && !delegationToken.isEmpty()))}}.
(5) Added {{package-info.java}} instead of {{package.html}}.
(6) Created Util methods to avoid duplicate code wherever possible.
(7), (8), (9) and (10) comments are related changes in HADOOP-13930 and already
addressed there.
(11) Handled it appropriately to the best of my knowledge, let me know if think
otherwise.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch,
> HADOOP-13945.3.patch, HADOOP-13945.4.patch, HADOOP-13945.5.patch,
> HADOOP-13945.6.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14150) Implement getHomeDirectory() method in NativeAzureFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-14150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-14150:
--
Attachment: HADOOP-14150.1.patch
Attaching initial version of the patch containing proposed changes.
> Implement getHomeDirectory() method in NativeAzureFileSystem
>
>
> Key: HADOOP-14150
> URL: https://issues.apache.org/jira/browse/HADOOP-14150
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-14150.1.patch
>
>
> {{org.apache.hadoop.fs.azure.NativeAzureFileSystem}} does not override
> {{FileSystem#getHomeDirectory()}} method.
> So, whenever {{NativeAzureFileSystem#getHomeDirectory()}} gets called
> {{getHomeDirectory()}} from {{FileSystem}} will be invoked, which has code
> like below,
> {code}
> public Path getHomeDirectory() {
> return this.makeQualified(
> new Path(USER_HOME_PREFIX + "/" + System.getProperty("user.name")));
> }
> {code}
> In secure environment, it returns home directory of
> {{System.getProperty("user.name")}} instead of Kerberos principal/UGI.
> So, the proposal is to override the {{getHomeDirectory()}} method in
> {{NativeAzureFileSystem}} and have it return the home directory for the
> Kerberos principal/ugi.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14150) Implement getHomeDirectory() method in NativeAzureFileSystem
Santhosh G Nayak created HADOOP-14150:
-
Summary: Implement getHomeDirectory() method in
NativeAzureFileSystem
Key: HADOOP-14150
URL: https://issues.apache.org/jira/browse/HADOOP-14150
Project: Hadoop Common
Issue Type: Improvement
Components: fs/azure
Affects Versions: 2.8.0
Reporter: Santhosh G Nayak
Assignee: Santhosh G Nayak
{{org.apache.hadoop.fs.azure.NativeAzureFileSystem}} does not override
{{FileSystem#getHomeDirectory()}} method.
So, whenever {{NativeAzureFileSystem#getHomeDirectory()}} gets called
{{getHomeDirectory()}} from {{FileSystem}} will be invoked, which has code like
below,
{code}
public Path getHomeDirectory() {
return this.makeQualified(
new Path(USER_HOME_PREFIX + "/" + System.getProperty("user.name")));
}
{code}
In secure environment, it returns home directory of
{{System.getProperty("user.name")}} instead of Kerberos principal/UGI.
So, the proposal is to override the {{getHomeDirectory()}} method in
{{NativeAzureFileSystem}} and have it return the home directory for the
Kerberos principal/ugi.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.5.patch
[~liuml07], Thanks for reviewing the patch.
In the latest patch, I have incorporated following,
- @return annotation in the {{init()}} is removed as it is void return type.
- Marked {{DEFAULT_AZURE_AUTHORIZATION}} as final.
- Bug fix - Using {{UserGroupInformation#getLoginUser()}} whenever kerberos
credentials are not available for {{UserGroupInformation#getCurrentUser()}}.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch,
> HADOOP-13945.3.patch, HADOOP-13945.4.patch, HADOOP-13945.5.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.4.patch
Thanks [~liuml07], I have fixed the unit test
{{hadoop.conf.TestCommonConfigurationFields}} failure in the new patch.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch,
> HADOOP-13945.3.patch, HADOOP-13945.4.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.3.patch
Adding a patch which is created on top of
https://issues.apache.org/jira/secure/attachment/12845516/HADOOP-13930.002.patch,
Patch contains following additional changes,
- Kerberos support for {{RemoteWasbAuthorizerImpl#authorize()}} request to
remote server.
- Added a configuration property {{fs.azure.enable.kerberos.support}} to
enable/disable Kerberos support.
- Support for impersonation(doAs) of a user with remote service.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch,
> HADOOP-13945.3.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798285#comment-15798285
]
Santhosh G Nayak edited comment on HADOOP-13945 at 1/4/17 1:42 PM:
---
[~liuml07], Thanks for reviewing the patch. I have attached another working
patch addressing following review comments,
1) Added log statements with meaningful messages.
2) I will try to add tests it in the next iteration.
3) Marked Constants class as final with a private constructor.
4) Not added {{getSASKey()}} method as static in the current iteration, as the
method depends on the instance delegation token.
5) Fixed most of checkstyle related warnings.
6) I have not incorporated this change in the current iteration, as
{{AbstractDelegationTokenSelector#selectToken()}} selects the token based on
service and token kind. We wanted only based on the kind of the token.
There are some more changes planned as part of this JIRA in subsequent
iterations,
- Adding retries and client side fail over mechanisms while trying to obtain
delegation tokens and Azure Storage SAS keys from remote service.
- Ability to provide customized connection configurator while trying to connect
to remote service.
was (Author: snayak):
[~liuml07], Thanks for reviewing the patch. I have attached another working
patch addressing following review comments,
1) Added log statements with meaningful messages.
2) I will try to add tests it in the next iteration.
3) Marked Constants class as final with a private constructor.
4) Not added {{getSASKey()}} method as static in the current iteration, as the
method depends on the instance delegation token.
5) Fixed most of checkstyle related warnings.
6) I have not incorporated this change in the current iteration, as
{{AbstractDelegationTokenSelector#selectToken()}} selects the token based on
service and token kind. We wanted only based on the kind of the token.
There are some more changes planned as part of this JIRA,
- Adding retries and client side fail over mechanisms while trying to obtain
delegation tokens and Azure Storage SAS keys from remote service.
- Ability to provide customized connection configurator while trying to connect
to remote service.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.2.patch
[~liuml07], Thanks for reviewing the patch. I have attached another working
patch addressing following review comments,
1) Added log statements with meaningful messages.
2) I will try to add tests it in the next iteration.
3) Marked Constants class as final with a private constructor.
4) Not added {{getSASKey()}} method as static in the current iteration, as the
method depends on the instance delegation token.
5) Fixed most of checkstyle related warnings.
6) I have not incorporated this change in the current iteration, as
{{AbstractDelegationTokenSelector#selectToken()}} selects the token based on
service and token kind. We wanted only based on the kind of the token.
There are some more changes planned as part of this JIRA,
- Adding retries and client side fail over mechanisms while trying to obtain
delegation tokens and Azure Storage SAS keys from remote service.
- Ability to provide customized connection configurator while trying to connect
to remote service.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch, HADOOP-13945.2.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.1.patch
Attaching initial version of the patch containing proposed changes.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
Santhosh G Nayak created HADOOP-13945:
-
Summary: Azure: Add Kerberos and Delegation token support to WASB
client.
Key: HADOOP-13945
URL: https://issues.apache.org/jira/browse/HADOOP-13945
Project: Hadoop Common
Issue Type: Improvement
Components: azure, fs/azure
Affects Versions: 2.8.0
Reporter: Santhosh G Nayak
Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
support Kerberos Authentication and FileSystem authorization, which makes it
unusable in secure environments with multi user setup.
To make {{WASB}} client more suitable to run in Secure environments, there are
2 initiatives under way for providing the authorization (HADOOP-13930) and fine
grained access control (HADOOP-13863) support.
This JIRA is created to add Kerberos and delegation token support to {{WASB}}
client to fetch Azure Storage SAS keys (from Remote service as discussed in
HADOOP-13863), which provides fine grained timed access to containers and
blobs.
For delegation token management, the proposal is it use the same REST service
which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
