[
https://issues.apache.org/jira/browse/HADOOP-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17755153#comment-17755153
]
ASF GitHub Bot commented on HADOOP-18610:
-----------------------------------------
creste opened a new pull request, #5953:
URL: https://github.com/apache/hadoop/pull/5953
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Add support for [Azure Active Directory (Azure AD) workload
identities](https://learn.microsoft.com/en-us/azure/active-directory/workload-identities/workload-identities-overview)
which integrate with the Kubernetes's native capabilities to federate with any
external identity provider.
This PR is based on Haifeng Chen's patch attached to
[HADOOP-18610](https://issues.apache.org/jira/browse/HADOOP-18610). I fixed a
few typos and linter errors but did not modify the core functionality.
### How was this patch tested?
New ABFS OAuth test configuration added for WorkloadIdentityTokenProvider.
Complete test suite was run against Azure Blob Storage in Central US region.
------------------------
:::: AGGREGATED TEST RESULT ::::
HNS-OAuth
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 587, Failures: 0, Errors: 0, Skipped: 99
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:206 »
TestTimedOut test t...
[INFO]
[ERROR] Tests run: 339, Failures: 0, Errors: 1, Skipped: 56
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testThrottlingIntercept:106 »
KeyProvider Failure t...
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 1, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 587, Failures: 0, Errors: 0, Skipped: 68
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:206 »
TestTimedOut test t...
[INFO]
[ERROR] Tests run: 339, Failures: 0, Errors: 1, Skipped: 43
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testThrottlingIntercept:106 »
KeyProvider Failure t...
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 1, Skipped: 11
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181
Expecting org.apache.hadoop.security.AccessControlException with text "This
request is not authorized to perform this operation using this permission.",
403 but got : "void"
[INFO]
[ERROR] Tests run: 587, Failures: 1, Errors: 0, Skipped: 277
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:206 »
TestTimedOut test t...
[INFO]
[ERROR] Tests run: 339, Failures: 0, Errors: 1, Skipped: 46
AppendBlob-HNS-OAuth
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:177->twoWriters:165
» AbfsRestOperation
[INFO]
[ERROR] Tests run: 587, Failures: 0, Errors: 1, Skipped: 99
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89
The actual value of 99 was not equal to the expected value
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:206 »
TestTimedOut test t...
[INFO]
[ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 80
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
- [X] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
> ---------------------------------------------------------------------
>
> Key: HADOOP-18610
> URL: https://issues.apache.org/jira/browse/HADOOP-18610
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools
> Affects Versions: 3.3.4
> Reporter: Haifeng Chen
> Priority: Critical
> Attachments: HADOOP-18610-preview.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity
> with with [Azure Active Directory (Azure AD) workload
> identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview]
> (preview), which integrate with the Kubernetes native capabilities to
> federate with any external identity providers. This approach is simpler to
> use and deploy.
> Refer to
> [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview|https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.]
> and [https://azure.github.io/azure-workload-identity/docs/introduction.html]
> for more details.
> The basic use scenario is to access Azure cloud resources (such as cloud
> storage) from Kubernetes (such as AKS) workload using Azure managed identity
> federated with Kubernetes service account. The credential environment
> variables in pod projected by Azure AD workload identity are like following:
> AZURE_AUTHORITY_HOST: (Injected by the webhook,
> [https://login.microsoftonline.com/])
> AZURE_CLIENT_ID: (Injected by the webhook)
> AZURE_TENANT_ID: (Injected by the webhook)
> AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook,
> /var/run/secrets/azure/tokens/azure-identity-token)
> The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON
> Web Token) client assertion token which we can use to request to
> AZURE_AUTHORITY_HOST (url is AZURE_AUTHORITY_HOST + tenantId +
> "/oauth2/v2.0/token") for a AD token which can be used to directly access
> the Azure cloud resources.
> This approach is very common and similar among cloud providers such as AWS
> and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to
> handle the same case.
> The existing MsiTokenProvider can only handle the managed identity associated
> with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider
> which handle Azure Workload Identity case. For this, we need to add one
> method (getTokenUsingJWTAssertion) in AzureADAuthenticator which will be used
> by WorkloadIdentityTokenProvider.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
