[jira] [Created] (HADOOP-18545) hadoop-azure integration tests are failing on the trunk branch
Carl created HADOOP-18545:
-
Summary: hadoop-azure integration tests are failing on the trunk
branch
Key: HADOOP-18545
URL: https://issues.apache.org/jira/browse/HADOOP-18545
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure, hadoop-thirdparty
Reporter: Carl
I am getting errors when running the hadoop-azure integration test from the
trunk branch (commit 0ef572abed624011ad9a9e69d725774d6d00b75b).
This is running on an Azure instance, with HNS disabled and using a Shared Key
(NonHNS-SharedKey).
Here are the logs when running those tests:
{code:java}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClient
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream
[INFO] Running org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.976 s
- in org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.174 s
- in org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker
[ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.921 s
<<< FAILURE! - in org.apache.hadoop.fs.azurebfs.TestAccountConfiguration
[ERROR]
testConfigPropNotFound(org.apache.hadoop.fs.azurebfs.TestAccountConfiguration)
Time elapsed: 0.102 s <<< FAILURE!
java.lang.AssertionError: Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at
org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testMissingConfigKey(TestAccountConfiguration.java:399)
at
org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testConfigPropNotFound(TestAccountConfiguration.java:386)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.
[jira] [Updated] (HADOOP-18542) Azure Token provider requires tenant and client IDs despite being optional
[ https://issues.apache.org/jira/browse/HADOOP-18542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl updated HADOOP-18542: -- Description: The `AbfsConfiguration` class requires that we provide a tenant and client ID when using the `MsiTokenProvider` class to fetch an authentication token. The bug is that those fields are not required by the Azure API, which can infer those fields when the call is made from an Azure instance. The fix is to make tenant and client ID optional when getting an Azure token from the Azure Metadata Service. A fix has been submitted here: [https://github.com/apache/hadoop/pull/4262] The bug was introduced with HADOOP-17725 ([https://github.com/apache/hadoop/pull/3041/files]) was: The `AbfsConfiguration` class requires that we provide a tenant and client ID when using the `MsiTokenProvider` class. The bug is that those fields are not required by the Azure API, which can infer those fields when the call is made from an Azure instance. The fix is to make tenant and client ID optional when getting an Azure token from the Azure Metadata Service. A fix has been submitted here: [https://github.com/apache/hadoop/pull/4262] The bug was introduced with HADOOP-17725 (https://github.com/apache/hadoop/pull/3041/files) > Azure Token provider requires tenant and client IDs despite being optional > -- > > Key: HADOOP-18542 > URL: https://issues.apache.org/jira/browse/HADOOP-18542 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, hadoop-thirdparty >Affects Versions: 3.3.2, 3.3.3, 3.3.4 >Reporter: Carl >Priority: Major > Labels: pull-request-available > > The `AbfsConfiguration` class requires that we provide a tenant and client ID > when using the `MsiTokenProvider` class to fetch an authentication token. The > bug is that those fields are not required by the Azure API, which can infer > those fields when the call is made from an Azure instance. > The fix is to make tenant and client ID optional when getting an Azure token > from the Azure Metadata Service. > A fix has been submitted here: [https://github.com/apache/hadoop/pull/4262] > The bug was introduced with HADOOP-17725 > ([https://github.com/apache/hadoop/pull/3041/files]) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18542) Azure Token provider requires tenant and client IDs despite being optional
Carl created HADOOP-18542: - Summary: Azure Token provider requires tenant and client IDs despite being optional Key: HADOOP-18542 URL: https://issues.apache.org/jira/browse/HADOOP-18542 Project: Hadoop Common Issue Type: Bug Components: fs/azure, hadoop-thirdparty Affects Versions: 3.3.4, 3.3.3, 3.3.2 Reporter: Carl The `AbfsConfiguration` class requires that we provide a tenant and client ID when using the `MsiTokenProvider` class. The bug is that those fields are not required by the Azure API, which can infer those fields when the call is made from an Azure instance. The fix is to make tenant and client ID optional when getting an Azure token from the Azure Metadata Service. A fix has been submitted here: [https://github.com/apache/hadoop/pull/4262] The bug was introduced with HADOOP-17725 (https://github.com/apache/hadoop/pull/3041/files) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17725) Improve error message for token providers in ABFS
[
https://issues.apache.org/jira/browse/HADOOP-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558102#comment-17558102
]
Carl commented on HADOOP-17725:
---
The changes related to this ticket force to set optional fields and prevent
from relying on Azure's metadata service to set them automatically.
Both [~vjasani] 's PR ([https://github.com/apache/hadoop/pull/3788)] and mine
fix it
([https://github.com/apache/hadoop/pull/4262|https://github.com/apache/hadoop/pull/4262)]).
Can we get one of these merged please ?
> Improve error message for token providers in ABFS
> -
>
> Key: HADOOP-17725
> URL: https://issues.apache.org/jira/browse/HADOOP-17725
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure, hadoop-thirdparty
>Affects Versions: 3.3.0
>Reporter: Ivan Sadikov
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> It would be good to improve error messages for token providers in ABFS.
> Currently, when a configuration key is not found or mistyped, the error is
> not very clear on what went wrong. It would be good to indicate that the key
> was required but not found in Hadoop configuration when creating a token
> provider.
> For example, when running the following code:
> {code:java}
> import org.apache.hadoop.conf._
> import org.apache.hadoop.fs._
> val conf = new Configuration()
> conf.set("fs.azure.account.auth.type", "OAuth")
> conf.set("fs.azure.account.oauth.provider.type",
> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
> conf.set("fs.azure.account.oauth2.client.id", "my-client-id")
> //
> conf.set("fs.azure.account.oauth2.client.secret.my-account.dfs.core.windows.net",
> "my-secret")
> conf.set("fs.azure.account.oauth2.client.endpoint", "my-endpoint")
> val path = new Path("abfss://contai...@my-account.dfs.core.windows.net/")
> val fs = path.getFileSystem(conf)
> fs.getFileStatus(path){code}
> The following exception is thrown:
> {code:java}
> TokenAccessProviderException: Unable to load OAuth token provider class.
> ...
> Caused by: UncheckedExecutionException: java.lang.NullPointerException:
> clientSecret
> ...
> Caused by: NullPointerException: clientSecret {code}
> which does not tell what configuration key was not loaded.
>
> IMHO, it would be good if the exception was something like this:
> {code:java}
> TokenAccessProviderException: Unable to load OAuth token provider class.
> ...
> Caused by: ConfigurationPropertyNotFoundException: Configuration property
> fs.azure.account.oauth2.client.secret not found. {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
