[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790284#comment-17790284 ] Antonio Murgia commented on HADOOP-17372: - Provided a PR with tests + doc as HADOOP-18993 > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.1 > > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789547#comment-17789547 ] Steve Loughran commented on HADOOP-17372: - well, an option to disable isolated classloader is something which could be added, just think of a test > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.1 > > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[
https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788742#comment-17788742
]
Antonio Murgia commented on HADOOP-17372:
-
I add a workaround for anyone having the same issue described by [~brandonvin].
In your own project (where you defined a custom credential provider, that needs
a compile time dependency on hadoop-aws) create a new java class like the
following:
{{package org.apache.hadoop.fs.s3a;}}
{{{}public class PatchedS3AFileSystem extends S3AFileSystem {{
Then proceed configuring your Spark app with the following property:
{{spark.hadoop.fs.s3a.impl = org.apache.hadoop.fs.s3a.PatchedS3AFileSystem}}
as well as you provider configuration:
{{fs.s3a.aws.credentials.provider =
com.enel.platform.batch.commons.aws.auth.v2.FileSystemReaderSessionCredentialsProvider}}
Doing so, the filesystem implementation AND the credential provider will be
loaded by Spark {{MutableClassloader}} (which is a child of {{Launcher}}
classloader, so can still access also all the classes loaded by it). In this
way the {{conf.setClassloader}} call will be performed by
{{PatchedS3AFileSystem}} will not have any effect because the classloader set
is the same that loaded the {{Configuration}} object itself.
[~ste...@apache.org] I'm still open to provide a PR to make this workaround
un-necessary.
> S3A AWS Credential provider loading gets confused with isolated classloaders
>
>
> Key: HADOOP-17372
> URL: https://issues.apache.org/jira/browse/HADOOP-17372
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.1
>
>
> Problem: exception in loading S3A credentials for an FS, "Class class
> com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
> AWSCredentialsProvider"
> Location: S3A + Spark dataframes test
> Hypothesised cause:
> Configuration.getClasses() uses the context classloader, and with the spark
> isolated CL that's different from the one the s3a FS uses, so it can't load
> AWS credential providers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788308#comment-17788308 ] Antonio Murgia commented on HADOOP-17372: - Given the issue reported by [~brandonvin] , would you consider a PR that is guarding the "conf.setClassLoader(this.getClass().getClassLoader())" call? Something like: "fs.s3a.fixclassloader" (any advice here is more than welcome) that is by default true and if set to false will skip the call? This would be a huge quality of life improvement at least for mine and Brandon team :) > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.1 > > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545824#comment-17545824 ] Steve Loughran commented on HADOOP-17372: - hmm. i don't want to get into second guess spark classloader games. too much risk of weirdness happening. we get a lot of escalations of basic s3a functionality not work when a HiveConfiguration is passed in, aws sdk provider referenced etc etc. You are into advanced use and it's out of scope. sorry. do try packaging the provider into an org.apache.hadoop or org.apache.spark sub package and see if that makes things go away. > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.1 > > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[
https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545665#comment-17545665
]
Brandon commented on HADOOP-17372:
--
Hello [~ste...@apache.org], thank you for your work on S3A.
In our Spark jobs, we use a custom AWS credentials provider class which is
bundled into the Spark application jar. This worked on Hadoop 3.2.1, but
unfortunately this class can't be found after upgrading to Hadoop 3.3.3. This
surfaces as a ClassNotFoundException in S3AFileSystem's initialization:
{noformat}
java.io.IOException: From option fs.s3a.aws.credentials.provider
java.lang.ClassNotFoundException: Class [custom AWS credentials provider class]
not found
at org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses (S3AUtils.java:657)
org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList (S3AUtils.java:680)
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet
(S3AUtils.java:631)
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient (S3AFileSystem.java:877)
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize (S3AFileSystem.java:534)
org.apache.hadoop.fs.FileSystem.createFileSystem
(FileSystem.java:3469){noformat}
We were able to track this down to the change in this ticket. I believe what's
happening here is:
* The S3AFileSystem class is provided by a jar on disk. This jar is added to
the java classpath via the normal java command-line option. So, the classloader
of S3AFileSystem is a java application classloader.
* The Spark application jar which contains our AWS credentials provider class
is downloaded at runtime by Spark and then "patched into" the java classpath
via Spark's mutable classloader.
* Therefore, classes in the application jar are not visible to the classloader
that loaded S3AFileSystem.
In the meantime, I think our most reasonable path forward is to pull the custom
AWS credentials provider out of the application jar, install it in a jar on
disk, and add it to the java command-line classpath like hadoop-aws itself. Not
too bad, but certainly more complicated than the prior setup with Hadoop 3.2.1.
> S3A AWS Credential provider loading gets confused with isolated classloaders
>
>
> Key: HADOOP-17372
> URL: https://issues.apache.org/jira/browse/HADOOP-17372
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.1
>
>
> Problem: exception in loading S3A credentials for an FS, "Class class
> com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
> AWSCredentialsProvider"
> Location: S3A + Spark dataframes test
> Hypothesised cause:
> Configuration.getClasses() uses the context classloader, and with the spark
> isolated CL that's different from the one the s3a FS uses, so it can't load
> AWS credential providers.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[
https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493612#comment-17493612
]
Ivan Sadikov commented on HADOOP-17372:
---
Hi [~ste...@apache.org]. Thanks for fixing the issue, I confirmed it worked in
my case.
Looking at the code, I was wondering if ideally {{conf.setClassLoader()}}
should be setting the class loader for each getClass/getClassByName call from
the interface that it was trying to cast to instead of S3AFileSystem class
loader? I understand that the same class loader would likely load both
S3AFileSystem and AWS SDK but in case different class loaders are used, would
the solution be different?
> S3A AWS Credential provider loading gets confused with isolated classloaders
>
>
> Key: HADOOP-17372
> URL: https://issues.apache.org/jira/browse/HADOOP-17372
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.1
>
>
> Problem: exception in loading S3A credentials for an FS, "Class class
> com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
> AWSCredentialsProvider"
> Location: S3A + Spark dataframes test
> Hypothesised cause:
> Configuration.getClasses() uses the context classloader, and with the spark
> isolated CL that's different from the one the s3a FS uses, so it can't load
> AWS credential providers.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299807#comment-17299807 ] Steve Loughran commented on HADOOP-17372: - adding to HADOOP-17511 as auditing plugins are the kind of thing which breaks > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229890#comment-17229890 ] Steve Loughran commented on HADOOP-17372: - Thinking about this proposed (here and in ABFS) in initialize(), the cloned Configuration SHALL have its classloader set to this.getClass.getCloassloader so that all classes loaded in it are visible/consistent with the FS > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229527#comment-17229527 ] Steve Loughran commented on HADOOP-17372: - setting com.aws in "spark.sql.hive.metastore.sharedPrefixes" should be enough. > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[
https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229521#comment-17229521
]
Steve Loughran commented on HADOOP-17372:
-
Probable cause
* cluster fs = s3a
* FileSystem.get of default fs is using the HiveConf config (loaded in
isolation, with its classloader referenced)
* S3A FS init creates a new Configuration object, but it copies the classloader
ref of the hive conf
list of classes to load is effectively
{code}
conf.getClasses("fs.s3a.aws.credentials.provider",
"org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
{code}
the spark isolated CL for hive passes the o.a.h. classes through fine, but
com.amazon one is being loaded in the hive CL, and so the
EnvironmentVariableCredentialsProvider really isn't a valid provider
workaround: subclass that provider into org.a.h.fs.s3a.auth
but: all other s3a extension points (signer, delegation token provider) need to
reference implementation classes in org.apache.hadoop, or there is a risk that
spark can't load them through hive -at least if s3a is made the cluster FS.
> S3A AWS Credential provider loading gets confused with isolated classloaders
>
>
> Key: HADOOP-17372
> URL: https://issues.apache.org/jira/browse/HADOOP-17372
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> Problem: exception in loading S3A credentials for an FS, "Class class
> com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
> AWSCredentialsProvider"
> Location: S3A + Spark dataframes test
> Hypothesised cause:
> Configuration.getClasses() uses the context classloader, and with the spark
> isolated CL that's different from the one the s3a FS uses, so it can't load
> AWS credential providers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229483#comment-17229483 ] Steve Loughran commented on HADOOP-17372: - SPARK-9206 is related-ish, as it shows where the hive classloader was tweaked to allow gcs classes through > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[
https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229269#comment-17229269
]
Steve Loughran commented on HADOOP-17372:
-
{code}
2020-11-10 05:27:33,517 [ScalaTest-main-running-S3DataFrameExampleSuite] WARN
fs.FileSystem (FileSystem.java:createFileSystem(3466)) - Failed to initialize
fileystem s3a://stevel-ireland: java.io.IOException: Class class
com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
AWSCredentialsProvider
- DataFrames *** FAILED ***
org.apache.spark.sql.AnalysisException: java.lang.RuntimeException:
java.io.IOException: Class class
com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
AWSCredentialsProvider;
at
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at
org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:218)
at
org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:151)
at
org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:139)
at
org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:178)
at
org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:173)
at
org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
at
org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:91)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:91)
...
Cause: java.lang.RuntimeException: java.io.IOException: Class class
com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
AWSCredentialsProvider
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:686)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:621)
at
org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:219)
at
org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:126)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:306)
at
org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:433)
...
Cause: java.io.IOException: Class class
com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
AWSCredentialsProvider
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:722)
at org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:687)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:620)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:673)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:414)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3462)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:171)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3522)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3496)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:591)
...
{code}
> S3A AWS Credential provider loading gets confused with isolated classloaders
>
>
> Key: HADOOP-17372
> URL: https://issues.apache.org/jira/browse/HADOOP-17372
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> Problem: exception in loading S3A credentials for an FS, "Class class
> com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement
> AWSCredentialsProvider"
> Location: S3A + Spark dataframes test
> Hypothesised cause:
> Configuration.getClasses() uses the context classloader, and with the spark
> isolated CL that's different from the one the s3a FS uses, so it can't load
> AWS credential providers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
