[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617608#comment-17617608
]
ASF GitHub Bot commented on HADOOP-18481:
-
steveloughran merged PR #4973:
URL: https://github.com/apache/hadoop/pull/4973
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617609#comment-17617609
]
ASF GitHub Bot commented on HADOOP-18481:
-
steveloughran commented on PR #4973:
URL: https://github.com/apache/hadoop/pull/4973#issuecomment-1278768576
merged to trunk; will backport
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617560#comment-17617560
]
ASF GitHub Bot commented on HADOOP-18481:
-
ahmarsuhail commented on PR #4973:
URL: https://github.com/apache/hadoop/pull/4973#issuecomment-1278675683
Yes, sorry should have mentioned! Tested by setting
`fs.s3a.aws.credentials.provider` to
`com.amazonaws.auth.AWSStaticCredentialsProvider`, and ran an Integration test
(`ITestPrefetchingInputStream`) which logs warning. Then tested with
`com.amazonaws.auth.EnvironmentVariableCredentialsProvider`,
`com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper` and
`com.amazonaws.auth.InstanceProfileCredentialsProvider` which do not log
warning.
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617182#comment-17617182
]
ASF GitHub Bot commented on HADOOP-18481:
-
steveloughran commented on PR #4973:
URL: https://github.com/apache/hadoop/pull/4973#issuecomment-1277893256
and you tested it against an s3 store again, right?
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616644#comment-17616644
]
ASF GitHub Bot commented on HADOOP-18481:
-
hadoop-yetus commented on PR #4973:
URL: https://github.com/apache/hadoop/pull/4973#issuecomment-1276591714
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 56s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 38s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 101m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4973/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4973 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 00d9d8c2ca79 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 192d039bb21c7a653602f71a88177af63146af02 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4973/2/testReport/ |
| Max. process+thread count | 562 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4973/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616592#comment-17616592
]
ASF GitHub Bot commented on HADOOP-18481:
-
ahmarsuhail commented on PR #4973:
URL: https://github.com/apache/hadoop/pull/4973#issuecomment-1276483155
thanks @steveloughran, have added `EC2ContainerCredentialsProviderWrapper`
and `InstanceProfileCredentialsProvider`
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17615211#comment-17615211
]
ASF GitHub Bot commented on HADOOP-18481:
-
steveloughran commented on code in PR #4973:
URL: https://github.com/apache/hadoop/pull/4973#discussion_r991486622
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java:
##
@@ -640,7 +640,8 @@ public static AWSCredentialProviderList
buildAWSProviderList(
AWSCredentialProviderList providers = new AWSCredentialProviderList();
for (Class aClass : awsClasses) {
- if (aClass.getName().contains(AWS_AUTH_CLASS_PREFIX)) {
+ if
(!aClass.getSimpleName().equals("EnvironmentVariableCredentialsProvider")
Review Comment:
how about having an array of simple names to skip on, with that,
EC2ContainerCredentialsProviderWrapper, and InstanceProfileCredentialsProvider
in. we only added our own IAMInstanceCredentialsProvider with delegation
tokens, HADOOP-15449, and until then just included the aws sdk one directly.
any refs to either can be mapped to IAMInstanceCredentialsProvider before
trying to instantiate
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17615114#comment-17615114
]
ASF GitHub Bot commented on HADOOP-18481:
-
ahmarsuhail commented on PR #4973:
URL: https://github.com/apache/hadoop/pull/4973#issuecomment-1273310987
@steveloughran could you please review? I'll then pick up testing for
https://github.com/apache/hadoop/pull/4698 against `branch-3.3`.
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613101#comment-17613101
]
Steve Loughran commented on HADOOP-18481:
-
thanks, there's probably a few which can be silently migrated
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613069#comment-17613069
]
Ahmar Suhail commented on HADOOP-18481:
---
Opened [PR|https://github.com/apache/hadoop/pull/4973]. Will update class name
as part of V2 move
> AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
>
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Major
>
> looking at test output with the sdk warnings enabled, it is now always
> warning of a v1 provider reference, even if the user hasn't set any
> fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
> (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
> provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK
> V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
> (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class
> is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the
> default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared
> a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be
> patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass
> of this; and while that is potentially good in future it is a bit more effort
> for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot
> explicitly switch to it in their configs and be confident it will always be
> there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
