[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567625#comment-16567625 ] Steve Loughran commented on HADOOP-14237: - I'm looking at this again, going to move to 3.3 along with most of the other outstainding s3 for 3.2 features. * I don't like saving the full secrets (unencrypted) to HDFS * session secrets could work, though of course they'll expire within 24h. once HADOOP-15883 is in I'm going to revisit HADOOP-14556, which lets the s3a client to serialize its secrets as a filesystem delegation token, something apps (hive, spark, MR) know to ask for -and which YARN knows how to securely marshall to launched apps. With this feature you could launch things into a pool of VMs with reduced privilege IAM roles, sending in higher privilege credentials with the request. Would that work? I've also created HADOOP-15650 to cover the issue of better retry logic on credential retrieval. I see there's an async option, which might be more responsive, but could put even more load on the service unless managed carefully. What it could do though, is handle retries much better (though it'd also be a more more complicated) > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura >Priority: Major > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558788#comment-16558788
]
Steve Loughran commented on HADOOP-14237:
-
Revisiting this
* what was the use case which triggered this. You were trying to log in and
things were failing?
* And this was in EC2, so you were using the IAM role aut, which does an HTTP
Get?
* Can we have a stack trace?
We could address this by having our own subclass of the
{{InstanceProfileCredentialsProvider}} whose getCredentials retries on whatever
error gets raised by the service. That'd have to be a very different retry
policy from {{S3ARetryPolicy}}, which tries to reconnect on network/connection
refused. We will want failfast there.
All I really need to know is the error raised & error text, and we can recover
from failures here with retry & backoff.
Looks like com.amazonaws.retry.RetryUtils has a predicate to see if an
exception is for throttling. If we use that in translateException, it'll work
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>Assignee: Kazuyuki Tanimura
>Priority: Major
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969742#comment-15969742 ] Kazuyuki Tanimura commented on HADOOP-14237: I will make this an independnet credential provider. > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967323#comment-15967323 ] Steve Loughran commented on HADOOP-14237: - +need to think about handling session token refresh > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966314#comment-15966314 ] Mingliang Liu commented on HADOOP-14237: You may need to rebase the patch as [HADOOP-14248] and/or [HADOOP-14301]. Thanks, > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943104#comment-15943104 ] Steve Loughran commented on HADOOP-14237: - +persisted data structure should use the JECKS encrypted credential mechanism, so that it isn't stored in plaintext, even in HDFS. Processes which can access the data would need to be given the shared key and path needed to find and read the data, > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943071#comment-15943071 ] Steve Loughran commented on HADOOP-14237: - I see: it's extracting the credentials, then saving them to the cluster FS, so that no clients need to hit the IAM infra so much. if it overloads, it reads back from HDFS. If this is to go in, as well as needing per-user temp dir, and all the various tests, maybe even expiry of credentials, this MUST be its own credential provider. This needs to be optional, and now we've added the ability to declare your own providers, that'll be how people use it. Test plan: * split save/load from rest of provider, test independently, including handling some read/write failure conditions. * verify that credentials are saved on successful auth. * maybe using mocking simulate IAM overload > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943068#comment-15943068
]
ASF GitHub Bot commented on HADOOP-14237:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/207#discussion_r108144627
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SharedInstanceProfileCredentialsProvider.java
---
@@ -58,6 +71,84 @@ public static SharedInstanceProfileCredentialsProvider
getInstance() {
return INSTANCE;
}
+ private AWSCredentials readCredentialsFromHDFS() {
+try {
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(s3crednetialPath)));
+ String accessKey = br.readLine();
+ String secretKey = br.readLine();
+ String token = br.readLine();
+ AWSCredentials credentials;
+ if (StringUtils.isEmpty(accessKey) ||
StringUtils.isEmpty(secretKey)) {
+// if there are no accessKey nor secretKey return null
+return null;
+ } else if (StringUtils.isNotEmpty(token)) {
+credentials = new BasicSessionCredentials(accessKey, secretKey,
token);
+ } else {
+credentials = new BasicAWSCredentials(accessKey, secretKey);
+ }
+ return credentials;
+} catch (Exception e) {
+ return null; // ignore the read errors
+ // throw new AmazonServiceException("Failed reading S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ private void writeCredentialsToHDFS(AWSCredentials credentials) {
+try {
+ // Simulate atomic write by creating a new s3credential file with
random string suffix and rename to s3crednetialPath
+ Path newS3crednetialPath = new Path(s3crednetialPath.toUri() +
RandomStringUtils.randomAlphanumeric(8));
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedWriter br = new BufferedWriter(new
OutputStreamWriter(fs.create(newS3crednetialPath, true)));
+ String accessKey = credentials.getAWSAccessKeyId();
+ String secretKey = credentials.getAWSSecretKey();
+ String token = "";
+ if (credentials instanceof BasicSessionCredentials) {
--- End diff --
I would only allow session credentials to persist, so as to reduce risk of
leakage of persistent secrets
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943069#comment-15943069
]
ASF GitHub Bot commented on HADOOP-14237:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/207#discussion_r108144687
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SharedInstanceProfileCredentialsProvider.java
---
@@ -58,6 +71,84 @@ public static SharedInstanceProfileCredentialsProvider
getInstance() {
return INSTANCE;
}
+ private AWSCredentials readCredentialsFromHDFS() {
+try {
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(s3crednetialPath)));
+ String accessKey = br.readLine();
+ String secretKey = br.readLine();
+ String token = br.readLine();
+ AWSCredentials credentials;
+ if (StringUtils.isEmpty(accessKey) ||
StringUtils.isEmpty(secretKey)) {
+// if there are no accessKey nor secretKey return null
+return null;
+ } else if (StringUtils.isNotEmpty(token)) {
+credentials = new BasicSessionCredentials(accessKey, secretKey,
token);
+ } else {
+credentials = new BasicAWSCredentials(accessKey, secretKey);
+ }
+ return credentials;
+} catch (Exception e) {
+ return null; // ignore the read errors
+ // throw new AmazonServiceException("Failed reading S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ private void writeCredentialsToHDFS(AWSCredentials credentials) {
+try {
+ // Simulate atomic write by creating a new s3credential file with
random string suffix and rename to s3crednetialPath
+ Path newS3crednetialPath = new Path(s3crednetialPath.toUri() +
RandomStringUtils.randomAlphanumeric(8));
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedWriter br = new BufferedWriter(new
OutputStreamWriter(fs.create(newS3crednetialPath, true)));
+ String accessKey = credentials.getAWSAccessKeyId();
+ String secretKey = credentials.getAWSSecretKey();
+ String token = "";
+ if (credentials instanceof BasicSessionCredentials) {
+token = ((BasicSessionCredentials) credentials).getSessionToken();
+ }
+ br.write(accessKey);
+ br.newLine();
+ br.write(secretKey);
+ br.newLine();
+ br.write(token);
+ br.newLine();
+ br.close();
+ fs.delete(s3crednetialPath, false);
+ fs.rename(newS3crednetialPath, s3crednetialPath);
+} catch (Exception e) {
+ // ignore write errors
+ // throw new AmazonServiceException("Failed writing S3 credentials
from HDFS " + e.getStackTrace());
--- End diff --
log @ debug at the very least
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943066#comment-15943066
]
ASF GitHub Bot commented on HADOOP-14237:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/207#discussion_r108144389
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SharedInstanceProfileCredentialsProvider.java
---
@@ -58,6 +71,84 @@ public static SharedInstanceProfileCredentialsProvider
getInstance() {
return INSTANCE;
}
+ private AWSCredentials readCredentialsFromHDFS() {
+try {
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(s3crednetialPath)));
+ String accessKey = br.readLine();
+ String secretKey = br.readLine();
+ String token = br.readLine();
+ AWSCredentials credentials;
+ if (StringUtils.isEmpty(accessKey) ||
StringUtils.isEmpty(secretKey)) {
+// if there are no accessKey nor secretKey return null
+return null;
+ } else if (StringUtils.isNotEmpty(token)) {
+credentials = new BasicSessionCredentials(accessKey, secretKey,
token);
+ } else {
+credentials = new BasicAWSCredentials(accessKey, secretKey);
+ }
+ return credentials;
+} catch (Exception e) {
+ return null; // ignore the read errors
+ // throw new AmazonServiceException("Failed reading S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ private void writeCredentialsToHDFS(AWSCredentials credentials) {
+try {
+ // Simulate atomic write by creating a new s3credential file with
random string suffix and rename to s3crednetialPath
+ Path newS3crednetialPath = new Path(s3crednetialPath.toUri() +
RandomStringUtils.randomAlphanumeric(8));
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedWriter br = new BufferedWriter(new
OutputStreamWriter(fs.create(newS3crednetialPath, true)));
+ String accessKey = credentials.getAWSAccessKeyId();
+ String secretKey = credentials.getAWSSecretKey();
+ String token = "";
+ if (credentials instanceof BasicSessionCredentials) {
+token = ((BasicSessionCredentials) credentials).getSessionToken();
+ }
+ br.write(accessKey);
+ br.newLine();
+ br.write(secretKey);
+ br.newLine();
+ br.write(token);
+ br.newLine();
+ br.close();
+ fs.delete(s3crednetialPath, false);
+ fs.rename(newS3crednetialPath, s3crednetialPath);
+} catch (Exception e) {
+ // ignore write errors
+ // throw new AmazonServiceException("Failed writing S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ @Override
+ public AWSCredentials getCredentials() {
+for (int retry = 0; retry < maxRetries; retry++) {
+ try {
+AWSCredentials newCredentials = super.getCredentials();
+// if this new credentials is different from HDFS write back
+if (credentials == null ||
(!newCredentials.getAWSSecretKey().equals(credentials.getAWSSecretKey( {
+ credentials = newCredentials;
+ writeCredentialsToHDFS(credentials);
+}
+break;
+ } catch (Exception e) {
--- End diff --
I't use our normal Retry logic here, consider some sleep + jitter if it
really is caused by throttling
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: co
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941523#comment-15941523 ] Kazuyuki Tanimura commented on HADOOP-14237: True. Just to be clear, this patch is for making sure the credentials is shared among all Hadoop nodes not only shared within a node. As I add more nodes to a cluster, it was too easy to hit the account level limits. > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
