[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941523#comment-15941523
]
Kazuyuki Tanimura edited comment on HADOOP-14237 at 3/25/17 2:29 AM:
-
True.
Just to be clear, this patch is for making sure the credentials is shared among
all Hadoop nodes not only shared within a node.
As I add more nodes to a cluster, it was too easy to hit the AWS account level
limits.
was (Author: kazuyukitanimura):
True.
Just to be clear, this patch is for making sure the credentials is shared among
all Hadoop nodes not only shared within a node.
As I add more nodes to a cluster, it was too easy to hit the account level
limits.
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
