[jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
[ https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323220#comment-15323220 ] Steve Loughran commented on HADOOP-9680: Dmitry, I'm reluctant to go near a jets3t upgrade as there are normally consequences somewhere else. IAM and STS Temporary credential support will be in S3A in Hadoop 2.8.0 > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials > -- > > Key: HADOOP-9680 > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.1.0-beta, 3.0.0-alpha1 >Reporter: Robert Gibbon >Priority: Minor > Attachments: s3fs-temp-iam-creds.diff.patch > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > Temporary Security Credentials secured interactions with S3 from Hadoop. > It bumps the JetS3t release version up to 0.9.0. > To use a temporary security credential set, you need to provide the following > properties, depending on the implementation (s3 or s3native): > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > issued by AWS IAM > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret > access key issued by AWS IAM > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > by AWS IAM along with the temporary key > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
[ https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319160#comment-15319160 ] Dmitry Vasilenko commented on HADOOP-9680: -- +1 > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials > -- > > Key: HADOOP-9680 > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.1.0-beta, 3.0.0-alpha1 >Reporter: Robert Gibbon >Priority: Minor > Attachments: s3fs-temp-iam-creds.diff.patch > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > Temporary Security Credentials secured interactions with S3 from Hadoop. > It bumps the JetS3t release version up to 0.9.0. > To use a temporary security credential set, you need to provide the following > properties, depending on the implementation (s3 or s3native): > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > issued by AWS IAM > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret > access key issued by AWS IAM > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > by AWS IAM along with the temporary key > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
[ https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068615#comment-15068615 ] Brendan Maguire commented on HADOOP-9680: - I see this is still open. Is there still no way to use S3 from Hadoop using temporary credentials? > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials > -- > > Key: HADOOP-9680 > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.1.0-beta, 3.0.0 >Reporter: Robert Gibbon >Priority: Minor > Attachments: s3fs-temp-iam-creds.diff.patch > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > Temporary Security Credentials secured interactions with S3 from Hadoop. > It bumps the JetS3t release version up to 0.9.0. > To use a temporary security credential set, you need to provide the following > properties, depending on the implementation (s3 or s3native): > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > issued by AWS IAM > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret > access key issued by AWS IAM > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > by AWS IAM along with the temporary key > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
[ https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900112#comment-13900112 ] Daisuke Miyamoto commented on HADOOP-9680: -- +1 > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials > -- > > Key: HADOOP-9680 > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Robert Gibbon >Priority: Minor > Attachments: s3fs-temp-iam-creds.diff.patch > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > Temporary Security Credentials secured interactions with S3 from Hadoop. > It bumps the JetS3t release version up to 0.9.0. > To use a temporary security credential set, you need to provide the following > properties, depending on the implementation (s3 or s3native): > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > issued by AWS IAM > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret > access key issued by AWS IAM > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > by AWS IAM along with the temporary key > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
[ https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697089#comment-13697089 ] Robert Gibbon commented on HADOOP-9680: --- I took a look at your patch in HADOOP-9623. Some comments: * Bucket keyspace listings running over a s3-native fs will be broken by your patch, they make use of the method org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(String key) when querying S3 for the given URI; if the URI does not correspond to a single key in a bucket (ie a single object) an exception is thrown. In the above mentioned method, the exception's Message property was being parsed for the string "ResponseCode=404" to interpret that the URI is not a single key. If the condition is met, it returns null. Its a horrible piece of code and a very poorly defined contract with the calling party. It is also broken by jets3t 0.9.0, which doesn't pass back that message anymore in that situation. I adapted it to look at the ResponseCode property for the integer 404 instead, but someone who knows that code better than me would do a good deed to fix it more sustainably. * I needed to upgrade jets3t to 0.9.0 because I need support for AWS IAM federated access tokens (temporary, time limited access credentials, tied to a session ticket). I don't see any support for that in the patch in HADOOP-9623, for me its of no value unless it supports temp security tokens. I think we're aligned on the need for an uprev of the jets3t implementation in hadoop. HTH > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials > -- > > Key: HADOOP-9680 > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Robert Gibbon >Priority: Minor > Attachments: s3fs-temp-iam-creds.diff.patch > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > Temporary Security Credentials secured interactions with S3 from Hadoop. > It bumps the JetS3t release version up to 0.9.0. > To use a temporary security credential set, you need to provide the following > properties, depending on the implementation (s3 or s3native): > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > issued by AWS IAM > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret > access key issued by AWS IAM > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > by AWS IAM along with the temporary key > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials
[ https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13697019#comment-13697019 ] Timothy St. Clair commented on HADOOP-9680: --- Seems directly related to--> https://issues.apache.org/jira/browse/HADOOP-9623 > Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials > -- > > Key: HADOOP-9680 > URL: https://issues.apache.org/jira/browse/HADOOP-9680 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Robert Gibbon >Priority: Minor > Attachments: s3fs-temp-iam-creds.diff.patch > > > Here is a patch in unified diff format to enable Amazon Web Services IAM > Temporary Security Credentials secured interactions with S3 from Hadoop. > It bumps the JetS3t release version up to 0.9.0. > To use a temporary security credential set, you need to provide the following > properties, depending on the implementation (s3 or s3native): > fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id > issued by AWS IAM > fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret > access key issued by AWS IAM > fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued > by AWS IAM along with the temporary key > fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
