Hi,
keep in mind that you're going to have a bad time if your secret key
contains a "/"
This is due to old and stupid hadoop bug:
https://issues.apache.org/jira/browse/HADOOP-3733
Best way is to regenerate the key so it does not include a "/"
/Raf
Akhil Das wrote:
> Try the following:
>
> 1. Set the access key and secret key in the sparkContext:
>
> sparkContext.set("
>
> AWS_ACCESS_KEY_ID",yourAccessKey)
>
> sparkContext.set("
>
> AWS_SECRET_ACCESS_KEY",yourSecretKey)
>
>
> 2. Set the access key and secret key in the environment before
> starting your application:
>
>
>
> export
>
> AWS_ACCESS_KEY_ID=<your access>
>
> export
>
> AWS_SECRET_ACCESS_KEY=<your secret>
>
>
> 3. Set the access key and secret key inside the hadoop configurations
>
> val hadoopConf=sparkContext.hadoopConfiguration;
>
> hadoopConf.set("fs.s3.impl",
> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
>
> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
>
> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>
>
> 4. You can also try:
>
> val lines =
>
> s
>
> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@<yourBucket>/path/")
>
>
> Thanks
> Best Regards
>
> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <sra...@gmail.com
> <mailto:sra...@gmail.com>> wrote:
>
> Hi
>
> I am trying to access files/buckets in S3 and encountering a
> permissions issue. The buckets are configured to authenticate
> using an IAMRole provider.
> I have set the KeyId and Secret using environment variables
> (AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
> unable to access the S3 buckets.
>
> Before setting the access key and secret the error was:
> "java.lang.IllegalArgumentException: AWS Access Key ID and Secret
> Access Key must be specified as the username or password
> (respectively) of a s3n URL, or by setting the
> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
> (respectively)."
>
> After setting the access key and secret, the error is: "The AWS
> Access Key Id you provided does not exist in our records."
>
> The id/secret being set are the right values. This makes me
> believe that something else ("token", etc.) needs to be set as well.
> Any help is appreciated.
>
>
> - Ranga
>
>
