Is there a way to specify a request header during the <sparkContext>.textFile call?
- Ranga On Mon, Oct 13, 2014 at 11:03 AM, Ranga <sra...@gmail.com> wrote: > Hi > > I am trying to access files/buckets in S3 and encountering a permissions > issue. The buckets are configured to authenticate using an IAMRole provider. > I have set the KeyId and Secret using environment variables ( > AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still unable > to access the S3 buckets. > > Before setting the access key and secret the error was: > "java.lang.IllegalArgumentException: > AWS Access Key ID and Secret Access Key must be specified as the username > or password (respectively) of a s3n URL, or by setting the > fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties > (respectively)." > > After setting the access key and secret, the error is: "The AWS Access > Key Id you provided does not exist in our records." > > The id/secret being set are the right values. This makes me believe that > something else ("token", etc.) needs to be set as well. > Any help is appreciated. > > > - Ranga >