(Copying the user list)
You should use spark_ec2 script to configure the cluster. If you use trunk
version you can use the new --copy-aws-credentials option to configure the
S3 parameters automatically, otherwise either include them in your
SparkConf variable or add them to
/root/spark/ephemeral-hdfs/conf/core-site.xml
On Mon, Oct 13, 2014 at 2:56 PM, Ranga <sra...@gmail.com> wrote:
> The cluster is deployed on EC2 and I am trying to access the S3 files from
> within a spark-shell session.
>
> On Mon, Oct 13, 2014 at 2:51 PM, Daniil Osipov <daniil.osi...@shazam.com>
> wrote:
>
>> So is your cluster running on EC2, or locally? If you're running locally,
>> you should still be able to access S3 files, you just need to locate the
>> core-site.xml and add the parameters as defined in the error.
>>
>> On Mon, Oct 13, 2014 at 2:49 PM, Ranga <sra...@gmail.com> wrote:
>>
>>> Hi Daniil
>>>
>>> No. I didn't create the spark-cluster using the ec2 scripts. Is that
>>> something that I need to do? I just downloaded Spark-1.1.0 and Hadoop-2.4.
>>> However, I am trying to access files on S3 from this cluster.
>>>
>>>
>>> - Ranga
>>>
>>> On Mon, Oct 13, 2014 at 2:36 PM, Daniil Osipov <daniil.osi...@shazam.com
>>> > wrote:
>>>
>>>> Did you add the fs.s3n.aws* configuration parameters in
>>>> /root/spark/ephemeral-hdfs/conf/core-ste.xml?
>>>>
>>>> On Mon, Oct 13, 2014 at 11:03 AM, Ranga <sra...@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I am trying to access files/buckets in S3 and encountering a
>>>>> permissions issue. The buckets are configured to authenticate using an
>>>>> IAMRole provider.
>>>>> I have set the KeyId and Secret using environment variables (
>>>>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
>>>>> unable to access the S3 buckets.
>>>>>
>>>>> Before setting the access key and secret the error was:
>>>>> "java.lang.IllegalArgumentException:
>>>>> AWS Access Key ID and Secret Access Key must be specified as the username
>>>>> or password (respectively) of a s3n URL, or by setting the
>>>>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>>>>> (respectively)."
>>>>>
>>>>> After setting the access key and secret, the error is: "The AWS
>>>>> Access Key Id you provided does not exist in our records."
>>>>>
>>>>> The id/secret being set are the right values. This makes me believe
>>>>> that something else ("token", etc.) needs to be set as well.
>>>>> Any help is appreciated.
>>>>>
>>>>>
>>>>> - Ranga
>>>>>
>>>>
>>>>
>>>
>>
>
