Hi Daniil Could you provide some more details on how the cluster should be launched/configured? The EC2 instance that I am dealing with uses the concept of IAMRoles. I don't have any "keyfile" to specify to the spark-ec2 script. Thanks for your help.
- Ranga On Mon, Oct 13, 2014 at 3:04 PM, Daniil Osipov <daniil.osi...@shazam.com> wrote: > (Copying the user list) > You should use spark_ec2 script to configure the cluster. If you use trunk > version you can use the new --copy-aws-credentials option to configure the > S3 parameters automatically, otherwise either include them in your > SparkConf variable or add them to > /root/spark/ephemeral-hdfs/conf/core-site.xml > > On Mon, Oct 13, 2014 at 2:56 PM, Ranga <sra...@gmail.com> wrote: > >> The cluster is deployed on EC2 and I am trying to access the S3 files >> from within a spark-shell session. >> >> On Mon, Oct 13, 2014 at 2:51 PM, Daniil Osipov <daniil.osi...@shazam.com> >> wrote: >> >>> So is your cluster running on EC2, or locally? If you're running >>> locally, you should still be able to access S3 files, you just need to >>> locate the core-site.xml and add the parameters as defined in the error. >>> >>> On Mon, Oct 13, 2014 at 2:49 PM, Ranga <sra...@gmail.com> wrote: >>> >>>> Hi Daniil >>>> >>>> No. I didn't create the spark-cluster using the ec2 scripts. Is that >>>> something that I need to do? I just downloaded Spark-1.1.0 and Hadoop-2.4. >>>> However, I am trying to access files on S3 from this cluster. >>>> >>>> >>>> - Ranga >>>> >>>> On Mon, Oct 13, 2014 at 2:36 PM, Daniil Osipov < >>>> daniil.osi...@shazam.com> wrote: >>>> >>>>> Did you add the fs.s3n.aws* configuration parameters in >>>>> /root/spark/ephemeral-hdfs/conf/core-ste.xml? >>>>> >>>>> On Mon, Oct 13, 2014 at 11:03 AM, Ranga <sra...@gmail.com> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> I am trying to access files/buckets in S3 and encountering a >>>>>> permissions issue. The buckets are configured to authenticate using an >>>>>> IAMRole provider. >>>>>> I have set the KeyId and Secret using environment variables ( >>>>>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still >>>>>> unable to access the S3 buckets. >>>>>> >>>>>> Before setting the access key and secret the error was: >>>>>> "java.lang.IllegalArgumentException: >>>>>> AWS Access Key ID and Secret Access Key must be specified as the username >>>>>> or password (respectively) of a s3n URL, or by setting the >>>>>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties >>>>>> (respectively)." >>>>>> >>>>>> After setting the access key and secret, the error is: "The AWS >>>>>> Access Key Id you provided does not exist in our records." >>>>>> >>>>>> The id/secret being set are the right values. This makes me believe >>>>>> that something else ("token", etc.) needs to be set as well. >>>>>> Any help is appreciated. >>>>>> >>>>>> >>>>>> - Ranga >>>>>> >>>>> >>>>> >>>> >>> >> >