If you are on EMR, these can go into your hdfs site config. And will work with Spark on YARN by default.
Regards Sab On 11-Jan-2016 5:16 pm, "Krishna Rao" <[email protected]> wrote: > Hi all, > > Is there a method for reading from s3 without having to hard-code keys? > The only 2 ways I've found both require this: > > 1. Set conf in code e.g.: > sc.hadoopConfiguration().set("fs.s3.awsAccessKeyId", "<aws_key>") > sc.hadoopConfiguration().set("fs.s3.awsSecretAccessKey", > "<aws_secret_key>") > > 2. Set keys in URL, e.g.: > sc.textFile("s3n://<aws_key>@<aws_secret_key>/bucket/test/testdata") > > > Both if which I'm reluctant to do within production code! > > > Cheers >
