Try the following: 1. Set the access key and secret key in the sparkContext:
ssc.sparkContext.hadoopConfiguration.set("AWS_ACCESS_KEY_ID",yourAccessKey) ssc.sparkContext.hadoopConfiguration.set("AWS_SECRET_ACCESS_KEY",yourSecretKey) 2. Set the access key and secret key in the environment before starting your application: export AWS_ACCESS_KEY_ID=<your access> export AWS_SECRET_ACCESS_KEY=<your secret> 3. Set the access key and secret key inside the hadoop configurations val hadoopConf=ssc.sparkContext.hadoopConfiguration; hadoopConf.set("fs.s3.impl","org.apache.hadoop.fs.s3native.NativeS3FileSystem") hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey) hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey) 4. You can also try: val stream = ssc.textFileStream("s3n://yourAccessKey:yourSecretKey@ <yourBucket>/path/") Thanks Best Regards On Tue, Feb 10, 2015 at 8:27 PM, Marc Limotte <mslimo...@gmail.com> wrote: > I see that StreamingContext has a hadoopConfiguration() method, which can > be used like this sample I found: > > sc.hadoopConfiguration().set("fs.s3.awsAccessKeyId", "XXXXXX"); >> sc.hadoopConfiguration().set("fs.s3.awsSecretAccessKey", "XXXXXX"); > > > But StreamingContext doesn't have the same thing. I want to use a > StreamingContext with s3n: text file input, but can't find a way to set the > AWS credentials. I also tried (with no success): > > > - adding the properties to conf/spark-defaults.conf > - $HADOOP_HOME/conf/hdfs-site.xml > - ENV variables > - Embedded as user:password in s3n://user:password@... (w/ url > encoding) > - Setting the conf as above on a new SparkContext and passing that the > StreamingContext constructor: StreamingContext(sparkContext: SparkContext, > batchDuration: Duration) > > Can someone point me in the right direction for setting AWS creds (hadoop > conf options) for streamingcontext? > > thanks, > Marc Limotte > Climate Corporation >