Try the following:

1. Set the access key and secret key in the sparkContext:

ssc.sparkContext.hadoopConfiguration.set("AWS_ACCESS_KEY_ID",yourAccessKey)

ssc.sparkContext.hadoopConfiguration.set("AWS_SECRET_ACCESS_KEY",yourSecretKey)



2. Set the access key and secret key in the environment before starting
your application:

​
export AWS_ACCESS_KEY_ID=<your access>

export AWS_SECRET_ACCESS_KEY=<your secret>​



3. Set the access key and secret key inside the hadoop configurations

val hadoopConf=ssc.sparkContext.hadoopConfiguration;
hadoopConf.set("fs.s3.impl","org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)


4. You can also try:

val stream = ssc.textFileStream("s3n://yourAccessKey:yourSecretKey@
<yourBucket>/path/")


Thanks
Best Regards

On Tue, Feb 10, 2015 at 8:27 PM, Marc Limotte <mslimo...@gmail.com> wrote:

> I see that StreamingContext has a hadoopConfiguration() method, which can
> be used like this sample I found:
>
> sc.hadoopConfiguration().set("fs.s3.awsAccessKeyId", "XXXXXX");
>> sc.hadoopConfiguration().set("fs.s3.awsSecretAccessKey", "XXXXXX");
>
>
> But StreamingContext doesn't have the same thing.  I want to use a
> StreamingContext with s3n: text file input, but can't find a way to set the
> AWS credentials.  I also tried (with no success):
>
>
>    - adding the properties to conf/spark-defaults.conf
>    - $HADOOP_HOME/conf/hdfs-site.xml
>    - ENV variables
>    - Embedded as user:password in s3n://user:password@... (w/ url
>    encoding)
>    - Setting the conf as above on a new SparkContext and passing that the
>    StreamingContext constructor: StreamingContext(sparkContext: SparkContext,
>    batchDuration: Duration)
>
> Can someone point me in the right direction for setting AWS creds (hadoop
> conf options) for streamingcontext?
>
> thanks,
> Marc Limotte
> Climate Corporation
>

Reply via email to