Re: use S3-Compatible Storage with spark

2015-07-28 Thread Schmirr Wurst
Hi recompiled and retried, now its looking like this with s3a : com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain S3n is working find, (only problem is still the endpoint) - To

Re: use S3-Compatible Storage with spark

2015-07-28 Thread Akhil Das
With s3n try this out: *s3service.s3-endpoint*The host name of the S3 service. You should only ever change this value from the default if you need to contact an alternative S3 endpoint for testing purposes. Default: s3.amazonaws.com Thanks Best Regards On Tue, Jul 28, 2015 at 1:54 PM, Schmirr

Re: use S3-Compatible Storage with spark

2015-07-28 Thread Schmirr Wurst
I tried those 3 possibilities, and everything is working = endpoint param is not working : sc.hadoopConfiguration.set(s3service.s3-endpoint,test) sc.hadoopConfiguration.set(fs.s3n.endpoint,test) sc.hadoopConfiguration.set(fs.s3n.s3-endpoint,test) 2015-07-28 10:28 GMT+02:00 Akhil Das

Re: use S3-Compatible Storage with spark

2015-07-27 Thread Schmirr Wurst
...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere

Re: use S3-Compatible Storage with spark

2015-07-27 Thread Akhil Das
to hit. Thanks Best Regards On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can

Re: use S3-Compatible Storage with spark

2015-07-27 Thread Akhil Das
: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere ? - To unsubscribe, e-mail: user

Re: use S3-Compatible Storage with spark

2015-07-27 Thread Schmirr Wurst
: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere ? - To unsubscribe, e-mail: user-unsubscr

Re: use S3-Compatible Storage with spark

2015-07-22 Thread Schmirr Wurst
your spark job add --jars path/to/thejar From: Schmirr Wurst schmirrwu...@gmail.com Sent: Wednesday, July 22, 2015 12:06 PM To: Thomas Demoor Subject: Re: use S3-Compatible Storage with spark Hi Thomas, thanks, could you just tell me what exaclty I

Re: use S3-Compatible Storage with spark

2015-07-21 Thread Akhil Das
provides a S3 like RestAPI endpoint for you to hit. Thanks Best Regards On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon

Re: use S3-Compatible Storage with spark

2015-07-21 Thread Schmirr Wurst
, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere

Re: use S3-Compatible Storage with spark

2015-07-21 Thread Akhil Das
Best Regards On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere

Re: use S3-Compatible Storage with spark

2015-07-21 Thread Schmirr Wurst
provides a S3 like RestAPI endpoint for you to hit. Thanks Best Regards On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point

Re: use S3-Compatible Storage with spark

2015-07-20 Thread Akhil Das
: Could you name the Storage service that you are using? Most of them provides a S3 like RestAPI endpoint for you to hit. Thanks Best Regards On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark

Re: use S3-Compatible Storage with spark

2015-07-20 Thread Schmirr Wurst
are using? Most of them provides a S3 like RestAPI endpoint for you to hit. Thanks Best Regards On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema

Re: use S3-Compatible Storage with spark

2015-07-19 Thread Akhil Das
Could you name the Storage service that you are using? Most of them provides a S3 like RestAPI endpoint for you to hit. Thanks Best Regards On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n

Fwd: use S3-Compatible Storage with spark

2015-07-19 Thread Schmirr Wurst
On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere

use S3-Compatible Storage with spark

2015-07-17 Thread Schmirr Wurst
Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere ? - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

Re: use S3-Compatible Storage with spark

2015-07-17 Thread Sujit Pal
, 2015 at 1:36 AM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible Storage in Spark ? If I'm using s3n:// url schema, the it will point to amazon, is there a way I can specify the host somewhere

Re: use S3-Compatible Storage with spark

2015-07-17 Thread Ankur Chauhan
}[/${subfolder_name}]*. Bucket names are unique across S3, so the resulting path is also unique. There is no concept of hostname in s3 urls as far as I know. -sujit On Fri, Jul 17, 2015 at 1:36 AM, Schmirr Wurst schmirrwu...@gmail.com wrote: Hi, I wonder how to use S3 compatible