Re: Role-based S3 access outside of EMR

2016-08-14 Thread Steve Loughran
On 29 Jul 2016, at 00:07, Everett Anderson > wrote: Hey, Just wrapping this up -- I ended up following the instructions to build a custom Spark release with Hadoop 2.7.2,

Re: Role-based S3 access outside of EMR

2016-07-28 Thread Everett Anderson
rson [mailto:ever...@nuna.com.INVALID > <ever...@nuna.com.INVALID>] > *Sent:* 21 July 2016 17:01 > *To:* Gourav Sengupta <gourav.sengu...@gmail.com> > *Cc:* Teng Qiu <teng...@gmail.com>; Andy Davidson < > a...@santacruzintegration.com>; user <user@s

Re: Role-based S3 access outside of EMR

2016-07-23 Thread Steve Loughran
dson <a...@santacruzintegration.com<mailto:a...@santacruzintegration.com>>; user <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Role-based S3 access outside of EMR Hey, FWIW, we are using EMR, actually, in production. The main case I have for wa

RE: Role-based S3 access outside of EMR

2016-07-21 Thread Ewan Leith
Sengupta <gourav.sengu...@gmail.com> Cc: Teng Qiu <teng...@gmail.com>; Andy Davidson <a...@santacruzintegration.com>; user <user@spark.apache.org> Subject: Re: Role-based S3 access outside of EMR Hey, FWIW, we are using EMR, actually, in production. The main case I have

Re: Role-based S3 access outside of EMR

2016-07-21 Thread Everett Anderson
;> >>> local machine >> >>> >> >>> My main() has an optional command line argument ‘- - runLocal’ >> Normally I >> >>> load data from either hdfs:/// or S3n:// . If the arg is set I read >> from >> >>> file:/// >

Re: Role-based S3 access outside of EMR

2016-07-21 Thread Gourav Sengupta
r S3n:// . If the arg is set I read > from > >>> file:/// > >>> > >>> Sometime I use a CLI arg ‘- -dataFileURL’ > >>> > >>> So in your case I would log into my data cluster and use “AWS s3 cp" to > >>> copy the data into

Re: Role-based S3 access outside of EMR

2016-07-21 Thread Teng Qiu
// >>> >>> Sometime I use a CLI arg ‘- -dataFileURL’ >>> >>> So in your case I would log into my data cluster and use “AWS s3 cp" to >>> copy the data into my cluster and then use “SCP” to copy the data from the >>> data center back to my lo

Re: Role-based S3 access outside of EMR

2016-07-21 Thread Gourav Sengupta
>> copy the data into my cluster and then use “SCP” to copy the data from the >> data center back to my local env. >> >> Andy >> >> From: Everett Anderson <ever...@nuna.com.INVALID> >> Date: Tuesday, July 19, 2016 at 2:30 PM >> To: "user @spark&qu

Re: Role-based S3 access outside of EMR

2016-07-20 Thread Everett Anderson
the data from the > data center back to my local env. > > Andy > > From: Everett Anderson <ever...@nuna.com.INVALID> > Date: Tuesday, July 19, 2016 at 2:30 PM > To: "user @spark" <user@spark.apache.org> > Subject: Role-based S3 access outside o

Re: Role-based S3 access outside of EMR

2016-07-19 Thread Andy Davidson
<ever...@nuna.com.INVALID> Date: Tuesday, July 19, 2016 at 2:30 PM To: "user @spark" <user@spark.apache.org> Subject: Role-based S3 access outside of EMR > Hi, > > When running on EMR, AWS configures Hadoop to use their EMRFS Hadoop > FileSystem implementat

Role-based S3 access outside of EMR

2016-07-19 Thread Everett Anderson
Hi, When running on EMR, AWS configures Hadoop to use their EMRFS Hadoop FileSystem implementation for s3:// URLs and seems to install the necessary S3 credentials properties, as well. Often, it's nice during development to run outside of a cluster even with the "local" Spark master, though,