Re: Spark, Mesos, Docker and S3

2016-01-29 Thread Mao Geng
esos slaves also run Spark executor docker container in >>>>>>> --net=host mode, so that the AWS profile of Mesos slaves will take >>>>>>> effect. >>>>>>> >>>>>>> Hope it helps, >>>>>>

Re: Spark, Mesos, Docker and S3

2016-01-29 Thread Sathish Kumaran Vairavelu
ogic.com> wrote: >>>>> >>>>>> Hi Sathish, >>>>>> >>>>>> The docker image is normal, no AWS profile included. >>>>>> >>>>>> When the driver container runs with --net=host, the driver host's AWS >&g

Re: Spark, Mesos, Docker and S3

2016-01-28 Thread Sathish Kumaran Vairavelu
runs with --net=host, the driver host's AWS >>>> profile will take effect so that the driver can access the protected s3 >>>> files. >>>> >>>> Similarly, Mesos slaves also run Spark executor docker container in >>>> --net=host mode, so that th

Re: Spark, Mesos, Docker and S3

2016-01-28 Thread Mao Geng
host's AWS >>>>> profile will take effect so that the driver can access the protected s3 >>>>> files. >>>>> >>>>> Similarly, Mesos slaves also run Spark executor docker container in >>>>> --net=host mode,

Re: Spark, Mesos, Docker and S3

2016-01-28 Thread Mao Geng
es also run Spark executor docker container in >>> --net=host mode, so that the AWS profile of Mesos slaves will take effect. >>> >>> Hope it helps, >>> Mao >>> >>> On Jan 26, 2016, at 9:15 PM, Sathish Kumaran Vairavelu < >>> vsathish

Re: Spark, Mesos, Docker and S3

2016-01-27 Thread Sathish Kumaran Vairavelu
On Jan 26, 2016, at 9:15 PM, Sathish Kumaran Vairavelu < >> vsathishkuma...@gmail.com> wrote: >> >> Hi Mao, >> >> I want to check on accessing the S3 from Spark docker in Mesos. The EC2 >> instance that I am using has the AWS profile/IAM included.

Re: Spark, Mesos, Docker and S3

2016-01-27 Thread Sathish Kumaran Vairavelu
Jan 26, 2016, at 9:15 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > > Hi Mao, > > I want to check on accessing the S3 from Spark docker in Mesos. The EC2 > instance that I am using has the AWS profile/IAM included. Should we build > the docker image with any A

Re: Spark, Mesos, Docker and S3

2016-01-27 Thread Mao Geng
=host mode, so that the AWS profile of Mesos slaves will take effect. Hope it helps, Mao > On Jan 26, 2016, at 9:15 PM, Sathish Kumaran Vairavelu > <vsathishkuma...@gmail.com> wrote: > > Hi Mao, > > I want to check on accessing the S3 from Spark docker in Mesos. The E

Re: Spark, Mesos, Docker and S3

2016-01-26 Thread Sathish Kumaran Vairavelu
Hi Mao, I want to check on accessing the S3 from Spark docker in Mesos. The EC2 instance that I am using has the AWS profile/IAM included. Should we build the docker image with any AWS profile settings or --net=host docker option takes care of it? Please help Thanks Sathish On Tue, Jan 26

Spark, Mesos, Docker and S3

2016-01-26 Thread Mao Geng
Hi there, I am trying to run Spark on Mesos using a Docker image as executor, as mentioned http://spark.apache.org/docs/latest/running-on-mesos.html#mesos-docker-support . I built a docker image using the following Dockerfile (which is based on

Re: Spark, Mesos, Docker and S3

2016-01-26 Thread Jerry Lam
Hi Mao, Can you try --jars to include those jars? Best Regards, Jerry Sent from my iPhone > On 26 Jan, 2016, at 7:02 pm, Mao Geng wrote: > > Hi there, > > I am trying to run Spark on Mesos using a Docker image as executor, as > mentioned >

Re: Spark, Mesos, Docker and S3

2016-01-26 Thread Mao Geng
image already has them), but that's ok - I am happy seeing Spark + Mesos + Docker + S3 worked together! Thanks, Mao 16/01/27 02:54:45 INFO Executor: Using REPL class URI: http://172.16.3.98:33771 16/01/27 02:55:12 INFO CoarseGrainedExecutorBackend: Got assigned task 0 16/01/27 02:55:12 INFO Exec