Re: Standardized Spark dev environment

o: "Sean Owen" >> Cc: "dev" , "jay vyas" , >> "Paolo Platter" >> , "Nicholas Chammas" >> , "Will Benton" >> Sent: Wednesday, January 21, 2015 2:09:35 AM >> Subject: Re: Standardized Spark dev envir

Re: Standardized Spark dev environment

- Original Message - > From: "Patrick Wendell" > To: "Sean Owen" > Cc: "dev" , "jay vyas" , > "Paolo Platter" > , "Nicholas Chammas" , > "Will Benton" > Sent: Wednesday, January 21, 2015 2:09:3

Re: Standardized Spark dev environment

Sure, can Jenkins use this new image too? If not then it doesn't help with reproducing a Jenkins failure, most of which even Jenkins can't reproduce. But if it does and it can be used for builds then that does seem like it is reducing rather than increasing environment configurations which is good.

Re: Standardized Spark dev environment

> If the goal is a reproducible test environment then I think that is what > Jenkins is. Granted you can only ask it for a test. But presumably you get > the same result if you start from the same VM image as Jenkins and run the > same steps. But the issue is when users can't reproduce Jenkins fai

Re: Standardized Spark dev environment

Inviato: 21/01/2015 04:45 > > A: Nicholas Chammas<mailto:nicholas.cham...@gmail.com> > > Cc: Will Benton<mailto:wi...@redhat.com>; Spark dev list dev@spark.apache.org> > > Oggetto: Re: Standardized Spark dev environment > > > > I can comment on both... hi will and

Re: Standardized Spark dev environment

lt;mailto:nicholas.cham...@gmail.com> > Cc: Will Benton<mailto:wi...@redhat.com>; Spark dev > list<mailto:dev@spark.apache.org> > Oggetto: Re: Standardized Spark dev environment > > I can comment on both... hi will and nate :) > > 1) Will's Dockerfile solution

Re: Standardized Spark dev environment

I can comment on both... hi will and nate :) 1) Will's Dockerfile solution is the most simple direct solution to the dev environment question : its a efficient way to build and develop spark environments for dev/test.. It would be cool to put that Dockerfile (and/or maybe a shell script which

Re: Standardized Spark dev environment

How many profiles (hadoop / hive /scala) would this development environment support ? As many as we want. We probably want to cover a good chunk of the build matrix that Spark officially supports. What does this provide, concretely? It provides

Re: Standardized Spark dev environment

Hey Nick, I did something similar with a Docker image last summer; I haven't updated the images to cache the dependencies for the current Spark master, but it would be trivial to do so: http://chapeau.freevariable.com/2014/08/jvm-test-docker.html best, wb - Original Message - > From

RE: Standardized Spark dev environment

Alto office on Jan 27th if any folks are interested. Nate -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: Tuesday, January 20, 2015 5:09 PM To: Nicholas Chammas Cc: dev Subject: Re: Standardized Spark dev environment My concern would mostly be maintenance. It adds t

Re: Standardized Spark dev environment

My concern would mostly be maintenance. It adds to an already very complex build. It only assists developers who are a small audience. What does this provide, concretely? On Jan 21, 2015 12:14 AM, "Nicholas Chammas" wrote: > What do y'all think of creating a standardized Spark development > envir

Re: Standardized Spark dev environment

How many profiles (hadoop / hive /scala) would this development environment support ? Cheers On Tue, Jan 20, 2015 at 4:13 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > What do y'all think of creating a standardized Spark development > environment, perhaps encoded as a Vagrantfile,

Re: Standardized Spark dev environment

Great suggestion. On Jan 20, 2015 7:14 PM, "Nicholas Chammas" wrote: > What do y'all think of creating a standardized Spark development > environment, perhaps encoded as a Vagrantfile, and publishing it under > `dev/`? > > The goal would be to make it easier for new developers to get started with