Re: Standardized Spark dev environment
Yep, I think it's only useful (and likely to be maintained) if we actually use this on Jenkins. So that was my proposal. Basically give people a docker file so they can understand exactly what versions of everything we use for our reference build. And if they don't want to use docker directly, this will at least serve as an up-to-date list of packages/versions they should try to install locally in whatever environment they have. - Patrick On Wed, Jan 21, 2015 at 5:42 AM, Will Benton wi...@redhat.com wrote: - Original Message - From: Patrick Wendell pwend...@gmail.com To: Sean Owen so...@cloudera.com Cc: dev dev@spark.apache.org, jay vyas jayunit100.apa...@gmail.com, Paolo Platter paolo.plat...@agilelab.it, Nicholas Chammas nicholas.cham...@gmail.com, Will Benton wi...@redhat.com Sent: Wednesday, January 21, 2015 2:09:35 AM Subject: Re: Standardized Spark dev environment But the issue is when users can't reproduce Jenkins failures. Yeah, to answer Sean's question, this was part of the problem I was trying to solve. The other part was teasing out differences between the Fedora Java environment and a more conventional Java environment. I agree with Sean (and I think this is your suggestion as well, Patrick) that making the environment Jenkins runs a standard image that is available for public consumption would be useful in general. best, wb - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Standardized Spark dev environment
If the goal is a reproducible test environment then I think that is what Jenkins is. Granted you can only ask it for a test. But presumably you get the same result if you start from the same VM image as Jenkins and run the same steps. But the issue is when users can't reproduce Jenkins failures. We don't publish anywhere what the exact set of packages and versions is that is installed on Jenkins. And it can change since it's a shared infrastructure with other projects. So why not publish this manifest as a docker file and then have it run on jenkins using that image? My point is that this VM image + steps is not public anywhere. I bet it is not hard to set up and maintain. I bet it is easier than a VM. But unless Jenkins is using it aren't we just making another different standard build env in an effort to standardize? If it is not the same then it loses value as being exactly the same as the reference build env. Has a problem come up that this solves? Right now the reference build env is an AMI I created and keep adding stuff to when Spark gets new dependencies (e.g. the version of ruby we need to create the docs, new python stats libraries, etc). So if we had a docker image, then I would use that for making the RC's as well and it could serve as a definitive reference for people who want to understand exactly what set of things they need to build Spark. If the goal is just easing developer set up then what does a Docker image do - what does it set up for me? I don't know of stuff I need set up on OS X for me beyond the IDE. There are actually a good number of packages you need to do a full build of Spark including a compliant python version, Java version, certain python packages, ruby and jekyll stuff for the docs, etc (mentioned a bit earlier). - Patrick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Standardized Spark dev environment
What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick
Re: Standardized Spark dev environment
Great suggestion. On Jan 20, 2015 7:14 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick
Re: Standardized Spark dev environment
How many profiles (hadoop / hive /scala) would this development environment support ? Cheers On Tue, Jan 20, 2015 at 4:13 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick
Re: Standardized Spark dev environment
How many profiles (hadoop / hive /scala) would this development environment support ? As many as we want. We probably want to cover a good chunk of the build matrix https://issues.apache.org/jira/browse/SPARK-2004 that Spark officially supports. What does this provide, concretely? It provides a reliable way to create a “good” Spark development environment. Roughly speaking, this probably should mean an environment that matches Jenkins, since that’s where we run “official” testing and builds. For example, Spark has to run on Java 6 and Python 2.6. When devs build and run Spark locally, we can make sure they’re doing it on these versions of the languages with a simple vagrant up. Nate, could you comment on how something like this would relate to the Bigtop effort? http://chapeau.freevariable.com/2014/08/jvm-test-docker.html Will, that’s pretty sweet. I tried something similar a few months ago as an experiment to try building/testing Spark within a container. Here’s the shell script I used https://gist.github.com/nchammas/60b04141f3b9f053faaa against the base CentOS Docker image to setup an environment ready to build and test Spark. We want to run Spark unit tests within containers on Jenkins, so it might make sense to develop a single Docker image that can be used as both a “dev environment” as well as execution container on Jenkins. Perhaps that’s the approach to take instead of looking into Vagrant. Nick On Tue Jan 20 2015 at 8:22:41 PM Will Benton wi...@redhat.com wrote: Hey Nick, I did something similar with a Docker image last summer; I haven't updated the images to cache the dependencies for the current Spark master, but it would be trivial to do so: http://chapeau.freevariable.com/2014/08/jvm-test-docker.html best, wb - Original Message - From: Nicholas Chammas nicholas.cham...@gmail.com To: Spark dev list dev@spark.apache.org Sent: Tuesday, January 20, 2015 6:13:31 PM Subject: Standardized Spark dev environment What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick
Re: Standardized Spark dev environment
I can comment on both... hi will and nate :) 1) Will's Dockerfile solution is the most simple direct solution to the dev environment question : its a efficient way to build and develop spark environments for dev/test.. It would be cool to put that Dockerfile (and/or maybe a shell script which uses it) in the top level of spark as the build entry point. For total platform portability, u could wrap in a vagrantfile to launch a lightweight vm, so that windows worked equally well. 2) However, since nate mentioned vagrant and bigtop, i have to chime in :) the vagrant recipes in bigtop are a nice reference deployment of how to deploy spark in a heterogenous hadoop style environment, and tighter integration testing w/ bigtop for spark releases would be lovely ! The vagrant stuff use puppet to deploy an n node VM or docker based cluster, in which users can easily select components (including spark,yarn,hbase,hadoop,etc...) by simnply editing a YAML file : https://github.com/apache/bigtop/blob/master/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml As nate said, it would be alot of fun to get more cross collaboration between the spark and bigtop communities. Input on how we can better integrate spark (wether its spork, hbase integration, smoke tests aroudn the mllib stuff, or whatever, is always welcome ) On Tue, Jan 20, 2015 at 10:21 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: How many profiles (hadoop / hive /scala) would this development environment support ? As many as we want. We probably want to cover a good chunk of the build matrix https://issues.apache.org/jira/browse/SPARK-2004 that Spark officially supports. What does this provide, concretely? It provides a reliable way to create a “good” Spark development environment. Roughly speaking, this probably should mean an environment that matches Jenkins, since that’s where we run “official” testing and builds. For example, Spark has to run on Java 6 and Python 2.6. When devs build and run Spark locally, we can make sure they’re doing it on these versions of the languages with a simple vagrant up. Nate, could you comment on how something like this would relate to the Bigtop effort? http://chapeau.freevariable.com/2014/08/jvm-test-docker.html Will, that’s pretty sweet. I tried something similar a few months ago as an experiment to try building/testing Spark within a container. Here’s the shell script I used https://gist.github.com/nchammas/60b04141f3b9f053faaa against the base CentOS Docker image to setup an environment ready to build and test Spark. We want to run Spark unit tests within containers on Jenkins, so it might make sense to develop a single Docker image that can be used as both a “dev environment” as well as execution container on Jenkins. Perhaps that’s the approach to take instead of looking into Vagrant. Nick On Tue Jan 20 2015 at 8:22:41 PM Will Benton wi...@redhat.com wrote: Hey Nick, I did something similar with a Docker image last summer; I haven't updated the images to cache the dependencies for the current Spark master, but it would be trivial to do so: http://chapeau.freevariable.com/2014/08/jvm-test-docker.html best, wb - Original Message - From: Nicholas Chammas nicholas.cham...@gmail.com To: Spark dev list dev@spark.apache.org Sent: Tuesday, January 20, 2015 6:13:31 PM Subject: Standardized Spark dev environment What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick -- jay vyas
RE: Standardized Spark dev environment
If there is some interest in more standardization and setup of dev/test environments spark community might be interested in starting to participate in apache bigtop effort: http://bigtop.apache.org/ While the project had its start and initial focus on packaging, testing, deploying Hadoop/hdfs related stack its looking like we will be targeting data engineers going forward, thus spark is looking to become bigger central piece to bigtop effort as the project moves towards a v1 release. We will be doing a bigtop/bigdata workshop late Feb at the SocalLinux Conference: http://www.socallinuxexpo.org/scale/13x Right now scoping some content that will be getting started spark related for the event, targeted intro of bigtop/spark puppet powered deployment components going into the event as well. Also the group will be holding a meetup at Amazon's Palo Alto office on Jan 27th if any folks are interested. Nate -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: Tuesday, January 20, 2015 5:09 PM To: Nicholas Chammas Cc: dev Subject: Re: Standardized Spark dev environment My concern would mostly be maintenance. It adds to an already very complex build. It only assists developers who are a small audience. What does this provide, concretely? On Jan 21, 2015 12:14 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Standardized Spark dev environment
Hey Nick, I did something similar with a Docker image last summer; I haven't updated the images to cache the dependencies for the current Spark master, but it would be trivial to do so: http://chapeau.freevariable.com/2014/08/jvm-test-docker.html best, wb - Original Message - From: Nicholas Chammas nicholas.cham...@gmail.com To: Spark dev list dev@spark.apache.org Sent: Tuesday, January 20, 2015 6:13:31 PM Subject: Standardized Spark dev environment What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Standardized Spark dev environment
To respond to the original suggestion by Nick. I always thought it would be useful to have a Docker image on which we run the tests and build releases, so that we could have a consistent environment that other packagers or people trying to exhaustively run Spark tests could replicate (or at least look at) to understand exactly how we recommend building Spark. Sean - do you think that is too high of overhead? In terms of providing images that we encourage as standard deployment images of Spark and want to make portable across environments, that's a much larger project and one with higher associated maintenance overhead. So I'd be interested in seeing that evolve as its own project (spark-deploy) or something associated with bigtop, etc. - Patrick On Tue, Jan 20, 2015 at 10:30 PM, Paolo Platter paolo.plat...@agilelab.it wrote: Hi all, I also tried the docker way and it works well. I suggest to look at sequenceiq/spark dockers, they are very active on that field. Paolo Inviata dal mio Windows Phone Da: jay vyasmailto:jayunit100.apa...@gmail.com Inviato: 21/01/2015 04:45 A: Nicholas Chammasmailto:nicholas.cham...@gmail.com Cc: Will Bentonmailto:wi...@redhat.com; Spark dev listmailto:dev@spark.apache.org Oggetto: Re: Standardized Spark dev environment I can comment on both... hi will and nate :) 1) Will's Dockerfile solution is the most simple direct solution to the dev environment question : its a efficient way to build and develop spark environments for dev/test.. It would be cool to put that Dockerfile (and/or maybe a shell script which uses it) in the top level of spark as the build entry point. For total platform portability, u could wrap in a vagrantfile to launch a lightweight vm, so that windows worked equally well. 2) However, since nate mentioned vagrant and bigtop, i have to chime in :) the vagrant recipes in bigtop are a nice reference deployment of how to deploy spark in a heterogenous hadoop style environment, and tighter integration testing w/ bigtop for spark releases would be lovely ! The vagrant stuff use puppet to deploy an n node VM or docker based cluster, in which users can easily select components (including spark,yarn,hbase,hadoop,etc...) by simnply editing a YAML file : https://github.com/apache/bigtop/blob/master/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml As nate said, it would be alot of fun to get more cross collaboration between the spark and bigtop communities. Input on how we can better integrate spark (wether its spork, hbase integration, smoke tests aroudn the mllib stuff, or whatever, is always welcome ) On Tue, Jan 20, 2015 at 10:21 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: How many profiles (hadoop / hive /scala) would this development environment support ? As many as we want. We probably want to cover a good chunk of the build matrix https://issues.apache.org/jira/browse/SPARK-2004 that Spark officially supports. What does this provide, concretely? It provides a reliable way to create a good Spark development environment. Roughly speaking, this probably should mean an environment that matches Jenkins, since that's where we run official testing and builds. For example, Spark has to run on Java 6 and Python 2.6. When devs build and run Spark locally, we can make sure they're doing it on these versions of the languages with a simple vagrant up. Nate, could you comment on how something like this would relate to the Bigtop effort? http://chapeau.freevariable.com/2014/08/jvm-test-docker.html Will, that's pretty sweet. I tried something similar a few months ago as an experiment to try building/testing Spark within a container. Here's the shell script I used https://gist.github.com/nchammas/60b04141f3b9f053faaa against the base CentOS Docker image to setup an environment ready to build and test Spark. We want to run Spark unit tests within containers on Jenkins, so it might make sense to develop a single Docker image that can be used as both a dev environment as well as execution container on Jenkins. Perhaps that's the approach to take instead of looking into Vagrant. Nick On Tue Jan 20 2015 at 8:22:41 PM Will Benton wi...@redhat.com wrote: Hey Nick, I did something similar with a Docker image last summer; I haven't updated the images to cache the dependencies for the current Spark master, but it would be trivial to do so: http://chapeau.freevariable.com/2014/08/jvm-test-docker.html best, wb - Original Message - From: Nicholas Chammas nicholas.cham...@gmail.com To: Spark dev list dev@spark.apache.org Sent: Tuesday, January 20, 2015 6:13:31 PM Subject: Standardized Spark dev environment What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make
R: Standardized Spark dev environment
Hi all, I also tried the docker way and it works well. I suggest to look at sequenceiq/spark dockers, they are very active on that field. Paolo Inviata dal mio Windows Phone Da: jay vyasmailto:jayunit100.apa...@gmail.com Inviato: 21/01/2015 04:45 A: Nicholas Chammasmailto:nicholas.cham...@gmail.com Cc: Will Bentonmailto:wi...@redhat.com; Spark dev listmailto:dev@spark.apache.org Oggetto: Re: Standardized Spark dev environment I can comment on both... hi will and nate :) 1) Will's Dockerfile solution is the most simple direct solution to the dev environment question : its a efficient way to build and develop spark environments for dev/test.. It would be cool to put that Dockerfile (and/or maybe a shell script which uses it) in the top level of spark as the build entry point. For total platform portability, u could wrap in a vagrantfile to launch a lightweight vm, so that windows worked equally well. 2) However, since nate mentioned vagrant and bigtop, i have to chime in :) the vagrant recipes in bigtop are a nice reference deployment of how to deploy spark in a heterogenous hadoop style environment, and tighter integration testing w/ bigtop for spark releases would be lovely ! The vagrant stuff use puppet to deploy an n node VM or docker based cluster, in which users can easily select components (including spark,yarn,hbase,hadoop,etc...) by simnply editing a YAML file : https://github.com/apache/bigtop/blob/master/bigtop-deploy/vm/vagrant-puppet/vagrantconfig.yaml As nate said, it would be alot of fun to get more cross collaboration between the spark and bigtop communities. Input on how we can better integrate spark (wether its spork, hbase integration, smoke tests aroudn the mllib stuff, or whatever, is always welcome ) On Tue, Jan 20, 2015 at 10:21 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: How many profiles (hadoop / hive /scala) would this development environment support ? As many as we want. We probably want to cover a good chunk of the build matrix https://issues.apache.org/jira/browse/SPARK-2004 that Spark officially supports. What does this provide, concretely? It provides a reliable way to create a “good” Spark development environment. Roughly speaking, this probably should mean an environment that matches Jenkins, since that’s where we run “official” testing and builds. For example, Spark has to run on Java 6 and Python 2.6. When devs build and run Spark locally, we can make sure they’re doing it on these versions of the languages with a simple vagrant up. Nate, could you comment on how something like this would relate to the Bigtop effort? http://chapeau.freevariable.com/2014/08/jvm-test-docker.html Will, that’s pretty sweet. I tried something similar a few months ago as an experiment to try building/testing Spark within a container. Here’s the shell script I used https://gist.github.com/nchammas/60b04141f3b9f053faaa against the base CentOS Docker image to setup an environment ready to build and test Spark. We want to run Spark unit tests within containers on Jenkins, so it might make sense to develop a single Docker image that can be used as both a “dev environment” as well as execution container on Jenkins. Perhaps that’s the approach to take instead of looking into Vagrant. Nick On Tue Jan 20 2015 at 8:22:41 PM Will Benton wi...@redhat.com wrote: Hey Nick, I did something similar with a Docker image last summer; I haven't updated the images to cache the dependencies for the current Spark master, but it would be trivial to do so: http://chapeau.freevariable.com/2014/08/jvm-test-docker.html best, wb - Original Message - From: Nicholas Chammas nicholas.cham...@gmail.com To: Spark dev list dev@spark.apache.org Sent: Tuesday, January 20, 2015 6:13:31 PM Subject: Standardized Spark dev environment What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagrant, we may even be able to make it so that a single Vagrantfile creates equivalent development environments across OS X, Linux, and Windows, without having to do much (or any) OS-specific work. I imagine for committers and regular contributors, this exercise may seem pointless, since y'all are probably already very comfortable with your workflow. I wonder, though, if any of you think this would be worthwhile as a improvement to the new Spark developer experience. Nick -- jay vyas
