Usually you can just run Spark in local mode on a single machine for most dev/testing.
If you want to simulate a cluster locally using multiple Spark worker processes, you can use the undocumented local cluster mode, e.g. local-cluster[2,1,512] this launches two worker processes, each with one core and 512m of ram. On Wed, Apr 9, 2014 at 3:00 PM, Sujeet Varakhedi <svsuj...@gmail.com> wrote: > Another starter question which probably should have asked before is what is > the most efficient way to iterate quickly on dev/test. I am currently using > a local cluster (via vagrant and shared folders) and also spark-shell. > > Sujeet > > > On Tue, Apr 8, 2014 at 9:50 AM, Michael Ernest <mfern...@cloudera.com > >wrote: > > > Ha ha! nice try, sheepherder! ;-) > > > > > > On Tue, Apr 8, 2014 at 12:37 PM, Matei Zaharia <matei.zaha...@gmail.com > > >wrote: > > > > > Shh, maybe I really wanted people to fix that one issue. > > > > > > On Apr 8, 2014, at 9:34 AM, Aaron Davidson <ilike...@gmail.com> wrote: > > > > > > > Matei's link seems to point to a specific starter project as part of > > the > > > > starter list, but here is the list itself: > > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened) > > > > > > > > > > > > On Mon, Apr 7, 2014 at 10:22 PM, Matei Zaharia < > > matei.zaha...@gmail.com > > > >wrote: > > > > > > > >> I'd suggest looking for the issues labeled "Starter" on JIRA. You > can > > > find > > > >> them here: > > > >> > > > > > > https://issues.apache.org/jira/browse/SPARK-1438?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened) > > > >> > > > >> Matei > > > >> > > > >> On Apr 7, 2014, at 9:45 PM, Mukesh G <muk...@gmail.com> wrote: > > > >> > > > >>> Hi Sujeet, > > > >>> > > > >>> Thanks. I went thru the website and looks great. Is there a list > of > > > >>> items that I can choose from, for contribution? > > > >>> > > > >>> Thanks > > > >>> > > > >>> Mukesh > > > >>> > > > >>> > > > >>> On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi > > > >>> <svarakh...@gopivotal.com>wrote: > > > >>> > > > >>>> This is a good place to start: > > > >>>> > > > > https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark > > > >>>> > > > >>>> Sujeet > > > >>>> > > > >>>> > > > >>>> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <muk...@gmail.com> > wrote: > > > >>>> > > > >>>>> Hi, > > > >>>>> > > > >>>>> How I contribute to Spark and it's associated projects? > > > >>>>> > > > >>>>> Appreciate the help... > > > >>>>> > > > >>>>> Thanks > > > >>>>> > > > >>>>> Mukesh > > > >>>>> > > > >>>> > > > >> > > > >> > > > > > > > > > > > > -- > > Michael Ernest > > Sr. Solutions Consultant > > West Coast > > >
