Also, I'm following to master students at the University of Liège (one for computing prob conditional density on massive data and the other implementing a Markov Chain method on georasters), I proposed them to use the Spark-Notebook to learn the framework, they're quite happy with it (so far at least).
I know it's not a dev option, but it can help for proofing some sequence of steps - it's easier than REPL to run the whole thing when we change intermediate step for instance, and it's lightweight compared to sbt runs specially using testing framework). my0.1c cheers, aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] <http://about.me/noootsab> On Tue, Oct 28, 2014 at 6:01 PM, Matt Narrell <matt.narr...@gmail.com> wrote: > So, Im using Intellij 13.x, and Scala Spark jobs. > > Make sure you have singletons (objects, not classes), then simply debug > the main function. You’ll need to set your master to some derivation of > “local”, but thats it. Spark Streaming is kinda wonky when debugging, but > data-at-rest behaves like you’d expect. > > hth > > mn > > On Oct 27, 2014, at 3:03 PM, Eric Tanner <eric.tan...@justenough.com> > wrote: > > I am a Scala / Spark newbie (attending Paco Nathan's class). > > What I need is some advice as to how to set up intellij (or eclipse) to be > able to attache to the process executing to the debugger. I know that this > is not feasible if the code is executing within the cluster. However, if > spark is running locally (on my laptop) I would like to attach the debugger > process to the spark program that is running locally to be able to step > through the program. > > Any advice will be is helpful. > > Eric > > -- > > > > > > *Eric Tanner*Big Data Developer > > <image005.png> > > 15440 Laguna Canyon, Suite 100 > > Irvine, CA 92618 > > > Cell: > Tel: > Skype: > Web: > > +1 (951) 313-9274 > +1 (949) 706-0400 > e <http://tonya.nicholls.je/>ric.tanner.je > www.justenough.com > > Confidentiality Note: The information contained in this email and > document(s) attached are for the exclusive use of the addressee and may > contain confidential, privileged and non-disclosable information. If the > recipient of this email is not the addressee, such recipient is strictly > prohibited from reading, photocopying, distribution or otherwise using this > email or its contents in any way. > > >
