The spark-shell is a special version of the Scala REPL that serves the
classes created for each line over HTTP. Do you know if the InteliJ Spark
console is just the normal Scala repl in a GUI wrapper, or if it is
something else entirely? If its the former, perhaps it might be possible
to tell InteliJ to bring up the spark version instead.
On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
jonatha...@supersonicads.com> wrote:
> Hi all,
>
> TLDR: running spark locally through IntelliJ IDEA Scala Console results
> in java.lang.ClassNotFoundException
>
> Long version:
>
> I'm an algorithms developer in SupersonicAds - an ad network. We are
> building a major new big data project and we are now in the process of
> selecting our tech stack & tools.
>
> I'm new to Spark, but I'm very excited about it. It is my opinion that
> Spark can be a great tool for us, and that we might be able to build most
> of our toolchain on top of it.
>
> We currently develop in Scala and we are using IntelliJ IDEA as our IDE
> (we love it). One of the features I love about IDEA is the Scala Console
> which lets me work interactively with all of my project's code available
> and all of the IDE's features & convenience. That is as opposed to the
> Scala Shell & Spark Shell which I dislike because it is based on JLine and
> doesn't behave like a good shell would (I cant even Ctrl-c to abort a line
> without crashing the whole thing). Of course, as an algo guy, having a good
> REPL is crucial to me.
>
> To get started, I added the following line to build.sbt:
>
>> "org.apache.spark" %% "spark-core" % "0.9.1"
>
>
> Then, added the following main class:
>
> import org.apache.spark.SparkContext
>> import org.apache.spark.SparkContext._
>>
>
>
> object Main extends App {
>> val sc = new SparkContext("local", "myApp")
>> val r = sc.parallelize(1 to 1000)
>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>> 1).first())
>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>> 1).count())
>> }
>
>
> Make, Run, Works perfectly.
>
> Next, I try running the same in the scala console.
> Bad news - the last line throws an exception:
>
>> ERROR executor.Executor: Exception in task ID 0
>> java.lang.ClassNotFoundException: $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>
>
> It is my guess that for some reason Spark is not able to find the
> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
> not provide any jars. For some reason when using first() instead of count()
> it works. Needless to say it also works in Spark Shell but as I stated,
> working with it is not an option.
>
> This issue brings much sadness to my heart, and I could not find a
> solution on the mailing list archives or elsewhere. I am hoping someone
> here might offer some help.
>
> Thanks,
> Jon
>
