Had the same issue. I can't remember what the issue was but this works:
libraryDependencies ++= {
val sparkVersion = "1.2.0"
Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming-twitter" % sparkVersion %
"provided",
"org.apache.spark" %% "spark-streaming-kafka" % sparkVersion %
"provided",
"javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided"
)
}
In order to run classes in "main" source in Intellij, you must invoke it
from a source under "test" as Intellij won't provide the "provided" scope
libraries when running code in "main" source (but it will for sources under
"test").
With this config you can "sbt assembly" in order to get the fat jar without
Spark jars.
ᐧ
On Tue, Jan 13, 2015 at 12:16 PM, Night Wolf <nightwolf...@gmail.com> wrote:
> Hi,
>
> I'm trying to load up an SBT project in IntelliJ 14 (windows) running 1.7
> JDK, SBT 0.13.5 -I seem to be getting errors with the project.
>
> The build.sbt file is super simple;
>
> name := "scala-spark-test1"
> version := "1.0"
>
> scalaVersion := "2.10.4"
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
>
>
> Then I have a super simple test class;
>
> package test
>
> import org.apache.spark.{SparkContext, SparkConf}
>
> case class Blah(s: Int, d: String)
>
> object Test1 {
> def main(args: Array[String]): Unit = {
> val sparkconf = new
> SparkConf().setMaster("local[4]").setAppName("test-spark")
> val sc = new SparkContext(sparkconf)
>
> val rdd = sc.parallelize(Seq(
> Blah(1,"dsdsd"),
> Blah(2,"daaa"),
> Blah(3,"dhghghgh")
> ))
>
> rdd.collect().foreach(println)
>
> }
> }
>
>
> When I try to run the Test1 object in IntelliJ I get the following error;
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> javax/servlet/http/HttpServletResponse
> at org.apache.spark.HttpServer.org
> $apache$spark$HttpServer$$doStart(HttpServer.scala:73)
> at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)
> at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)
> at
> org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1676)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1667)
> at org.apache.spark.HttpServer.start(HttpServer.scala:60)
> at org.apache.spark.HttpFileServer.initialize(HttpFileServer.scala:45)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:304)
> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:232)
> at test.Test1$.main(Test1.scala:10)
> at test.Test1.main(Test1.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: java.lang.ClassNotFoundException:
> javax.servlet.http.HttpServletResponse
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 18 more
>
>
> For whatever reason it seems that IntelliJ isnt pulling in these deps.
> Doing an sbt run works fine. Looking at the project structure it seems that
> 7 libs dont get marked as a dependency for my module... But they are on the
> dep tree.... http://pastebin.com/REkQh5ux
>
> [image: Inline image 1]
>
> Is this something to do with the libs and scoping or shading in Spark and
> its associated libs? Has anyone else seen this issue?
>
> Cheers,
> NW
>
