Re: Dependency Problem with Spark / ScalaTest / SBT
Hello, When I remove the line and try to execute sbt run, I end up with the following lines: 14/09/15 10:11:35 INFO ui.SparkUI: Stopped Spark web UI at http://base:4040 [...] 14/09/15 10:11:05 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 14/09/15 10:11:15 INFO client.AppClient$ClientActor: Connecting to master spark://base:7077... It seems that the configuration within sbt doesn't use my original Spark, because my original Spark web UI is running under http://base:8080. Seems like sbt is starting another spark instance?? Best regards Thorsten Am 14.09.2014 um 18:56 schrieb Dean Wampler: Sorry, I meant any *other* SBT files. However, what happens if you remove the line: exclude(org.eclipse.jetty.orbit, javax.servlet) dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition http://shop.oreilly.com/product/0636920033073.do (O'Reilly) Typesafe http://typesafe.com @deanwampler http://twitter.com/deanwampler http://polyglotprogramming.com On Sun, Sep 14, 2014 at 11:53 AM, Dean Wampler deanwamp...@gmail.com mailto:deanwamp...@gmail.com wrote: Can you post your whole SBT build file(s)? Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition http://shop.oreilly.com/product/0636920033073.do (O'Reilly) Typesafe http://typesafe.com @deanwampler http://twitter.com/deanwampler http://polyglotprogramming.com On Wed, Sep 10, 2014 at 6:48 AM, Thorsten Bergler sp...@tbonline.de mailto:sp...@tbonline.de wrote: Hi, I just called: test or run Thorsten Am 10.09.2014 um 13:38 schrieb arthur.hk.c...@gmail.com mailto:arthur.hk.c...@gmail.com: Hi, What is your SBT command and the parameters? Arthur On 10 Sep, 2014, at 6:46 pm, Thorsten Bergler sp...@tbonline.de mailto:sp...@tbonline.de wrote: Hello, I am writing a Spark App which is already working so far. Now I started to build also some UnitTests, but I am running into some dependecy problems and I cannot find a solution right now. Perhaps someone could help me. I build my Spark Project with SBT and it seems to be configured well, because compiling, assembling and running the built jar with spark-submit are working well. Now I started with the UnitTests, which I located under /src/test/scala. When I call test in sbt, I get the following: 14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered BlockManager 14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server [trace] Stack trace suppressed: run last test:test for the full output. [error] Could not run test test.scala.SetSuite: java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse [info] Run completed in 626 milliseconds. [info] Total number of tests run: 0 [info] Suites: completed 0, aborted 0 [info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [error] Error during tests: [error] test.scala.SetSuite [error] (test:test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 3 s, completed 10.09.2014 12:22:06 last test:test gives me the following: last test:test [debug] Running TaskDef(test.scala.SetSuite, org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector]) java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse at org.apache.spark.HttpServer.start(HttpServer.scala:54) at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156) at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127) at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31) at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48) at org.apache.spark.broadcast.BroadcastManager.init(BroadcastManager.scala:35) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218) at org.apache.spark.SparkContext.init
Dependency Problem with Spark / ScalaTest / SBT
Hello,
I am writing a Spark App which is already working so far.
Now I started to build also some UnitTests, but I am running into some
dependecy problems and I cannot find a solution right now. Perhaps
someone could help me.
I build my Spark Project with SBT and it seems to be configured well,
because compiling, assembling and running the built jar with
spark-submit are working well.
Now I started with the UnitTests, which I located under /src/test/scala.
When I call test in sbt, I get the following:
14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered BlockManager
14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server
[trace] Stack trace suppressed: run last test:test for the full output.
[error] Could not run test test.scala.SetSuite:
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
[info] Run completed in 626 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Error during tests:
[error] test.scala.SetSuite
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 3 s, completed 10.09.2014 12:22:06
last test:test gives me the following:
last test:test
[debug] Running TaskDef(test.scala.SetSuite,
org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector])
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
at org.apache.spark.HttpServer.start(HttpServer.scala:54)
at
org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
at
org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
at
org.apache.spark.broadcast.BroadcastManager.init(BroadcastManager.scala:35)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkContext.init(SparkContext.scala:202)
at test.scala.SetSuite.init(SparkTest.scala:16)
I also noticed right now, that sbt run is also not working:
14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP Server
[error] (run-main-2) java.lang.NoClassDefFoundError:
javax/servlet/http/HttpServletResponse
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
at org.apache.spark.HttpServer.start(HttpServer.scala:54)
at
org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
at
org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
at
org.apache.spark.broadcast.BroadcastManager.init(BroadcastManager.scala:35)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkContext.init(SparkContext.scala:202)
at
main.scala.PartialDuplicateScanner$.main(PartialDuplicateScanner.scala:29)
at
main.scala.PartialDuplicateScanner.main(PartialDuplicateScanner.scala)
Here is my Testprojekt.sbt file:
name := Testprojekt
version := 1.0
scalaVersion := 2.10.4
libraryDependencies ++= {
Seq(
org.apache.lucene % lucene-core % 4.9.0,
org.apache.lucene % lucene-analyzers-common % 4.9.0,
org.apache.lucene % lucene-queryparser % 4.9.0,
(org.apache.spark %% spark-core % 1.0.2).
exclude(org.mortbay.jetty, servlet-api).
exclude(commons-beanutils, commons-beanutils-core).
exclude(commons-collections, commons-collections).
exclude(commons-collections, commons-collections).
exclude(com.esotericsoftware.minlog, minlog).
exclude(org.eclipse.jetty.orbit, javax.mail.glassfish).
exclude(org.eclipse.jetty.orbit, javax.transaction).
exclude(org.eclipse.jetty.orbit, javax.servlet)
)
}
resolvers += Akka Repository at http://repo.akka.io/releases/;
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Dependency Problem with Spark / ScalaTest / SBT
Hi,
I just called:
test
or
run
Thorsten
Am 10.09.2014 um 13:38 schrieb arthur.hk.c...@gmail.com:
Hi,
What is your SBT command and the parameters?
Arthur
On 10 Sep, 2014, at 6:46 pm, Thorsten Bergler sp...@tbonline.de wrote:
Hello,
I am writing a Spark App which is already working so far.
Now I started to build also some UnitTests, but I am running into some
dependecy problems and I cannot find a solution right now. Perhaps someone
could help me.
I build my Spark Project with SBT and it seems to be configured well, because
compiling, assembling and running the built jar with spark-submit are working
well.
Now I started with the UnitTests, which I located under /src/test/scala.
When I call test in sbt, I get the following:
14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered BlockManager
14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server
[trace] Stack trace suppressed: run last test:test for the full output.
[error] Could not run test test.scala.SetSuite: java.lang.NoClassDefFoundError:
javax/servlet/http/HttpServletResponse
[info] Run completed in 626 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Error during tests:
[error] test.scala.SetSuite
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 3 s, completed 10.09.2014 12:22:06
last test:test gives me the following:
last test:test
[debug] Running TaskDef(test.scala.SetSuite,
org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector])
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
at org.apache.spark.HttpServer.start(HttpServer.scala:54)
at
org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
at
org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
at
org.apache.spark.broadcast.BroadcastManager.init(BroadcastManager.scala:35)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkContext.init(SparkContext.scala:202)
at test.scala.SetSuite.init(SparkTest.scala:16)
I also noticed right now, that sbt run is also not working:
14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP Server
[error] (run-main-2) java.lang.NoClassDefFoundError:
javax/servlet/http/HttpServletResponse
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
at org.apache.spark.HttpServer.start(HttpServer.scala:54)
at
org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
at
org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
at
org.apache.spark.broadcast.BroadcastManager.init(BroadcastManager.scala:35)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkContext.init(SparkContext.scala:202)
at
main.scala.PartialDuplicateScanner$.main(PartialDuplicateScanner.scala:29)
at main.scala.PartialDuplicateScanner.main(PartialDuplicateScanner.scala)
Here is my Testprojekt.sbt file:
name := Testprojekt
version := 1.0
scalaVersion := 2.10.4
libraryDependencies ++= {
Seq(
org.apache.lucene % lucene-core % 4.9.0,
org.apache.lucene % lucene-analyzers-common % 4.9.0,
org.apache.lucene % lucene-queryparser % 4.9.0,
(org.apache.spark %% spark-core % 1.0.2).
exclude(org.mortbay.jetty, servlet-api).
exclude(commons-beanutils, commons-beanutils-core).
exclude(commons-collections, commons-collections).
exclude(commons-collections, commons-collections).
exclude(com.esotericsoftware.minlog, minlog).
exclude(org.eclipse.jetty.orbit, javax.mail.glassfish).
exclude(org.eclipse.jetty.orbit, javax.transaction).
exclude(org.eclipse.jetty.orbit, javax.servlet)
)
}
resolvers += Akka Repository at http://repo.akka.io/releases/;
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
