NoClassDefFoundError

[Reza zade]

Hello

I've extended the JavaSparkJob (job-server-0.6.2) and created an object
of SQLContext class. my maven project doesn't have any problem during
compile and packaging phase. but when I send .jar of project to sjs and run
it "NoClassDefFoundError" will be issued. the trace of exception is :


job-server[ERROR] Exception in thread "pool-20-thread-1"
java.lang.NoClassDefFoundError:
org/apache/spark/sql/SQLContext
job-server[ERROR]  at sparkdesk.SparkSQLJob2.runJob(SparkSQLJob2.java:61)
job-server[ERROR]  at sparkdesk.SparkSQLJob2.runJob(SparkSQLJob2.java:45)
job-server[ERROR]  at spark.jobserver.JavaSparkJob.
runJob(JavaSparkJob.scala:17)
job-server[ERROR]  at spark.jobserver.JavaSparkJob.
runJob(JavaSparkJob.scala:14)
job-server[ERROR]  at spark.jobserver.JobManagerActor$$anonfun$
spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:
301)
job-server[ERROR]  at scala.concurrent.impl.Future$
PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
job-server[ERROR]  at scala.concurrent.impl.Future$
PromiseCompletingRunnable.run(Future.scala:24)
job-server[ERROR]  at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
job-server[ERROR]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
job-server[ERROR]  at java.lang.Thread.run(Thread.java:745)
job-server[ERROR] Caused by: java.lang.ClassNotFoundException:
org.apache.spark.sql.SQLContext
job-server[ERROR]  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
job-server[ERROR]  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
job-server[ERROR]  at java.security.AccessController.doPrivileged(Native
Method)
job-server[ERROR]  at java.net.URLClassLoader.findClass(URLClassLoader.java:
354)
job-server[ERROR]  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
job-server[ERROR]  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
job-server[ERROR]  ... 10 more


what is the problem?
do you have any solution about this?

problem deploying spark-jobserver on CentOS

[Reza zade]

Hi

I'm going to deploy jobserver on my CentOS (spark is installed with cdh5.7).

I'm using oracle jdk1.8, sbt-0.13.13, spark-1.6.0 and jobserver-0.6.2.

When I run sbt command (after running sbt publish-local) I encountered the
bellow message :

[cloudera@quickstart spark-jobserver]$ sbt
[info] Loading project definition from /home/cloudera/spark-
jobserver/project
Missing bintray credentials /home/cloudera/.bintray/.credentials. Some
bintray features depend on this.
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
Missing bintray credentials /home/cloudera/.bintray/.credentials. Some
bintray features depend on this.
Missing bintray credentials /home/cloudera/.bintray/.credentials. Some
bintray features depend on this.
Missing bintray credentials /home/cloudera/.bintray/.credentials. Some
bintray features depend on this.
[info] Set current project to root (in build file:/home/cloudera/spark-
jobserver/)
> re-start
[info] scalastyle using config /home/cloudera/spark-
jobserver/scalastyle-config.xml
[info] Processed 35 file(s)
[info] Found 0 errors
[info] Found 0 warnings
[info] Found 0 infos
[info] Finished in 7 ms
[success] created output: /home/cloudera/spark-jobserver/job-server/target
[info] scalastyle using config /home/cloudera/spark-
jobserver/scalastyle-config.xml
[info] Processed 5 file(s)
[info] Found 0 errors
[info] Found 0 warnings
[info] Found 0 infos
[info] Finished in 1 ms
[success] created output: /home/cloudera/spark-jobserver/job-server-api/
target
[info] scalastyle using config /home/cloudera/spark-
jobserver/scalastyle-config.xml
[info] Processed 11 file(s)
[info] Found 0 errors
[info] Found 0 warnings
[info] Found 0 infos
[info] Finished in 1 ms
[success] created output: /home/cloudera/spark-jobserver/akka-app/target
[info] scalastyle using config /home/cloudera/spark-
jobserver/scalastyle-config.xml
[info] Processed 5 file(s)
[info] Found 0 errors
[info] Found 0 warnings
[info] Found 0 infos
[info] Finished in 1 ms
[success] created output: /home/cloudera/spark-jobserver/job-server-tests/
target
[info] scalastyle using config /home/cloudera/spark-
jobserver/scalastyle-config.xml
[info] Processed 0 file(s)
[info] Found 0 errors
[info] Found 0 warnings
[info] Found 0 infos
[info] Finished in 0 ms
[success] created output: /home/cloudera/spark-jobserver/target
[info] scalastyle using config /home/cloudera/spark-
jobserver/scalastyle-config.xml
[info] Processed 13 file(s)
[info] Found 0 errors
[info] Found 0 warnings
[info] Found 0 infos
[info] Finished in 1 ms
[success] created output: /home/cloudera/spark-jobserver/job-server-extras/
target
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[info] Updating {file:/home/cloudera/spark-jobserver/}job-server-api...
[info] Updating {file:/home/cloudera/spark-jobserver/}job-server-api...
[info] Updating {file:/home/cloudera/spark-jobserver/}job-server-api...
[info] Updating {file:/home/cloudera/spark-jobserver/}akka-app...
[info] Updating {file:/home/cloudera/spark-jobserver/}akka-app...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[info] Updating {file:/home/cloudera/spark-jobserver/}job-server-tests...
[info] Updating {file:/home/cloudera/spark-jobserver/}job-server...
[info] Updating {file:/home/cloudera/spark-jobserver/}job-server...
[info] Resolving org.scala-lang#scala-library;2.10.6 ...
[info] Updating {file:/home/cloudera/spark-jobserver/}job-server...
[info] Resolving org.uncommons.maths#uncommons-maths;1.2.2a ...
[warn] Credentials file /home/cloudera/.bintray/.credentials does not exist
[warn] Credentials file 

installing spark-jobserver on cdh 5.7 and yarn

[Reza zade]

Hi

I have set up a cloudera cluster and work with spark. I want to install
spark-jobserver on it. What should I do?

Fwd: A Spark long running program as web server

[Reza zade]

Hi

I have written multiple spark driver programs that load some data from HDFS
to data frames and accomplish spark sql queries on it and persist the
results in HDFS again. Now I need to provide a long running java program in
order to receive requests and their some parameters(such as the number of
top rows should be returned) from a web application (e.g. a dashboard) via
post and get and send back the results to web application. My web
application is somewhere out of the Spark cluster. Briefly my goal is to
send requests and their accompanying data from web application via
something such as POST to long running java program. then it receives the
request and runs the corresponding spark driver (spark app) and returns the
results for example in JSON format.


Whats is your suggestion to develop this use case?
Is Livy a good choise? If your answer is positive what should I do?

Thanks.