Hi firemonk9,
What you're doing looks interesting. Can you share some more details?
Are you running the same spark context for each job, or are you running a
seperate spark context for each job?
Does your system need sharing of rdd's across multiple jobs? If yes, how do
you implement that?
Also
I strongly recommend spawning a new process for the Spark jobs. Much
cleaner separation. Your driver program won't be clobbered if the Spark job
dies, etc. It can even watch for failures and restart.
In the Scala standard library, the sys.process package has classes for
constructing and
On 21 Apr 2015, at 17:34, Richard Marscher
rmarsc...@localytics.commailto:rmarsc...@localytics.com wrote:
- There are System.exit calls built into Spark as of now that could kill your
running JVM. We have shadowed some of the most offensive bits within our own
application to work around this.
.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Instantiating-starting-Spark-jobs-programmatically-tp22577p22584.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
/tips/best-practices in
this regard?
Cheers!
Ajay
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Instantiating-starting-Spark-jobs-programmatically-tp22577.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Instantiating-starting-Spark-jobs-programmatically-tp22577p22584.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e
