[
https://issues.apache.org/jira/browse/SPARK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493303#comment-14493303
]
Evan Chan commented on SPARK-6703:
----------------------------------
Hey folks,
Thought I would just put in my 2 cents as the author of the Spark Jobserver.
What is the envisioned way for multiple "applications" to share the same
SparkContext? Code has to be running in the same JVM, and for most
applications there already must exist some shared knowledge of the framework or
environment. This will affect whether this feature is useful or not.
For example, the Spark Jobserver requires jobs to implement an interface, and
also manages creation of the SparkContext. That way, jobs get the SparkContext
through a method call, and we can have other method calls to do things like
input validation.
What I'm saying is that this feature would have little existing value to job
server users, as jobs in job server already have a way to discover the existing
context, and to implement a good RESTful API, for example.
Another thing to think about is what about SQLContext, HiveContext. I realize
there is the JDBC server, but in job server we have a way to pass in
alternative forms of the contexts. I suppose you could then add this method to
a static SQLContext singleton as well.
> Provide a way to discover existing SparkContext's
> -------------------------------------------------
>
> Key: SPARK-6703
> URL: https://issues.apache.org/jira/browse/SPARK-6703
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Patrick Wendell
> Assignee: Ilya Ganelin
> Priority: Critical
>
> Right now it is difficult to write a Spark application in a way that can be
> run independently and also be composed with other Spark applications in an
> environment such as the JobServer, notebook servers, etc where there is a
> shared SparkContext.
> It would be nice to provide a rendez-vous point so that applications can
> learn whether an existing SparkContext already exists before creating one.
> The most simple/surgical way I see to do this is to have an optional static
> SparkContext singleton that people can be retrieved as follows:
> {code}
> val sc = SparkContext.getOrCreate(conf = new SparkConf())
> {code}
> And you could also have a setter where some outer framework/server can set it
> for use by multiple downstream applications.
> A more advanced version of this would have some named registry or something,
> but since we only support a single SparkContext in one JVM at this point
> anyways, this seems sufficient and much simpler. Another advanced option
> would be to allow plugging in some other notion of configuration you'd pass
> when retrieving an existing context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
