Marcelo Vanzin created SPARK-8055:
-------------------------------------

             Summary: Spark Launcher Improvements
                 Key: SPARK-8055
                 URL: https://issues.apache.org/jira/browse/SPARK-8055
             Project: Spark
          Issue Type: Umbrella
          Components: Spark Core
    Affects Versions: 1.4.0
            Reporter: Marcelo Vanzin


Filing a bug to track different enhancements to the Spark launcher library. 
Please file sub-tasks for each particular enhancement instead of tagging PRs 
with this bug's number.

After some discussion in the mailing list, people have requested different 
enhancements to the library. I'll try to capture those here but feel free to 
add more in the comments.

- Missing information about the launched Spark application.

Currently the library returns an opaque "Process" object that doesn't have a 
lot of Spark-related functionality. It would be useful to get at least some 
information about the underlying process; in the very least the application ID 
of the actual Spark job. Other useful information could be, for example, the 
current status of the submitted job.

- Ability to control the underlying application.

The "Process" object only allows you to kill the underlying application. It 
would be better to have better application-level APIs to try to stop the 
application more cleanly (e.g. by asking the cluster manager to kill it, or by 
stopping the SparkContext in client mode).

- Ability to run Spark applications in the same JVM.

This could potentially be done today for cluster mode apps without getting bit 
by the limitations of SparkContext. In the long run, it would be nice to also 
support client mode apps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to