[
https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733439#comment-15733439
]
Erik Erlandson commented on SPARK-18278:
----------------------------------------
As I understand it (and as I've built them) an "MVP" Apache Spark docker image
consists of:
1. Some base OS image, presumably some variant of linux, with some package
management, but starting from some minimalist install
2. A spark-compatible JRE
3. (if python support) whatever standard python installs are required to run
py-spark
4. A Spark distro, likely installed from an official distro tarball
Hopefully I'm not over-simplifying, but IIUC the licensing around all of those
is well understood and known to be FOSS compatible. Other non-minimal, or
non-standard images builds are definitely possible, but I'd consider those to
be under the purview of 3rd parties in the community.
Publishing "official Apache Spark" images would imply some new
responsibilities, including maintenance. A possible roadmap might be to add
"official" images as part of a subsequent phase, drawing on experience with
phase 1. A separate registry organization could in principle be used, for
example: https://hub.docker.com/u/k8s4spark/
A consequence of not having such an official image is that integration testing
would then be based, at least initially, on 3rd-party images.
> Support native submission of spark jobs to a kubernetes cluster
> ---------------------------------------------------------------
>
> Key: SPARK-18278
> URL: https://issues.apache.org/jira/browse/SPARK-18278
> Project: Spark
> Issue Type: Umbrella
> Components: Build, Deploy, Documentation, Scheduler, Spark Core
> Reporter: Erik Erlandson
> Attachments: SPARK-18278 - Spark on Kubernetes Design Proposal.pdf
>
>
> A new Apache Spark sub-project that enables native support for submitting
> Spark applications to a kubernetes cluster. The submitted application runs
> in a driver executing on a kubernetes pod, and executors lifecycles are also
> managed as pods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
