[
https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752907#comment-15752907
]
Andrew Ash commented on SPARK-18278:
------------------------------------
There are definitely challenges in building features that take longer than a
release cycle (quarterly for Spark).
We could maintain a long-running feature branch for spark-k8s that lasts
several months and then gets merged into Spark in a big-bang merge, with that
feature branch living either on apache/spark or in some other
community-accessible repo. I don't think there are many practical differences
between in apache/spark vs a different repo for where the source is hosted if
both are not in Apache releases.
Or we could merge many smaller commits for spark-k8s into the apache/spark
master branch along the way and release as an experimental feature when release
time comes. This enables more continuous code review but has the risk of
destabilizing the master branch if code reviews miss things.
Looking to past instances of large features spanning multiple release cycles
(like SparkSQL and YARN integration), both of those had work happening
primarily in-repo from what I can tell, and releases included large disclaimers
in release notes for those experimental features. That precedent seems to
suggest Kubernetes integration should follow a similar path.
Personally I lean towards the approach of more smaller commits into master
rather than a long-running feature branch. By code reviewing PRs into the main
repo as we go the feature will be easier to code review and will also get wider
feedback as an experimental feature than a side branch or side repo would get.
This also serves to include Apache committers from the start in understanding
the codebase, rather than foisting a foreign codebase onto the project and hope
committers grok it well enough to hold the line on high quality code reviews.
Looking to the future where Kubernetes integration is potentially included in
the mainline apache release (like Mesos and YARN), it's best to work as
contributor + committer together from the start for shared understanding.
Making an API for third party cluster managers sound great and the easy, clean
choice from a software engineering point of view, but I wonder how much value
the practical benefits of having a pluggable cluster manager actually gets the
Apache project. It seems like both Two Sigma and IBM have been able to
maintain their proprietary schedulers without the benefits of the API we're
considering building. Who / what workflows are we aiming to support with an
API?
> Support native submission of spark jobs to a kubernetes cluster
> ---------------------------------------------------------------
>
> Key: SPARK-18278
> URL: https://issues.apache.org/jira/browse/SPARK-18278
> Project: Spark
> Issue Type: Umbrella
> Components: Build, Deploy, Documentation, Scheduler, Spark Core
> Reporter: Erik Erlandson
> Attachments: SPARK-18278 - Spark on Kubernetes Design Proposal.pdf
>
>
> A new Apache Spark sub-project that enables native support for submitting
> Spark applications to a kubernetes cluster. The submitted application runs
> in a driver executing on a kubernetes pod, and executors lifecycles are also
> managed as pods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
