[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929591#comment-15929591 ]
Andrew Ash commented on SPARK-18278: ------------------------------------ As an update on this ticket: For those not already aware, work on native Spark integration with Kubernetes has been proceeding for the past several months in this repo https://github.com/apache-spark-on-k8s/spark in the {{branch-2.1-kubernetes}} branch, based off the 2.1.0 Apache release. We have an active core of about a half dozen contributors to the project with a wider group observing of about another dozen. Communication happens through the issues on the GitHub repo, a dedicated room in the Kubernetes Slack, and weekly video conferences hosted by the Kubernetes Big Data SIG. The full patch set is currently about 5500 lines, with about 500 of that as user/dev documentation. Infrastructure-wise, we have a cloud-hosted CI Jenkins instance set up donated by project members, which is running both unit tests and Kubernetes integration tests over the code. We recently entered a code freeze for our release branch and are preparing a first release to the wider community, which we plan to announce on the general Spark users list. It includes the completed "phase one" portion of the design doc shared a few months ago (https://docs.google.com/document/d/1_bBzOZ8rKiOSjQg78DXOA3ZBIo_KkDJjqxVuq0yXdew/edit#heading=h.fua3ml5mcolt), featuring cluster mode with static allocation of executors, submission of local resources, SSL throughout, and support for JVM languages (Java/Scala). After that release we'll be continuing to stabilize and improve the phase one feature set and move into a second phase of kubernetes work. It will likely be focused on support for dynamic allocation, though we haven't finalized planning for phase two yet. Working on the pluggable scheduler in SPARK-19700 may be included as well. Interested parties are of course welcome to watch the repo, join the weekly video conferences, give the code a shot, and contribute to the project! > Support native submission of spark jobs to a kubernetes cluster > --------------------------------------------------------------- > > Key: SPARK-18278 > URL: https://issues.apache.org/jira/browse/SPARK-18278 > Project: Spark > Issue Type: Umbrella > Components: Build, Deploy, Documentation, Scheduler, Spark Core > Reporter: Erik Erlandson > Attachments: SPARK-18278 - Spark on Kubernetes Design Proposal.pdf > > > A new Apache Spark sub-project that enables native support for submitting > Spark applications to a kubernetes cluster. The submitted application runs > in a driver executing on a kubernetes pod, and executors lifecycles are also > managed as pods. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org