GitHub user mccheah opened a pull request:
https://github.com/apache/spark/pull/20910
[SPARK-22839] [K8s] Refactor to unify driver and executor pod builder APIs
## What changes were proposed in this pull request?
Breaks down the construction of driver pods and executor pods in a way that
uses a common abstraction for both spark-submit creating the driver and
KubernetesClusterSchedulerBackend creating the executor. Encourages more code
reuse and is more legible than the older approach.
The high-level design is discussed in more detail on the JIRA ticket. This
pull request is the implementation of that design with some minor changes in
the implementation details.
No user-facing behavior should break as a result of this change.
## How was this patch tested?
Migrated all unit tests from the old submission steps architecture to the
new architecture. Integration tests should not have to change and pass given
that this shouldn't change any outward behavior.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mccheah/spark spark-22839-incremental
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20910.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20910
----
commit f1b8c08f90b3d3a689e0206d8f8c71c14bb91899
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:17:46Z
Fundamental building blocks for the new pod construction architecture.
commit 80e1562f4ec73f04a3009d16739f9d147625cfe8
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:22:00Z
Intermediate commit to move file.
commit c3460ae49a3206a4abd1e386caeacca249cb2e4e
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:26:31Z
Move basic driver configuration to new architecture.
commit 6d1711bbe8601340f5bfdc740177294cb5dd969c
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:28:46Z
Intermediate commit to move file
commit 4036d724268a8e9915f32838bacaff76c0754ada
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:31:20Z
Migrate mounting K8s credentials to the new architecture.
commit d46d6712b52247b43ecc9f8404c44c7eafba97ad
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:33:19Z
Intermediate commit to move file.
commit 2936aa5d7c98b15dcb37ecf8022aa7313062fb81
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:36:25Z
Migrate creating the driver service to the new architecture.
commit d2751b6696522bd775d6d9907574ebbe6aea6434
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:37:31Z
Remove dependency resolution step entirely.
It's now subsumed by the basicdriverfeaturestep.
commit 430fbb2a7d6559fd6c32a80a77e9d553df419223
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:43:01Z
Move mounting driver secrets to new architecture.
commit fd3e8e6eae52af46c9eafe569890ae225a11c596
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:48:18Z
Complete driver migration to new pod construction architecture.
commit f0ea6d9a87aa24c47dbcc3cf6e5d73a033a72601
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:49:33Z
Intermediate commit to move file
commit 67e9ca165282d05c3b3ada46827bfd0756e5d1c3
Author: mcheah <mcheah@...>
Date: 2018-03-26T22:53:15Z
Migrate executor pod construction to use the new architecture.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
