Github user mccheah commented on the issue: https://github.com/apache/spark/pull/20910 For all reviewers - this change is very large. Github's interpretation of the diff also doesn't present the changes in the most easily consumed manner. To account for this, the pull request is best reviewed and understood commit by commit. Each commit roughly translates one component from the old architecture to the new architecture. The changes are incrementally built as follows: 1. We begin by presenting the [base building blocks](https://github.com/apache/spark/pull/20910/commits/f1b8c08f90b3d3a689e0206d8f8c71c14bb91899) in [this commit](). 2. Migrate [basic driver configurations](https://github.com/apache/spark/pull/20910/commits/c3460ae49a3206a4abd1e386caeacca249cb2e4e). 3. Migrate [mounting Kubernetes credentials in the driver](https://github.com/apache/spark/pull/20910/commits/4036d724268a8e9915f32838bacaff76c0754ada). 4. Migrate [creation of the driver service](https://github.com/apache/spark/pull/20910/commits/2936aa5d7c98b15dcb37ecf8022aa7313062fb81). 5. [Remove dependency resolution](https://github.com/apache/spark/pull/20910/commits/d2751b6696522bd775d6d9907574ebbe6aea6434) as it is subsumed by `BasicDriverFeatureStep`. 6. Migrate [mounting user-provided driver secrets](https://github.com/apache/spark/pull/20910/commits/430fbb2a7d6559fd6c32a80a77e9d553df419223). 7. Wire steps 2-6 above to [build the entire driver spec](https://github.com/apache/spark/pull/20910/commits/fd3e8e6eae52af46c9eafe569890ae225a11c596), composing the incremental steps accordingly. Removes all of the remaining old driver-construction code that wasn't removed in steps 2-6. 8. Move all of the executor construction to [use the new architecture](https://github.com/apache/spark/pull/20910/commits/67e9ca165282d05c3b3ada46827bfd0756e5d1c3) as well. We can alternatively create multiple pull requests to merge this change incrementally, but each intermediate pull request would likely be broken in the K8s functionality. To ensure that master is never in a broken unusable state for K8s, we unfortunately need to merge the entire change at once.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org