2) will be ideal but given the velocity of main branch, what Mesos ended up doing was simply having a separate repo since it will take too long to merge back to main.
We ended up running it pre-release (or major PR merged) and not on every PR, I will also comment on asking users to run it. We did have conversations with Reynold about potentially have the ability to run the CI on every [Mesos] tagged PR but we never got there. Tim On Mon, Jan 8, 2018 at 10:16 PM, Anirudh Ramanathan <fox...@google.com.invalid> wrote: > This is with regard to the Kubernetes Scheduler Backend and scaling the > process to accept contributions. Given we're moving past upstreaming changes > from our fork, and into getting new patches, I wanted to start this > discussion sooner than later. This is more of a post-2.3 question - not > something we're looking to solve right away. > > While unit tests are handy, they're not nearly as good at giving us > confidence as a successful run of our integration tests against > single/multi-node k8s clusters. Currently, we have integration testing setup > at https://github.com/apache-spark-on-k8s/spark-integration and it's running > continuously against apache/spark:master in pepperdata-jenkins (on minikube) > & k8s-testgrid (in GKE clusters). Now, the question is - how do we make > integration-tests part of the PR author's workflow? > > 1. Keep the integration tests in the separate repo and require that > contributors run them, add new tests prior to accepting their PRs as a > policy. Given minikube is easy to setup and can run on a single-node, it > would certainly be possible. Friction however, stems from contributors > potentially having to modify the integration test code hosted in that > separate repository when adding/changing functionality in the scheduler > backend. Also, it's certainly going to lead to at least brief > inconsistencies between the two repositories. > > 2. Alternatively, we check in the integration tests alongside the actual > scheduler backend code. This would work really well and is what we did in > our fork. It would have to be a separate package which would take certain > parameters (like cluster endpoint) and run integration test code against a > local or remote cluster. It would include least some code dealing with > accessing the cluster, reading results from K8s containers, test fixtures, > etc. > > I see value in adopting (2), given it's a clearer path for contributors and > lets us keep the two pieces consistent, but it seems uncommon elsewhere. How > do the other backends, i.e. YARN, Mesos and Standalone deal with accepting > patches and ensuring that they do not break existing clusters? Is there > automation employed for this thus far? Would love to get opinions on (1) v/s > (2). > > Thanks, > Anirudh > > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
