Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

2018-08-16 Thread Felix Cheung
I fully support merging that SparkR support on k8s. If Ilan and other are willing to manually validate the RC I’m happy to voucher for it (I’m not 100% sure i have capacity to test it that way but certainly will try) Also +1 on revisiting Jenkins builds. tbh I’m not sure we depend on them too

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

2018-08-16 Thread Ilan Filonenko
Okay, if there is a consensus in merging the SparkR feature and its respective tests separately, I will split up the current PR to allow for the feature to be merged while the e2e tests to be used for locally testing until we are ready to merge (similar to PySpark). Will give it another day to

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

2018-08-16 Thread shane knapp
On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson wrote: > IMO sparkR support makes sense to merge for 2.4, as long as the release > wranglers agree that local integration testing is sufficiently convincing. > i agree w/this. testing for this stuff specifically will happen within a couple of

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

2018-08-16 Thread Erik Erlandson
IMO sparkR support makes sense to merge for 2.4, as long as the release wranglers agree that local integration testing is sufficiently convincing. Part of the intent here is to allow this to happen without Shane having to reorganize his complex upgrade schedule and make it even more complicated.