: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Okay, if there is a consensus in merging the SparkR feature and its respective
tests separately, I will split up the current PR to allow for the feature to be
merged while the e2e tests to be used for locally testing until we are ready to
merge
Okay, if there is a consensus in merging the SparkR feature and its
respective tests separately, I will split up the current PR to allow for
the feature to be merged while the e2e tests to be used for locally testing
until we are ready to merge (similar to PySpark). Will give it another day
to
On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson wrote:
> IMO sparkR support makes sense to merge for 2.4, as long as the release
> wranglers agree that local integration testing is sufficiently convincing.
>
i agree w/this. testing for this stuff specifically will happen within a
couple of
IMO sparkR support makes sense to merge for 2.4, as long as the release
wranglers agree that local integration testing is sufficiently convincing.
Part of the intent here is to allow this to happen without Shane having to
reorganize his complex upgrade schedule and make it even more complicated.
I'm also happy to see we have R support on k8s for Spark 2.4. I'll do the
manual testing for it if we don't want to upgrade the OS now. If the Python
support is also merged in this way, I think we can merge the R support PR
too?
On Thu, Aug 16, 2018 at 7:23 AM shane knapp wrote:
>
>> What is
>
>
> What is the current purpose of these builds?
>
> to be honest, i have absolutely no idea. :)
these were set up a long time ago, in a galaxy far far away, by someone who
is not me.
> - spark-docs seems to be building the docs, is that the only place
> where the docs build is tested?
>
> i
On Wed, Aug 15, 2018 at 1:35 PM, shane knapp wrote:
> in fact, i don't see us getting rid of all of the centos machines until EOY
> (see my above comment, re docs, release etc). these are the builds that
> will remain on centos for the near future:
>
On Wed, Aug 15, 2018 at 12:45 PM, Reynold Xin wrote:
> What's the reason we don't want to do the OS updates right now? Is it due
> to the unpredictability of potential issues that might happen and end up
> delaying 2.4 release?
>
> that is exactly it... i haven't had a chance to test everything
Correct, the OS change and updates would require more testing, from what
Shane has told me, and could potentially surface some issue that could
delay a major release.
So yes, the release manager would need to run the tests manually and after
the release we would switch to a fully integrated
Personally I'd love for R support to be in 2.4, but I don't consider
something "Done" unless tests are running ... Is the proposal: the release
manager manually run the R tests when preparing the release, and switch
over to fully integrated Jenkins after 2.4.0 is released?
On Wed, Aug 15, 2018 at
What's the reason we don't want to do the OS updates right now? Is it due
to the unpredictability of potential issues that might happen and end up
delaying 2.4 release?
On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson wrote:
> The SparkR support PR is finished, along with integration testing,
The SparkR support PR includes integration testing that can be tested on a
local Minikube instance by merely running the distribution with appropriate
flags (--r) and running the integration-tests similarly to as you would on
any k8s test. Maybe some others could locally test this, if there is any
The SparkR support PR is finished, along with integration testing, however
Shane has requested that the integration testing not be enabled until after
the 2.4 release because it requires the OS updates he wants to test *after*
the release.
The integration testing can be run locally, and so the
13 matches
Mail list logo
