Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-26 Thread Sean Owen
This is all merged to master/2.4. AFAIK there aren't any items I'm monitoring that are needed for 2.4. On Thu, Oct 25, 2018 at 6:54 PM Sean Owen wrote: > Yep, we're going to merge a change to separate the k8s tests into a > separate profile, and fix up the Scala 2.12 thing. While non-critical

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-26 Thread Stavros Kontopoulos
Sean, Yes, I updated the PR and re-run it. On Fri, Oct 26, 2018 at 2:54 AM, Sean Owen wrote: > Yep, we're going to merge a change to separate the k8s tests into a > separate profile, and fix up the Scala 2.12 thing. While non-critical those > are pretty nice to have for 2.4. I think that's

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Reynold Xin
I also think we should get this in: https://github.com/apache/spark/pull/22841 It's to deprecate a confusing & broken window function API, so we can remove them in 3.0 and redesign a better one. See https://issues.apache.org/jira/browse/SPARK-25841 for more information. On Thu, Oct 25, 2018 at

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Sean Owen
Yep, we're going to merge a change to separate the k8s tests into a separate profile, and fix up the Scala 2.12 thing. While non-critical those are pretty nice to have for 2.4. I think that's doable within the next 12 hours even. @skonto I think there's one last minor thing needed on this PR?

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Wenchen Fan
Any updates on this topic? https://github.com/apache/spark/pull/22827 is merged and 2.4 is unblocked. I'll cut RC5 shortly after the weekend, and it will be great to include the change proposed here. Thanks, Wenchen On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos <

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Stavros Kontopoulos
> > I think it's worth getting in a change to just not enable this module, > which ought to be entirely safe, and avoid two of the issues we > identified. > Besides disabling it, when someone wants to run the tests with 2.12 he should be able to do so. So propagating the Scala profile still makes

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Sean Owen
I think it's worth getting in a change to just not enable this module, which ought to be entirely safe, and avoid two of the issues we identified. that said it didn't block RC4 so need not block RC5. But should happen today if we're doing it. On Thu, Oct 25, 2018 at 10:47 AM Xiao Li wrote: > >

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Xiao Li
Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved. Thanks, Xiao Sean Owen 于2018年10月25日周四 上午8:44写道: > Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :) > > On Thu, Oct 25, 2018 at 10:41 AM

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Sean Owen
Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :) On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos wrote: > > I agree these tests should be manual for now but should be run somehow before > a release to make sure things are working right? > > For the other issue:

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Stavros Kontopoulos
I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right? For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 . On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com>

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Stavros Kontopoulos
I will open a jira for the profile propagation issue and have a look to fix it. Stavros On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson wrote: > > I would be comfortable making the integration testing manual for now. A > JIRA for ironing out how to make it reliable for automatic as a goal for

Re: What if anything to fix about k8s for the 2.4.0 RC5?

2018-10-25 Thread Erik Erlandson
I would be comfortable making the integration testing manual for now. A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea. On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: > Forking this thread. > > Because we'll have another RC, we could