Re: Quick sync: what goes in migration guide vs release notes?

2020-06-10 Thread Sean Owen
If you are proposing to keep all important changes in release notes as usual: Sure, add more to the migration guide, too. It can't hurt, going forward. But many release-note changes have not much more to say about migration. If you're proposing to not mention most things in the release notes,

Re: Quick sync: what goes in migration guide vs release notes?

2020-06-10 Thread Wenchen Fan
Yea we can't update the 3.0.0 migration guide now, but AFAIK we do mention most of the breaking changes there, except for the ML module. I think we can still put all the ML breaking changes in the release notes this time. But in the future, shall we put breaking changes in the migration guide? It

Re: [ANNOUNCE] Apache Spark 2.4.6 released

2020-06-10 Thread Takeshi Yamamuro
Congrats and thanks, Holden! Bests, Takeshi On Thu, Jun 11, 2020 at 11:16 AM Dongjoon Hyun wrote: > Thank you so much, Holden! :) > > On Wed, Jun 10, 2020 at 6:59 PM Hyukjin Kwon wrote: > >> Yay! >> >> 2020년 6월 11일 (목) 오전 10:38, Holden Karau 님이 작성: >> >>> We are happy to announce the

Re: [ANNOUNCE] Apache Spark 2.4.6 released

2020-06-10 Thread Dongjoon Hyun
Thank you so much, Holden! :) On Wed, Jun 10, 2020 at 6:59 PM Hyukjin Kwon wrote: > Yay! > > 2020년 6월 11일 (목) 오전 10:38, Holden Karau 님이 작성: > >> We are happy to announce the availability of Spark 2.4.6! >> >> Spark 2.4.6 is a maintenance release containing stability, correctness, >> and

Re: [ANNOUNCE] Apache Spark 2.4.6 released

2020-06-10 Thread Hyukjin Kwon
Yay! 2020년 6월 11일 (목) 오전 10:38, Holden Karau 님이 작성: > We are happy to announce the availability of Spark 2.4.6! > > Spark 2.4.6 is a maintenance release containing stability, correctness, > and security fixes. > This release is based on the branch-2.4 maintenance branch of Spark. We > strongly

Re: Quick sync: what goes in migration guide vs release notes?

2020-06-10 Thread Hyukjin Kwon
I think the proposal doesn't mean to don't add the JIRAs with release-notes into the release notes (?). People will still label the JIRAs when the change is significant or breaking whether it's a bug or not, and they will be in the release notes. I guess the proposal TL;DR is: - If that's a

[ANNOUNCE] Apache Spark 2.4.6 released

2020-06-10 Thread Holden Karau
We are happy to announce the availability of Spark 2.4.6! Spark 2.4.6 is a maintenance release containing stability, correctness, and security fixes. This release is based on the branch-2.4 maintenance branch of Spark. We strongly recommend all 2.4 users to upgrade to this stable release. To

Re: Quick sync: what goes in migration guide vs release notes?

2020-06-10 Thread Sean Owen
This seems like a change to current practice, as breaking changes are marked for release notes with release-notes, etc: https://spark.apache.org/contributing.html My only concrete concern, is this seems to imply (?) that many JIRAs with release-notes and Docs text are not going to get included in

Re: Quick sync: what goes in migration guide vs release notes?

2020-06-10 Thread Wenchen Fan
My 2 cents: Since we have a migration guide, I think people who hit problems when upgrading Spark will read it. We should mention all the breaking changes there, except for trivial ones like obvious bug fixes. Even if there is no meaningful migration to guide for things like removing a deprecated

Why spark streaming does not support groupingsets

2020-06-10 Thread ashok.blend
We have scenario where we wanted to generate groupy on specific set of dimensions. Grouping sets solves our purpose, but its not supported in spark streaming. We cannot go for cube as it generates lot of combination which will be useless for us. So wanted to know if there is any specific reason