Re: [VOTE] Release Spark 3.1.0 (RC1)

I plan to cut RC 1 for Spark 3.1.1 this friday or next monday. Let me know if there are any blocker. Thanks guys. 2021년 1월 12일 (화) 오전 1:22, Dongjoon Hyun 님이 작성: > Thank you, Hyukjin! > > Bests, > Dongjoon. > > On Mon, Jan 11, 2021 at 7:24 AM Hyukjin Kwon wrote: > >> I had a response from the

Re: [VOTE] Release Spark 3.1.0 (RC1)

Thank you, Hyukjin! Bests, Dongjoon. On Mon, Jan 11, 2021 at 7:24 AM Hyukjin Kwon wrote: > I had a response from the INFRA team and Sonatype. Just to share, the > removal is possible as an exception, but it's best to go ahead for 3.1.1 > for safety as we all discussed. > There are several

Re: [VOTE] Release Spark 3.1.0 (RC1)

I had a response from the INFRA team and Sonatype. Just to share, the removal is possible as an exception, but it's best to go ahead for 3.1.1 for safety as we all discussed. There are several non-regression but correctness issues being tracked under

Re: [VOTE] Release Spark 3.1.0 (RC1)

Hi Sean, +1 to leave it. Makes so much more sense (as that's what really happened and the history of Apache Spark is...irreversible). Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on

Re: [VOTE] Release Spark 3.1.0 (RC1)

> I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away. +1 Sean Owen 于2021年1月7日周四 上午6:44写道： > While we can delete the tag, maybe just leave it. As a general

Re: [VOTE] Release Spark 3.1.0 (RC1)

While we can delete the tag, maybe just leave it. As a general rule we would not remove anything pushed to the main git repo. On Thu, Jan 7, 2021 at 8:31 AM Jacek Laskowski wrote: > Hi, > > BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with > 3.1.1, what's gonna happen

Re: [VOTE] Release Spark 3.1.0 (RC1)

Hi, BTW, wondering aloud. Since it was agreed to skip 3.1.0 and go ahead with 3.1.1, what's gonna happen with v3.1.0 tag [1]? Is it going away and we'll see 3.1.1-rc1? [1] https://github.com/apache/spark/tree/v3.1.0-rc1 Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The

Re: [VOTE] Release Spark 3.1.0 (RC1)

Hi, I'm just reading this now. I'm for 3.1.1 with no 3.1.0 but the news that we're skipping that particular release. Gonna be more fun! :) Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on

Re: [VOTE] Release Spark 3.1.0 (RC1)

> - prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST) > - and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts +1 On Thu, Jan 7, 2021 at 7:54 PM Takeshi Yamamuro wrote: > > I will prepare to

Re: [VOTE] Release Spark 3.1.0 (RC1)

> I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away. +1 Thanks for working on this, Hyukjin! Bests, Takeshi On Thu, Jan 7, 2021 at 2:13 PM Hyukjin Kwon

Re: [VOTE] Release Spark 3.1.0 (RC1)

Thank you Holden and Wenchen! Let me: - prepare a PR for news in spark-website first about 3.1.0 accident late tonight (in KST) - and start to prepare 3.1.1 probably in few more days like next monday in case other people have different thoughts 2021년 1월 7일 (목) 오후 2:04, Holden Karau 님이 작성: > I

Re: [VOTE] Release Spark 3.1.0 (RC1)

I think that posting the 3.1.0 maven release was an accident and we're going to 3.1.1 RCs is the right step forward. I'd ask for maybe a day before cutting the 3.1.1 release, I think https://issues.apache.org/jira/browse/SPARK-34018 is also a blocker (at first I thought it was just a test issue,

Re: [VOTE] Release Spark 3.1.0 (RC1)

I agree with Jungtaek that people are likely to be biased when testing 3.1.0. At least this will not be the same community-blessed release as previous ones, because the voting is already affected by the fact that 3.1.0 is already in maven central. Skipping 3.1.0 sounds better to me. On Thu, Jan

Re: [VOTE] Release Spark 3.1.0 (RC1)

Okay, let me just start to prepare 3.1.1. I think that will address all concerns except that 3.1.0 will remain in Maven as incomplete. By right, removal in the Maven repo is disallowed. Overwrite is possible as far as I know but other mirrors that maintain cache will get affected. Maven is one of

Re: [VOTE] Release Spark 3.1.0 (RC1)

I think that It would be great though if we have a clear blocker that makes the release pointless if we want to drop this RC practically given that we will schedule 3.1.1 faster - non-regression bug fixes will be delivered to end users relatively fast. That would make it clear which option we

Re: [VOTE] Release Spark 3.1.0 (RC1)

I don't agree the first two are blockers for reasons I gave earlier. Those two do look like important issues - are they regressions from 3.0.1? I do agree we'd probably cut a new RC for those in any event, so agree with the plan to drop 3.1.0 (if the Maven release can't be overwritten) On Wed,

Re: [VOTE] Release Spark 3.1.0 (RC1)

Before we discover the pre-uploaded artifacts, both Jungtaek and Hyukjin already made two blockers shared here. IIUC, it meant implicitly RC1 failure at that time. In addition to that, there are two correctness issues. So, I made up my mind to cast -1 for this RC1 before joining this thread.

Re: [VOTE] Release Spark 3.1.0 (RC1)

I just don't see a reason to believe there's a rush? just test it as normal? I did, you can too, etc. Or specifically what blocks the current RC? On Wed, Jan 6, 2021 at 5:46 PM Jungtaek Lim wrote: > No worries about the accident. We're human beings, and everyone can make a > mistake. Let's wait

Re: [VOTE] Release Spark 3.1.0 (RC1)

No worries about the accident. We're human beings, and everyone can make a mistake. Let's wait and see the response of INFRA-21266. Just a 2 cents, I'm actually leaning toward to skip 3.1.0 and start the release process for 3.1.1, as anyone could be some sort of "rushing" on verification on

Re: [VOTE] Release Spark 3.1.0 (RC1)

Thanks Dongjoon, Sean and Tom. I just thought that we could have some more bug fixes or some changes if RC1 passes as a regular release due to the relatively fewer RCs. I agree that if this RC passes, it's just that an RC passed normally per the regular process, and there's nothing wrong here. By

Re: [VOTE] Release Spark 3.1.0 (RC1)

I think it makes sense to wait and see what they say on INFRA-21266.   In the mean time hopefully people can start testing it and if no other problems found and vote passes can stay published.  It seems like the 2 issues above wouldn't be blockers in my opinion and could be handled in a 3.1.1

Re: [VOTE] Release Spark 3.1.0 (RC1)

OK, we'll have to update the release page to clarify there was never a real 3.1.0 release then. But I'm not suggesting releasing 3.1.0 _because_ it was published accidentally. I'm suggesting we figure out normally whether we would have released it, and if so, great. If not, fine we must skip the

Re: [VOTE] Release Spark 3.1.0 (RC1)

Thank you, Jacek, Sean, and Hyukjin. The release is a human-driven process. Everyone can make mistakes. For example, I released Apache Spark 2.2.3 with a missing pandoc, but we didn't touch it because it's a community-blessed official version. https://pypi.org/project/pyspark/2.2.3/ For

Re: [VOTE] Release Spark 3.1.0 (RC1)

Yes, it was my mistake. I faced the same issue as INFRA-20651 , and it is worse in my case because I misunderstood that RC and releases are separately released out. Right after this, I filed an INFRA JIRA to revert this at INFRA-21266

Re: [VOTE] Release Spark 3.1.0 (RC1)

Er, yeah uh oh. Did the staging repo accidentally get closed/released? Maybe I'm also missing something. If so then one way or the other we can't undo that as the 3.1.0 release in Maven, as far as I know. We can make a 3.1.1, but then it's kind of weird there was never any 3.1.0 source release.

Re: [VOTE] Release Spark 3.1.0 (RC1)

Hi, I'm curious why Spark 3.1.0 is already available in repo1.maven.org? https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/3.1.0/ Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on

Re: [VOTE] Release Spark 3.1.0 (RC1)

Seems like we have two PRs for both blockers, and one is already merged, nice. I will wait for a couple of days more before starting a new RC to make sure we catch more regressions before the new RC. Please keep testing this RC. I would appreciate it :-). 2021년 1월 6일 (수) 오후 2:28, Hyukjin Kwon 님이

Re: [VOTE] Release Spark 3.1.0 (RC1)

Actually, I will mark https://issues.apache.org/jira/browse/SPARK-34021 as a blocker too. For CRAN submission, we should fix it. 2021년 1월 6일 (수) 오후 1:47, Jungtaek Lim 님이 작성: > There's an issue SPARK-33635 [1] reported due to performance regression on > Kafka read between Spark 2.4 vs 3.0, which

Re: [VOTE] Release Spark 3.1.0 (RC1)

There's an issue SPARK-33635 [1] reported due to performance regression on Kafka read between Spark 2.4 vs 3.0, which sounds like a blocker. I'll mark this as a blocker, unless anyone has different opinions. 1. https://issues.apache.org/jira/browse/SPARK-33635 On Wed, Jan 6, 2021 at 9:01 AM

[VOTE] Release Spark 3.1.0 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.1.0. The vote is open until January 8th 4PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.1.0 [ ] -1 Do not release this package because