Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Xinrong Meng
Thank you Hyukjin! :) I would prefer to cut v3.4.0-rc4 now if there are no objections. On Fri, Mar 10, 2023 at 7:01 AM Hyukjin Kwon wrote: > BTW doing another RC isn't a very big deal (compared to what I did before > :-) ) since it's not a canonical release yet. > > On Fri, Mar 10, 2023 at

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Hyukjin Kwon
BTW doing another RC isn't a very big deal (compared to what I did before :-) ) since it's not a canonical release yet. On Fri, Mar 10, 2023 at 7:58 AM Hyukjin Kwon wrote: > I guess directly tagging is fine too I guess. > I don't mind cutting the RC4 right away either if that's what you prefer.

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Hyukjin Kwon
I guess directly tagging is fine too I guess. I don't mind cutting the RC4 right away either if that's what you prefer. On Fri, Mar 10, 2023 at 7:06 AM Xinrong Meng wrote: > Hi All, > > Thank you all for catching that. Unfortunately, the release script failed > to push the release tag

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Xinrong Meng
Hi All, Thank you all for catching that. Unfortunately, the release script failed to push the release tag v3.4.0-rc3 to branch-3.4. Sorry about the issue. Shall we cut v3.4.0-rc4 immediately or wait until March 14th? On Fri, Mar 10, 2023 at 5:34 AM Sean Owen wrote: > If the issue were just

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Sean Owen
If the issue were just tags, then you can simply delete the tag and re-tag the right commit. That doesn't change a commit log. But is the issue that the relevant commits aren't in branch-3.4? Like I don't see the usual release commits in https://github.com/apache/spark/commits/branch-3.4 Yeah OK

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Dongjoon Hyun
No~ We cannot in the AS-IS commit log status because it's screwed already as Emil wrote. Did you check the branch-3.2 commit log, Sean? Dongjoon. On Thu, Mar 9, 2023 at 11:42 AM Sean Owen wrote: > We can just push the tags onto the branches as needed right? No need to > roll a new release > >

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Sean Owen
We can just push the tags onto the branches as needed right? No need to roll a new release On Thu, Mar 9, 2023, 1:36 PM Dongjoon Hyun wrote: > Yes, I also confirmed that the v3.4.0-rc3 tag is invalid. > > I guess we need RC4. > > Dongjoon. > > On Thu, Mar 9, 2023 at 7:13 AM Emil Ejbyfeldt >

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Dongjoon Hyun
Yes, I also confirmed that the v3.4.0-rc3 tag is invalid. I guess we need RC4. Dongjoon. On Thu, Mar 9, 2023 at 7:13 AM Emil Ejbyfeldt wrote: > It might being caused by the v3.4.0-rc3 tag not being part of the 3.4 > branch branch-3.4: > > $ git log --pretty='format:%d %h' --graph

Adding pause() method to pyspark.sql.streaming.StreamingQuery

2023-03-09 Thread Mich Talebzadeh
Hi, Currently for Spark Streaming we have the following class: pyspark.sql.streaming.StreamingQuery There are a number of useful methods, for example stop() which stops the streaming process gracefully. Can we add another method pause() so w can pause the processing. This will come handy

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Emil Ejbyfeldt
It might being caused by the v3.4.0-rc3 tag not being part of the 3.4 branch branch-3.4: $ git log --pretty='format:%d %h' --graph origin/branch-3.4 v3.4.0-rc3 | head -n 10 * (HEAD, origin/branch-3.4) e38e619946 * f3e69a1fe2 * 74cf1a32b0 * 0191a5bde0 * afced91348 | * (tag: v3.4.0-rc3)

[VOTE] Release Apache Spark 3.4.0 (RC3)

2023-03-09 Thread Xinrong Meng
Please vote on releasing the following candidate(RC3) as Apache Spark version 3.4.0. The vote is open until 11:59pm Pacific time *March 14th* and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.4.0 [ ] -1 Do not release this