Thanks for the suggestions from Dongjoon, Liangchi, Min, and Xiao! Now we make it clear that it's a soft cut and we can still merge important code changes to branch-3.2 before RC. Let's keep the branch cut date as July 1st.
On Thu, Jun 17, 2021 at 1:41 PM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > > First, I think you are saying "branch-3.2"; > > To Xiao. Yes, it's was a typo of "branch-3.2". > > > We do strongly prefer to cut the release for Spark 3.2.0 including all > the patches under SPARK-30602. > > This way, we can backport the other performance/operability > enhancements tickets under SPARK-33235 into branch-3.2 to be released in > future Spark 3.2.x patch releases. > > To Min, after releasing 3.2.0, only bug fixes are allowed for 3.2.1+ as > Xiao wrote. > > > > On Wed, Jun 16, 2021 at 9:42 PM Xiao Li <gatorsm...@gmail.com> wrote: > >> To Liang-Chi, I'm -1 for postponing the branch cut because this is a soft >>> cut and the committers still are able to commit to `branch-3.3` according >>> to their decisions. >> >> >> First, I think you are saying "branch-3.2"; >> >> Second, the "so cut" means no "code freeze", although we cut the branch. >> To avoid releasing half-baked and unready features, the release >> manager needs to be very careful when cutting the RC. Based on what is >> proposed here, the RC date is the actual code freeze date. >> >> This way, we can backport the other performance/operability enhancements >>> tickets under SPARK-33235 into branch-3.2 to be released in future Spark >>> 3.2.x patch releases. >> >> >> This is not allowed based on the policy. Only bug fixes can be merged to >> the patch releases. Thus, if we know it will introduce major performance >> regression, we have to turn the feature off by default. >> >> Xiao >> >> >> >> Min Shen <victor....@gmail.com> 于2021年6月16日周三 下午3:22写道： >> >>> Hi Gengliang, >>> >>> Thanks for volunteering as the release manager for Spark 3.2.0. >>> Regarding the ongoing work of push-based shuffle in SPARK-30602, we are >>> close to having all the patches merged to master to enable push-based >>> shuffle. >>> Currently, there are 2 PRs under SPARK-30602 that are under active >>> review (SPARK-32922 and SPARK-35671), and hopefully can be merged soon. >>> We should be able to post the PRs for the other 2 remaining tickets >>> (SPARK-32923 and SPARK-35546) early next week. >>> >>> The tickets under SPARK-30602 are the minimum set of patches to enable >>> push-based shuffle. >>> We do have other performance/operability enhancements tickets under >>> SPARK-33235 that are needed to fully contribute what we have internally for >>> push-based shuffle. >>> However, these are optional for enabling push-based shuffle. >>> We do strongly prefer to cut the release for Spark 3.2.0 including all >>> the patches under SPARK-30602. >>> This way, we can backport the other performance/operability enhancements >>> tickets under SPARK-33235 into branch-3.2 to be released in future Spark >>> 3.2.x patch releases. >>> I understand the preference of not postponing the branch cut date. >>> We will check with Dongjoon regarding the soft cut date and the >>> flexibility for including the remaining tickets under SPARK-30602 into >>> branch-3.2. >>> >>> Best, >>> Min >>> >>> On Wed, Jun 16, 2021 at 1:20 PM Liang-Chi Hsieh <vii...@gmail.com> >>> wrote: >>> >>>> >>>> Thanks Dongjoon. I've talked with Dongjoon offline to know more this. >>>> As it is soft cut date, there is no reason to postpone it. >>>> >>>> It sounds good then to keep original branch cut date. >>>> >>>> Thank you. >>>> >>>> >>>> >>>> Dongjoon Hyun-2 wrote >>>> > Thank you for volunteering, Gengliang. >>>> > >>>> > Apache Spark 3.2.0 is the first version enabling AQE by default. I'm >>>> also >>>> > watching some on-going improvements on that. >>>> > >>>> > https://issues.apache.org/jira/browse/SPARK-33828 (SQL Adaptive >>>> Query >>>> > Execution QA) >>>> > >>>> > To Liang-Chi, I'm -1 for postponing the branch cut because this is a >>>> soft >>>> > cut and the committers still are able to commit to `branch-3.3` >>>> according >>>> > to their decisions. >>>> > >>>> > Given that Apache Spark had 115 commits in a week in various areas >>>> > concurrently, we should start QA for Apache Spark 3.2 by creating >>>> > branch-3.3 and allowing only limited backporting. >>>> > >>>> > https://github.com/apache/spark/graphs/commit-activity >>>> > >>>> > Bests, >>>> > Dongjoon. >>>> > >>>> > >>>> > On Wed, Jun 16, 2021 at 9:19 AM Liang-Chi Hsieh < >>>> >>>> > viirya@ >>>> >>>> > > wrote: >>>> > >>>> >> First, thanks for being volunteer as the release manager of Spark >>>> 3.2.0, >>>> >> Gengliang! >>>> >> >>>> >> And yes, for the two important Structured Streaming features, RocksDB >>>> >> StateStore and session window, we're working on them and expect to >>>> have >>>> >> them >>>> >> in the new release. >>>> >> >>>> >> So I propose to postpone the branch cut date. >>>> >> >>>> >> Thank you! >>>> >> >>>> >> Liang-Chi >>>> >> >>>> >> >>>> >> Gengliang Wang-2 wrote >>>> >> > Thanks, Hyukjin. >>>> >> > >>>> >> > The expected target branch cut date of Spark 3.2 is *July 1st* on >>>> >> > https://spark.apache.org/versioning-policy.html. However, I >>>> notice that >>>> >> > there are still multiple important projects in progress now: >>>> >> > >>>> >> > [Core] >>>> >> > >>>> >> > - SPIP: Support push-based shuffle to improve shuffle efficiency >>>> >> > <https://issues.apache.org/jira/browse/SPARK-30602> >>>> >> > >>>> >> > [SQL] >>>> >> > >>>> >> > - Support ANSI SQL INTERVAL types >>>> >> > <https://issues.apache.org/jira/browse/SPARK-27790> >>>> >> > - Support Timestamp without time zone data type >>>> >> > <https://issues.apache.org/jira/browse/SPARK-35662> >>>> >> > - Aggregate (Min/Max/Count) push down for Parquet >>>> >> > <https://issues.apache.org/jira/browse/SPARK-34952> >>>> >> > >>>> >> > [Streaming] >>>> >> > >>>> >> > - EventTime based sessionization (session window) >>>> >> > <https://issues.apache.org/jira/browse/SPARK-10816> >>>> >> > - Add RocksDB StateStore as external module >>>> >> > <https://issues.apache.org/jira/browse/SPARK-34198> >>>> >> > >>>> >> > >>>> >> > I wonder whether we should postpone the branch cut date. >>>> >> > cc Min Shen, Yi Wu, Max Gekk, Huaxin Gao, Jungtaek Lim, Yuanjian >>>> >> > Li, Liang-Chi Hsieh, who work on the projects above. >>>> >> > >>>> >> > On Tue, Jun 15, 2021 at 4:34 PM Hyukjin Kwon < >>>> >> >>>> >> > gurwls223@ >>>> >> >>>> >> > > wrote: >>>> >> > >>>> >> >> +1, thanks. >>>> >> >> >>>> >> >> On Tue, 15 Jun 2021, 16:17 Gengliang Wang, < >>>> >> >>>> >> > ltnwgl@ >>>> >> >>>> >> > > wrote: >>>> >> >> >>>> >> >>> Hi, >>>> >> >>> >>>> >> >>> As the expected release date is close, I would like to >>>> volunteer as >>>> >> the >>>> >> >>> release manager for Apache Spark 3.2.0. >>>> >> >>> >>>> >> >>> Thanks, >>>> >> >>> Gengliang >>>> >> >>> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Sent from: >>>> http://apache-spark-developers-list.1001551.n3.nabble.com/ >>>> >> >>>> >> --------------------------------------------------------------------- >>>> >> To unsubscribe e-mail: >>>> >>>> > dev-unsubscribe@.apache >>>> >>>> >> >>>> >> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>>
