Re: [DISCUSS] Make release cadence predictable
While the point about being less time is probably correct, yeah if something is half-done, we should keep them in the master branch, and/or don't expose it to the end users (which I believe we usually do). Good thing is that we can make the schedule predictable. Suppose that the branchcut date is pinned, then people can schedule/estimate their work based on the pinned schedule :-). On Thu, 16 Feb 2023 at 04:32, Sean Owen wrote: > I don't think there is a delay per se, because there is no hard release > date to begin with, to delay with respect to. It's been driven by, "feels > like enough stuff has gone in" and "someone is willing to roll a release", > and that happens more like every 8-9 months. This would be a shift not only > in expectation - lower the threshold for 'enough stuff has gone in' to > probably match a 6 month cadence - but also a shift in policy to a release > train-like process. If something isn't ready then it just waits another 6 > months. > > You're right, the problem is kind of - what is something is in process in > a half-baked state? you don't really want to release half a thing, nor do > you want to develop it quite separately from the master branch. > It is worth asking what prompts this, too. Just, we want to release > earlier and more often? > > On Wed, Feb 15, 2023 at 1:19 PM Maciej wrote: > >> Hi, >> >> Sorry for a silly question, but do we know what exactly caused these >> delays? Are these avoidable? >> >> It is not a systematic observation, but my general impression is that we >> rarely delay for sake of individual features, unless there is some soft >> consensus about their importance. Arguably, these could be postponed, >> assuming we can adhere to the schedule. >> >> And then, we're left with large, multi-task features. A lot can be done >> with proper timing and design, but in our current process there is no way >> to guarantee that each of these can be delivered within given time window. >> How are we going to handle these? Delivering half-baked things is hardly >> satisfying solution and more rigid schedule can only increase pressure on >> maintainers. Do we plan to introduce something like feature branches for >> these, to isolate upcoming release in case of delay? >> >> On 2/14/23 19:53, Dongjoon Hyun wrote: >> >> +1 for Hyukjin and Sean's opinion. >> >> Thank you for initiating this discussion. >> >> If we have a fixed-predefined regular 6-month, I believe we can persuade >> the incomplete features to wait for next releases more easily. >> >> In addition, I want to add the first RC1 date requirement because RC1 >> always did a great job for us. >> >> I guess `branch-cut + 1M (no later than 1month)` could be the reasonable >> deadline. >> >> Thanks, >> Dongjoon. >> >> >> On Tue, Feb 14, 2023 at 6:33 AM Sean Owen wrote: >> >>> I'm fine with shifting to a stricter cadence-based schedule. Sometimes, >>> it'll mean some significant change misses a release rather than delays it. >>> If people are OK with that discipline, sure. >>> A hard 6-month cycle would mean the minor releases are more frequent and >>> have less change in them. That's probably OK. We could also decide to >>> choose a longer cadence like 9 months, but I don't know if that's better. >>> I assume maintenance releases would still be as-needed, and major >>> releases would also work differently - probably no 4.0 until next year at >>> the earliest. >>> >>> On Tue, Feb 14, 2023 at 3:01 AM Hyukjin Kwon >>> wrote: >>> Hi all, *TL;DR*: Branch cut for every 6 months (January and July). I would like to discuss/propose to make our release cadence predictable. In our documentation, we mention as follows: In general, feature (“minor”) releases occur about every 6 months. Hence, Spark 2.3.0 would generally be released about 6 months after 2.2.0. However, the reality is slightly different. Here is the time it took for the recent releases: - Spark 3.3.0 took 8 months - Spark 3.2.0 took 7 months - Spark 3.1 took 9 months Here are problems caused by such delay: - The whole related schedules are affected in all downstream projects, vendors, etc. - It makes the release date unpredictable to the end users. - Developers as well as the release managers have to rush because of the delay, which prevents us from focusing on having a proper regression-free release. My proposal is to branch cut every 6 months (January and July that avoids the public holidays / vacation period in general) so the release can happen twice every year regardless of the actual release date. I believe it both makes the release cadence predictable, and relaxes the burden about making releases. WDYT? >>> >> -- >> Best regards, >> Maciej Szymkiewicz >> >> Web: https://zero323.net >> PGP: A30CEF0C31A501EC >> >>
Re: [VOTE][RESULT] Release Spark 3.3.2 (RC1)
Awesome! On Thu, 16 Feb 2023 at 06:39, Dongjoon Hyun wrote: > Great! Thank you, Liang-Chi! > > Dongjoon. > > On Wed, Feb 15, 2023 at 9:22 AM L. C. Hsieh wrote: > >> The vote passes with 12 +1s (4 binding +1s). >> Thanks to all who helped with the release! >> >> (* = binding) >> +1: >> - Mridul Muralidharan (*) >> - Dongjoon Hyun (*) >> - Sean Owen (*) >> - Enrico Minack >> - Bjørn Jørgensen >> - Yikun Jiang >> - Yang Jie >> - Yuming Wang >> - John Zhuge >> - William Hyun >> - Chao Sun >> - L. C. Hsieh (*) >> >> +0: None >> >> -1: None >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: [VOTE][RESULT] Release Spark 3.3.2 (RC1)
Great! Thank you, Liang-Chi! Dongjoon. On Wed, Feb 15, 2023 at 9:22 AM L. C. Hsieh wrote: > The vote passes with 12 +1s (4 binding +1s). > Thanks to all who helped with the release! > > (* = binding) > +1: > - Mridul Muralidharan (*) > - Dongjoon Hyun (*) > - Sean Owen (*) > - Enrico Minack > - Bjørn Jørgensen > - Yikun Jiang > - Yang Jie > - Yuming Wang > - John Zhuge > - William Hyun > - Chao Sun > - L. C. Hsieh (*) > > +0: None > > -1: None > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Re: [DISCUSS] Make release cadence predictable
I don't think there is a delay per se, because there is no hard release date to begin with, to delay with respect to. It's been driven by, "feels like enough stuff has gone in" and "someone is willing to roll a release", and that happens more like every 8-9 months. This would be a shift not only in expectation - lower the threshold for 'enough stuff has gone in' to probably match a 6 month cadence - but also a shift in policy to a release train-like process. If something isn't ready then it just waits another 6 months. You're right, the problem is kind of - what is something is in process in a half-baked state? you don't really want to release half a thing, nor do you want to develop it quite separately from the master branch. It is worth asking what prompts this, too. Just, we want to release earlier and more often? On Wed, Feb 15, 2023 at 1:19 PM Maciej wrote: > Hi, > > Sorry for a silly question, but do we know what exactly caused these > delays? Are these avoidable? > > It is not a systematic observation, but my general impression is that we > rarely delay for sake of individual features, unless there is some soft > consensus about their importance. Arguably, these could be postponed, > assuming we can adhere to the schedule. > > And then, we're left with large, multi-task features. A lot can be done > with proper timing and design, but in our current process there is no way > to guarantee that each of these can be delivered within given time window. > How are we going to handle these? Delivering half-baked things is hardly > satisfying solution and more rigid schedule can only increase pressure on > maintainers. Do we plan to introduce something like feature branches for > these, to isolate upcoming release in case of delay? > > On 2/14/23 19:53, Dongjoon Hyun wrote: > > +1 for Hyukjin and Sean's opinion. > > Thank you for initiating this discussion. > > If we have a fixed-predefined regular 6-month, I believe we can persuade > the incomplete features to wait for next releases more easily. > > In addition, I want to add the first RC1 date requirement because RC1 > always did a great job for us. > > I guess `branch-cut + 1M (no later than 1month)` could be the reasonable > deadline. > > Thanks, > Dongjoon. > > > On Tue, Feb 14, 2023 at 6:33 AM Sean Owen wrote: > >> I'm fine with shifting to a stricter cadence-based schedule. Sometimes, >> it'll mean some significant change misses a release rather than delays it. >> If people are OK with that discipline, sure. >> A hard 6-month cycle would mean the minor releases are more frequent and >> have less change in them. That's probably OK. We could also decide to >> choose a longer cadence like 9 months, but I don't know if that's better. >> I assume maintenance releases would still be as-needed, and major >> releases would also work differently - probably no 4.0 until next year at >> the earliest. >> >> On Tue, Feb 14, 2023 at 3:01 AM Hyukjin Kwon wrote: >> >>> Hi all, >>> >>> *TL;DR*: Branch cut for every 6 months (January and July). >>> >>> I would like to discuss/propose to make our release cadence predictable. >>> In our documentation, we mention as follows: >>> >>> In general, feature (“minor”) releases occur about every 6 months. Hence, >>> Spark 2.3.0 would generally be released about 6 months after 2.2.0. >>> >>> However, the reality is slightly different. Here is the time it took for >>> the recent releases: >>> >>>- Spark 3.3.0 took 8 months >>>- Spark 3.2.0 took 7 months >>>- Spark 3.1 took 9 months >>> >>> Here are problems caused by such delay: >>> >>>- The whole related schedules are affected in all downstream >>>projects, vendors, etc. >>>- It makes the release date unpredictable to the end users. >>>- Developers as well as the release managers have to rush because of >>>the delay, which prevents us from focusing on having a proper >>>regression-free release. >>> >>> My proposal is to branch cut every 6 months (January and July that >>> avoids the public holidays / vacation period in general) so the release can >>> happen twice >>> every year regardless of the actual release date. >>> I believe it both makes the release cadence predictable, and relaxes the >>> burden about making releases. >>> >>> WDYT? >>> >> > -- > Best regards, > Maciej Szymkiewicz > > Web: https://zero323.net > PGP: A30CEF0C31A501EC > >
Re: [DISCUSS] Make release cadence predictable
Hi, Sorry for a silly question, but do we know what exactly caused these delays? Are these avoidable? It is not a systematic observation, but my general impression is that we rarely delay for sake of individual features, unless there is some soft consensus about their importance. Arguably, these could be postponed, assuming we can adhere to the schedule. And then, we're left with large, multi-task features. A lot can be done with proper timing and design, but in our current process there is no way to guarantee that each of these can be delivered within given time window. How are we going to handle these? Delivering half-baked things is hardly satisfying solution and more rigid schedule can only increase pressure on maintainers. Do we plan to introduce something like feature branches for these, to isolate upcoming release in case of delay? On 2/14/23 19:53, Dongjoon Hyun wrote: +1 for Hyukjin and Sean's opinion. Thank you for initiating this discussion. If we have a fixed-predefined regular 6-month, I believe we can persuade the incomplete features to wait for next releases more easily. In addition, I want to add the first RC1 date requirement because RC1 always did a great job for us. I guess `branch-cut + 1M (no later than 1month)` could be the reasonable deadline. Thanks, Dongjoon. On Tue, Feb 14, 2023 at 6:33 AM Sean Owen wrote: I'm fine with shifting to a stricter cadence-based schedule. Sometimes, it'll mean some significant change misses a release rather than delays it. If people are OK with that discipline, sure. A hard 6-month cycle would mean the minor releases are more frequent and have less change in them. That's probably OK. We could also decide to choose a longer cadence like 9 months, but I don't know if that's better. I assume maintenance releases would still be as-needed, and major releases would also work differently - probably no 4.0 until next year at the earliest. On Tue, Feb 14, 2023 at 3:01 AM Hyukjin Kwon wrote: Hi all, *TL;DR*: Branch cut for every 6 months (January and July). I would like to discuss/propose to make our release cadence predictable. In our documentation, we mention as follows: In general, feature (“minor”) releases occur about every 6 months. Hence, Spark 2.3.0 would generally be released about 6 months after 2.2.0. However, the reality is slightly different. Here is the time it took for the recent releases: * Spark 3.3.0 took 8 months * Spark 3.2.0 took 7 months * Spark 3.1 took 9 months Here are problems caused by such delay: * The whole related schedules are affected in all downstream projects, vendors, etc. * It makes the release date unpredictable to the end users. * Developers as well as the release managers have to rush because of the delay, which prevents us from focusing on having a proper regression-free release. My proposal is to branch cut every 6 months (January and July that avoids the public holidays / vacation period in general) so the release can happen twice every year regardless of the actual release date. I believe it both makes the release cadence predictable, and relaxes the burden about making releases. WDYT? -- Best regards, Maciej Szymkiewicz Web:https://zero323.net PGP: A30CEF0C31A501EC OpenPGP_signature Description: OpenPGP digital signature
[VOTE][RESULT] Release Spark 3.3.2 (RC1)
The vote passes with 12 +1s (4 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Mridul Muralidharan (*) - Dongjoon Hyun (*) - Sean Owen (*) - Enrico Minack - Bjørn Jørgensen - Yikun Jiang - Yang Jie - Yuming Wang - John Zhuge - William Hyun - Chao Sun - L. C. Hsieh (*) +0: None -1: None - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
