Re: On more stable curl releases
On Thu, 23 Mar 2023, Daniel Stenberg via curl-library wrote: Another adjustment to the cycle I think we can do is to even out the feature window and the feature freeze periods a little. A full three weeks for new features: PR proposal here: https://github.com/curl/curl/pull/10827 -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, 22 Mar 2023, Daniel Stenberg via curl-library wrote:
- Increase the post-release ("cool down") margin before we open the feature
window. We currently have it 5 days, we could double it to 10 days. That
would then reduce the feature window with the same amount of days leaving
us with 18 days to merge features.
I did not spot any complaints on that suggestion so I figure we should move in
that direction.
Another adjustment to the cycle I think we can do is to even out the feature
window and the feature freeze periods a little. A full three weeks for new
features:
Like this:
RELEASE (on a Wednesday)
10 days cool down (possiblity to decide to do a patch release)
FEATURE WINDOW (assuming nothing bad disturbed the peace)
21 days
BUG-FIX PERIOD
25 days of no new features/changes
RELEASE (on a Wednesday)
--
/ daniel.haxx.se
| Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://curl.se/support.html
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, Mar 22, 2023 at 04:10:32PM -0700, bch via curl-library wrote: > This is a curl binary, or a release tarball The daily tar balls are available at https://curl.se/snapshots/ > (how much processing *does* go on > from a repo checkout -> curl-x.y.z.tar.gz?)? I think it's just running autoreconf and maketgz. The latter does things like updates make pages, creates the MSVC build files, generates the CHANGES file and runs "make dist" to build the tar ball. -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, Mar 22, 2023 at 00:34 Daniel Stenberg via curl-library < curl-library@lists.haxx.se> wrote: > On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote: > > > It would be great if we can find more people to test pre-release curl > > versions > > I believe that is the holy grail we can only dream of. People are simply > hesitant to try pre-releases. Granted, we haven't tried this for a long > time > but we also know that barely anyone tries and finds issues in the nightly > builds we provide every day. This is a curl binary, or a release tarball (how much processing *does* go on from a repo checkout -> curl-x.y.z.tar.gz?)? -bch > > > Maybe we should label the daily snapshot one week before a scheduled > release > > as "rc1" and promote it that way and see what happens? > > That is a very cheap test we can certainly do. I'm pretty sure what the > outcome will be though... > > -- > > / daniel.haxx.se > | Commercial curl support up to 24x7 is available! > | Private help, bug fixes, support, ports, new features > | https://curl.se/support.html > -- > Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library > Etiquette: https://curl.se/mail/etiquette.html > -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, 22 Mar 2023, Jeffrey Walton wrote: To counter the lack of pre-release testing, schedule another release T+x days after the release of interest. The point of the "cool down" period after a release is to allow that *possibility*. We can do it if we feel we need to. In the projects I work with, master is always hot - it is always in a release state. Feature development and testing happen in a fork on a branch, so it does not taint master which is always hot. In fact, because it happens on a fork, it does not even taint the project itself. Sure, master is always hot for curl as well. That's not why we do feature freezes and cool down periods. We do them because A) merging features is riskier than fixing bugs. There is always the risk that something minor changed in a way that we didn't intend and that something needs an additional follow-up polish. By making sure we don't merge features too close to the release we reduce the risk that we ship problems caused by new features. B) by clearly articulating that "now we work on bugs" we communicate how we take fixing problems before a release very seriously and that we rather not waste maintainer's time and efforts on merging features then, but they are rather encouraged to help polish off any remaining rough corners. C) We like a clean linear commit history, so if we want to do a patch-release a week after a previous release, we better not merge any features into master before that patch-release happens. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, 22 Mar 2023, Dan Fandrich via curl-library wrote: The last two changes fix compile problems in two platforms, OS/400 and Haiku. Normally, I'd say that would be enough to trigger another release: users can't build curl when they used to be able to, but these are super marginal platforms. Are there even a dozen people out there compiling curl for them? I would guess there are significanltly less than a dozen people. Also, we get fixes for some platforms immediately after releases because that's when then build curl. If people with these lesser common platforms truly want to make sure that curl builds fine on their platforms on the day of the release, it would make sense for them to try it at least a few days *before* so that we could get the fix bundled. This kind of problems+fixes are thus a little self-inflicted and I would not want this behavior to be encouraged by us necessarily taking fixes like this as a reason to do another release. I really have no idea how many people there are using this support, though. Haiku's official curl package is only 6 releases old, so there's some development happening there. The Haiku build fix was only for cmake. I'm pretty sure that quirk already exists in the autotool build. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, Mar 22, 2023 at 09:17:48AM +0100, Daniel Stenberg via curl-library
wrote:
> So, how about this for adjusted release cycle and release management:
>
> - Increase the post-release ("cool down") margin before we open the feature
>window. We currently have it 5 days, we could double it to 10 days. That
>would then reduce the feature window with the same amount of days leaving
>us with 18 days to merge features.
>
> - Within the cool down period, we are only allowed to merge bug-fixes.
>
> - Lower the bar for a patch-release. Even "less important" regressions should
>be considered reason enough to do follow-up releases. And if they are not
>reported within the cool down period, chances are they are not important
>enough.
>
> - After a follow-up release, we start over with a new cool down period of 10
>days.
>
> - If we decide to do a patch release due to a regression, we set that release
>day N days into the future so that we can accumulate a few more fixes and
>get our ducks in order before we ship it. N is probably at least 7.
That looks pretty workable to me. The trick is going to be deciding when a
patch release is worthwhile. If we look at git now, 2 days after the last
release, there are already 7 commits. Two are CI adjustments (not
release-worthy), one fixes a problem introduced 9 years ago (not
release-worthy), one improves detection of bugs in debug builds (not
release-worthy) and one improves error detection in testing (not
release-worthy). So far it's pretty easy.
The last two changes fix compile problems in two platforms, OS/400 and Haiku.
Normally, I'd say that would be enough to trigger another release: users can't
build curl when they used to be able to, but these are super marginal
platforms. Are there even a dozen people out there compiling curl for them? If
the situation is such that we could send them all personal e-mails with the
patches then maybe it's not worth doing an entire new release. I really have no
idea how many people there are using this support, though. Haiku's official
curl package is only 6 releases old, so there's some development happening
there. There have also been at least 5 people contributing to OS/400 support
over the last few years, so maybe it's more than a dozen.
Perhaps the answer will be clear in another 8 days.
Dan
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, Mar 22, 2023 at 4:17 AM Daniel Stenberg via curl-library
wrote:
>
> On Wed, 22 Mar 2023, Stefan Eissing wrote:
>
> > Delaying the opening of the feature window after a release seems to be a
> > good balance. Right now, I have several PRs pending for the re-opening of
> > the window and it does not cost anything to have them wait a week more.
> > Feature branches already exist. Maintenance branches we'd like to avoid.
>
> So, how about this for adjusted release cycle and release management:
>
> - Increase the post-release ("cool down") margin before we open the feature
> window. We currently have it 5 days, we could double it to 10 days. That
> would then reduce the feature window with the same amount of days leaving
> us with 18 days to merge features.
>
> - Within the cool down period, we are only allowed to merge bug-fixes.
>
> - Lower the bar for a patch-release. Even "less important" regressions
> should
> be considered reason enough to do follow-up releases. And if they are not
> reported within the cool down period, chances are they are not important
> enough.
>
> - After a follow-up release, we start over with a new cool down period of 10
> days.
>
> - If we decide to do a patch release due to a regression, we set that
> release
> day N days into the future so that we can accumulate a few more fixes and
> get our ducks in order before we ship it. N is probably at least 7.
The last item is the important one due to lack of participation in
pre-release testing. You can ask for testers until you are blue in the
face, but you are only going to get a handful of testers. People won't
test the release until it is released.
To counter the lack of pre-release testing, schedule another release
T+x days after the release of interest. So if you release cURL 8.0.0
on January 1, then you schedule a 8.0.1 maintenance release on January
14 or January 28. You use the 10 days or 2 weeks to clean up the bugs
that crept in but were not caught in pre-release testing.
I don't know how that intersects with the "cool down" or feature
window. I don't use them. In the projects I work with, master is
always hot - it is always in a release state. Feature development and
testing happen in a fork on a branch, so it does not taint master
which is always hot. In fact, because it happens on a fork, it does
not even taint the project itself.
Another person came up with the strategy to counteract the lack of
testers. He told me I would not get enough testers to find the bugs we
hoped to catch before a release. I did not believe him. He was right.
Now I plan for the lack of testers by scheduling a maintenance release
following the release of interest.
Jeff
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
W dniu 2023-03-22 13:46, Daniel Stenberg napisał(a): On Wed, 22 Mar 2023, Daniel F via curl-library wrote: One option is to provide coverage data for each platform/test job, plus one extra created by merging results from all platform. With this approach you could check if there are any code paths which are not tested at all by checking merged results, and be able to check platform-specific coverage if needed. We have 130+ CI jobs. We could imagine that most of them could get a test coverage number. How would anyone make sensible use of a hundred different numbers? Not to mention how it would be difficult to present the numbers (and the view of what lines that were not executed). Not easily done I say. I think about something like lcov output, see examples at link below. It generates page for every source file, and code lines are marked in blue or red depending if code in that line was executed or not. This will give you an 0/1 answer if any test on any platform executed given line or not. Actual numbers (shown on the left) will not have any useful meaning for merged results. They may be more useful when checking results for one particular platform. https://medium.com/@xianpeng.shen/use-gcov-and-lcov-to-perform-code-coverage-testing-for-c-c-projects-c85708b91c78 Merging results from all these jobs is perhaps even more complicated since all of them run in their own sandboxes spread over a multitude of different services. These jobs could upload files with their coverage data somewhere. At the end of the day another job would take all files from that day and generate merged coverage report. lcov allows to merge multiple coverage files into one. -- Regards, Daniel -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, 22 Mar 2023, Daniel F via curl-library wrote: One option is to provide coverage data for each platform/test job, plus one extra created by merging results from all platform. With this approach you could check if there are any code paths which are not tested at all by checking merged results, and be able to check platform-specific coverage if needed. We have 130+ CI jobs. We could imagine that most of them could get a test coverage number. How would anyone make sensible use of a hundred different numbers? Not to mention how it would be difficult to present the numbers (and the view of what lines that were not executed). Not easily done I say. Merging results from all these jobs is perhaps even more complicated since all of them run in their own sandboxes spread over a multitude of different services. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
W dniu 2023-03-22 08:45, Daniel Stenberg via curl-library napisał(a): On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote: On a related note, what's the current code coverage? I haven't tried myself in a looong time, and there hasn't been a Coveralls build in 5 years. That would be a great graph to see on https://curl.se/dashboard.html But with all the different build configurations it would be hard to get a single meaningful number out of it; maybe that's why it hasn't happened since then. Yes. It is practically impossible for us to get a fair or even just almost correct coverage number due to the many build combinations and the way we run different tests in different jobs etc. And apart from not getting the plain number, using the coverage data to detect untested code paths is then also similarly complicated. If that was not enough, the coveralls service itself was flaky and often triggered false positives when the coverage suddenly "dropped to 0%" etc. Also, frankly, even when we *know* what parts that aren't tested very well that does not make the masses come running to help us write new or expand old tests to increase coverage. I am pretty sure that we can dig up code areas that currently are not tested well without any particular tools. Right now. One option is to provide coverage data for each platform/test job, plus one extra created by merging results from all platform. With this approach you could check if there are any code paths which are not tested at all by checking merged results, and be able to check platform-specific coverage if needed. -- Regards, Daniel -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, 22 Mar 2023, at 08:17, Daniel Stenberg via curl-library wrote:
> So, how about this for adjusted release cycle and release management:
>
> - Increase the post-release ("cool down") margin before we open the feature
> window. We currently have it 5 days, we could double it to 10 days. That
> would then reduce the feature window with the same amount of days leaving
> us with 18 days to merge features.
I think that's definitely a good idea. An advantage of 10 days is that it
always
spans one weekend whereas five days might - with poor luck - not include
any non weekday days.
Quite a lot of people - even in this always online world - would be more likely
to find time to test a new release during the upcoming weekend, but have no
chance of doing it on a working day/night.
--
Jeremy Nicoll - my opinions are my own.
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, 22 Mar 2023, Kamil Dudka wrote: If you look at the recent pull requests, you could hardly find one with green CI. Ah right, yes. Good point. This is also one of the reasons for regressions: our flaky CI setup makes it too easy to sometimes miss test failures because we have to live with them virtually never going all green. This said: fixing the CI to run green reliably is not an easy task. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wednesday, March 22, 2023 8:34:30 AM CET Daniel Stenberg via curl-library wrote: > On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote: > > > It would be great if we can find more people to test pre-release curl > > versions > > I believe that is the holy grail we can only dream of. People are simply > hesitant to try pre-releases. Granted, we haven't tried this for a long time > but we also know that barely anyone tries and finds issues in the nightly > builds we provide every day. Speaking of Fedora/CentOS, we could enable Packit in the upstream CI: https://packit.dev/ It automatically builds RPM packages for a matrix of supported Fedora/CentOS releases and the supported architectures, upon each pull request or update in the master branch. A successful build of RPMs implies that the upstream test-suite succeeded and the tests runs through valgrind on x86_64. The service also automatically creates package repositories (one for master branch and one for each pull request), which volunteers may consume to continuously test (lib)curl on their systems. Is this something that curl upstream would be interested in? The key problem I see with this approach is that Fedora developers do not have enough time to monitor and debug the (mostly random) failures of the upstream tests. And I think this is the main problem with the upstream CI already. If you look at the recent pull requests, you could hardly find one with green CI. Kamil > > Maybe we should label the daily snapshot one week before a scheduled > > release > > as "rc1" and promote it that way and see what happens? > > That is a very cheap test we can certainly do. I'm pretty sure what the > outcome will be though... -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, 22 Mar 2023, Stefan Eissing wrote:
Delaying the opening of the feature window after a release seems to be a
good balance. Right now, I have several PRs pending for the re-opening of
the window and it does not cost anything to have them wait a week more.
Feature branches already exist. Maintenance branches we'd like to avoid.
So, how about this for adjusted release cycle and release management:
- Increase the post-release ("cool down") margin before we open the feature
window. We currently have it 5 days, we could double it to 10 days. That
would then reduce the feature window with the same amount of days leaving
us with 18 days to merge features.
- Within the cool down period, we are only allowed to merge bug-fixes.
- Lower the bar for a patch-release. Even "less important" regressions should
be considered reason enough to do follow-up releases. And if they are not
reported within the cool down period, chances are they are not important
enough.
- After a follow-up release, we start over with a new cool down period of 10
days.
- If we decide to do a patch release due to a regression, we set that release
day N days into the future so that we can accumulate a few more fixes and
get our ducks in order before we ship it. N is probably at least 7.
--
/ daniel.haxx.se
| Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://curl.se/support.html
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote: On a related note, what's the current code coverage? I haven't tried myself in a looong time, and there hasn't been a Coveralls build in 5 years. That would be a great graph to see on https://curl.se/dashboard.html But with all the different build configurations it would be hard to get a single meaningful number out of it; maybe that's why it hasn't happened since then. Yes. It is practically impossible for us to get a fair or even just almost correct coverage number due to the many build combinations and the way we run different tests in different jobs etc. And apart from not getting the plain number, using the coverage data to detect untested code paths is then also similarly complicated. If that was not enough, the coveralls service itself was flaky and often triggered false positives when the coverage suddenly "dropped to 0%" etc. Also, frankly, even when we *know* what parts that aren't tested very well that does not make the masses come running to help us write new or expand old tests to increase coverage. I am pretty sure that we can dig up code areas that currently are not tested well without any particular tools. Right now. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
> Am 22.03.2023 um 00:10 schrieb Daniel Stenberg via curl-library
> :
>
> On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote:
>
>> Some examples of regressions would be the crash that prompted yesterday's
>> 8.0.1 point release, the noproxy host matching bug #9842 (fixed in 7.86.0)
>> and the wrong units bug in --write-out %{time_total} (fixed in 7.75.0). Of
>> these three examples, one resulted in a point release while the other two
>> did not.
>
> BTW, "regression" is just another word for "test coverage gap", since if we
> had tested the thing we would've detected the problem and the bug would not
> have been shipped. It is important that we learn from the regressions and
> improve the tests every time.
Yes, test coverage for regressions is the only thing that works *and* allows us
to evolve.
>
>> To look at it another way, between tags 7.75.0 and 7.88.1 (17 versions of
>> commits), 32 commits say "regression" in the comment, averaging 1.9
>> regressions per release.
>
> That is probably counting low since we don't always highlight them being
> regressions. Also, a regression can of course be something that *once* worked
> but that does not necessarily mean that it worked in the most previous
> release. It can also be five years ago.
>
>> I would like to see patch releases more often that include fixes that target
>> regressions.
>
>> A point release would only contain regression fixes and other fixes deemed
>> low risk.
>
> I agree that we do ship a certain rate of regressions and we fix them in
> subsequent releases.
>
> I understand your thinking with your proposal, but I am scared of going that
> route: because either we need to do the patch release management in a
> separate branch, so that we can allow the main branch to merge features for
> the next feature release and then we get a huge test and maintenance
> challenge: meaning we need to do everything that in two branches. Gone are
> the days of simple git history.
Speaking from experience in Apache httpd, release branch costs are considerable.
> Or we don't do different branches because the testing would be too difficult
> to handle. Then we need to handle this in the main branch and then it seems
> like we would basically not merge lots of things into the master branch for
> several weeks after a release. Also not ideal.
>
> I instead propose this much smaller and simpler change:
>
> We lower the bar for doing follow-up patch releases for regressions reported
> already within the period between the release and the re-opening of the
> feature window. For such we can do another release again already within a
> week or two. It is usually good to not stress them too much since then we get
> the chance to also find and fix other issues in the patch release.
Delaying the opening of the feature window after a release seems to be a good
balance. Right now, I have several PRs pending for the re-opening of the window
and it does not cost anything to have them wait a week more. Feature branches
already exist. Maintenance branches we'd like to avoid.
Kind Regards,
Stefan
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://curl.se/support.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote: It would be great if we can find more people to test pre-release curl versions I believe that is the holy grail we can only dream of. People are simply hesitant to try pre-releases. Granted, we haven't tried this for a long time but we also know that barely anyone tries and finds issues in the nightly builds we provide every day. Maybe we should label the daily snapshot one week before a scheduled release as "rc1" and promote it that way and see what happens? That is a very cheap test we can certainly do. I'm pretty sure what the outcome will be though... -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
RE: On more stable curl releases
> From: curl-library On Behalf Of Dan > Fandrich via curl-library > Sent: Wednesday, 22 March 2023 05:33 > To: curl-library@lists.haxx.se > Cc: Dan Fandrich > Subject: Re: On more stable curl releases > > Maybe we should label the daily snapshot one week before a > scheduled release as "rc1" and promote it that way and see what happens? I like that idea (and the other one about lowering the bar for patch releases too). Or even when the feature window closes to give testers more time. Personally, I always test official prereleases of libraries I'm using if time permits. Marcel -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Tue, Mar 21, 2023 at 12:40:28PM -0400, Timothe Litt via curl-library wrote: I expect that with frequent patch releases, curl would end up in the situation of most M$ releases whose strategy is- "wait for the other people to find the bugs and only take the nth patch release." And with fewer users, the bugs would take longer to turn up...which is a worsening spiral. That's true, but it's no worse than the situation we have now. I'm truly hoping that the big distros will continue releasing every version and not skip the regular releases or there's much less benefit to doing it this way. To make a more frequent patch release scheme work, you'd need to find a group of aggressive users/testers who'd find and report bugs early. And be willing to take the intermediate releases. It would be great if we can find more people to test pre-release curl versions, but my proposal essentially turns everyone who runs the regularly-scheduled release into such a tester, from a certain point of view. Some projects that do infrequent release might post an rc1 and rc2 release to get wider testing before releasing the final version. curl, embracing the release early, release often mantra and with its fast pace of development, doesn't really need to do that. Now, I'm deliberately mischaracterizing the proposal, but you could look at each regular .0 release as an rc1 and the patch release (if there is one) as the "real" release. In our case we truly believe based on our tests that the .0 is production ready and not an RC, but stuff happen. And if you can do that, you can also pull them into the regular release process... The kind of things that usually seem to go wrong are the marginal features that take a large user base to find. I don't know if there would be enough users in the early testing pools to make a big difference. If we're only leaving the patch release window open for a week, there's also not much time for said testers to do their testing thing. Maybe we should label the daily snapshot one week before a scheduled release as "rc1" and promote it that way and see what happens? Dan -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Wed, Mar 22, 2023 at 12:10:56AM +0100, Daniel Stenberg wrote: BTW, "regression" is just another word for "test coverage gap", since if we had tested the thing we would've detected the problem and the bug would not have been shipped. It is important that we learn from the regressions and improve the tests every time. That's a good way of looking at it! On a related note, what's the current code coverage? I haven't tried myself in a looong time, and there hasn't been a Coveralls build in 5 years. That would be a great graph to see on https://curl.se/dashboard.html But with all the different build configurations it would be hard to get a single meaningful number out of it; maybe that's why it hasn't happened since then. I understand your thinking with your proposal, but I am scared of going that route: because either we need to do the patch release management in a separate branch, so that we can allow the main branch to merge features for the next feature release and then we get a huge test and maintenance challenge: meaning we need to do everything that in two branches. Gone are the days of simple git history. My idea was to continue development in master as normal, but if something comes up that necessitates a point release, a point branch would be created and only the relevant commits would be cherry-picked from master into it. There would be nothing new in such a branch so people wouldn't need to look at it to figure out development history. But, you're right, you wouldn't be able to just run git log and search for the point release tags any more. IIRC, there's at least one point release in curl's history like that already. Or we don't do different branches because the testing would be too difficult to handle. We can configure most (all?) of the CI services to run on specific branches, so we should be able to have testing happen automatically on point branches. Worst case, someone would need to create a dummy PR before release, wait for the results then delete the PR. Not nice, but probably also not necessary. like we would basically not merge lots of things into the master branch for several weeks after a release. Also not ideal. That would definitely slow the pace of development too much. I instead propose this much smaller and simpler change: We lower the bar for doing follow-up patch releases for regressions reported already within the period between the release and the re-opening of the feature window. For such we can do another release again already within a week or two. It is usually good to not stress them too much since then we get the chance to also find and fix other issues in the patch release. That might be enough of a change to improve things. It's a minimal tweak to the existing workflow but with the improvement that more people get to eventually benefit from the quiet period after a release. I am a bit worried about the point Timothe brings up that if too many people (especially distributions) skip the regular releases and just wait for the point release, problems won't be found in time and people won't get a more stable point release. Still, that's really no worse than the place we're in now so it shouldn't stop us from trying. Dan -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
Daniel Stenberg via curl-library writes:
> On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote:
>
>> Some examples of regressions would be the crash that prompted
>> yesterday's 8.0.1 point release, the noproxy host matching bug #9842
>> (fixed in 7.86.0) and the wrong units bug in --write-out
>> %{time_total} (fixed in 7.75.0). Of these three examples, one
>> resulted in a point release while the other two did not.
>
> BTW, "regression" is just another word for "test coverage gap", since
> if we had tested the thing we would've detected the problem and the
> bug would not have been shipped. It is important that we learn from
> the regressions and improve the tests every time.
>
>> To look at it another way, between tags 7.75.0 and 7.88.1 (17
>> versions of commits), 32 commits say "regression" in the comment,
>> averaging 1.9 regressions per release.
>
> That is probably counting low since we don't always highlight them
> being regressions. Also, a regression can of course be something that
> *once* worked but that does not necessarily mean that it worked in the
> most previous release. It can also be five years ago.
>
>> I would like to see patch releases more often that include fixes
>> that target regressions.
>
>> A point release would only contain regression fixes and other fixes
>> deemed low risk.
>
> I agree that we do ship a certain rate of regressions and we fix them
> in subsequent releases.
>
> I understand your thinking with your proposal, but I am scared of
> going that route: because either we need to do the patch release
> management in a separate branch, so that we can allow the main branch
> to merge features for the next feature release and then we get a huge
> test and maintenance challenge: meaning we need to do everything that
> in two branches. Gone are the days of simple git history.
>
> Or we don't do different branches because the testing would be too
> difficult to handle. Then we need to handle this in the main branch
> and then it seems like we would basically not merge lots of things
> into the master branch for several weeks after a release. Also not
> ideal.
>
> I instead propose this much smaller and simpler change:
>
> We lower the bar for doing follow-up patch releases for regressions
> reported already within the period between the release and the
> re-opening of the feature window. For such we can do another release
> again already within a week or two. It is usually good to not stress
> them too much since then we get the chance to also find and fix other
> issues in the patch release.
Yes please. This'd be a big help for distributions. It means we don't
have to rely on additional bug reports in a distro it find out about
something fixed a day or two after the release.
best,
sam
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://curl.se/support.html
signature.asc
Description: PGP signature
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
On Tue, 21 Mar 2023, Dan Fandrich via curl-library wrote:
Some examples of regressions would be the crash that prompted yesterday's
8.0.1 point release, the noproxy host matching bug #9842 (fixed in 7.86.0)
and the wrong units bug in --write-out %{time_total} (fixed in 7.75.0). Of
these three examples, one resulted in a point release while the other two
did not.
BTW, "regression" is just another word for "test coverage gap", since if we
had tested the thing we would've detected the problem and the bug would not
have been shipped. It is important that we learn from the regressions and
improve the tests every time.
To look at it another way, between tags 7.75.0 and 7.88.1 (17 versions of
commits), 32 commits say "regression" in the comment, averaging 1.9
regressions per release.
That is probably counting low since we don't always highlight them being
regressions. Also, a regression can of course be something that *once* worked
but that does not necessarily mean that it worked in the most previous
release. It can also be five years ago.
I would like to see patch releases more often that include fixes that target
regressions.
A point release would only contain regression fixes and other fixes deemed
low risk.
I agree that we do ship a certain rate of regressions and we fix them in
subsequent releases.
I understand your thinking with your proposal, but I am scared of going that
route: because either we need to do the patch release management in a separate
branch, so that we can allow the main branch to merge features for the next
feature release and then we get a huge test and maintenance challenge: meaning
we need to do everything that in two branches. Gone are the days of simple git
history.
Or we don't do different branches because the testing would be too difficult
to handle. Then we need to handle this in the main branch and then it seems
like we would basically not merge lots of things into the master branch for
several weeks after a release. Also not ideal.
I instead propose this much smaller and simpler change:
We lower the bar for doing follow-up patch releases for regressions reported
already within the period between the release and the re-opening of the
feature window. For such we can do another release again already within a week
or two. It is usually good to not stress them too much since then we get the
chance to also find and fix other issues in the patch release.
--
/ daniel.haxx.se
| Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://curl.se/support.html
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: On more stable curl releases
I agree with the problem statement. But there's no easy answer.
I expect that with frequent patch releases, curl would end up in the
situation of most M$ releases whose strategy is- "wait for the other
people to find the bugs and only take the nth patch release." And with
fewer users, the bugs would take longer to turn up...which is a
worsening spiral.
To make a more frequent patch release scheme work, you'd need to find a
group of aggressive users/testers who'd find and report bugs early.
And be willing to take the intermediate releases.
And if you can do that, you can also pull them into the regular release
process...
Many projects have a separate release stream/channel to support this
sort of model. The challenge is recruiting and retaining the right
number and type of early release testers...
I've been down this road with OS releases - it's much more difficult to
implement an effective, lasting scheme than it first appears.
On 21-Mar-23 12:22, Dan Fandrich via curl-library wrote:
I've been getting a feeling that there have been more curl functional
regressions (compared to plain bugs) in releases over the last few
years. I don't know if that's true or not, but the pace of bug fixes
generally has been going up (see
https://curl.se/dashboard1.html#bugfix-frequency). I'm splitting off
regressions into a separate category of bugs since those directly fail
curl's objective of allowing applications to upgrade and without
noticing any negative change in behaviour. The remaining bugs genrally
fix issues that existed for a long time which applications presumably
hadn't hit yet, or aren't concerned about, or fix a problem created
since the last release.
Some examples of regressions would be the crash that prompted
yesterday's 8.0.1 point release, the noproxy host matching bug #9842
(fixed in 7.86.0) and the wrong units bug in --write-out %{time_total}
(fixed in 7.75.0). Of these three examples, one resulted in a point
release while the other two did not.
The problem with allowing regressions to live until the next scheduled
release without fixing them is that as the number of such regressions
goes up, the chances of everybody being able to upgrade to a scheduled
release without a problem goes down. If there ends up being one
regression in every release cycle then each curl release is no longer
better than the last one because it's going break somebody's
application. And the one after that is going to break something else.
There ends up being no release that that meets our goals for users
(assuming such a goal is even possible).
curl probably isn't at that point yet, but I would hate for it to get
there. To try to see how things stand, I looked at three Linux
distributions and counted the patches they added to curl over the past
two years of releases. These three distros, Fedora, Debian and
Alpine, are widely used and have frequent curl releases, and typically
upgrade to every new release, so I could get enough data points to be
interesting. I looked at patches between 7.75.0 (or 7.74.0 in one
case that didn't ship 7.75.0) and 7.88.1. I only counted patches that
seemed to be fixing curl functionality and ignored patches to do with
the packaging process itself.
Fedora: released 18 curl versions in that time, of which 10 included
functional patches: 56% were patched releases
Debian: released 11 curl versions in that time, of which 6 included
functional patches: 55% were patched releases
Alpine: released 17 curl versions in that time, of which 4 included
functional patches: 24% were patched releases
So, between all of them, 43% of their releases included patches that
the distros felt could not wait until the next release to include.
To look at it another way, between tags 7.75.0 and 7.88.1 (17 versions
of commits), 32 commits say "regression" in the comment, averaging 1.9
regressions per release.
The EARLY-RELEASE.md document states the criteria needed to do an
extraordinary release, but I think the bar should be lowered to
include more regression fixes. For example, the noproxy bug did not
get an out of band release, and we received bug reports about it for
weeks (IIRC, somebody was asking even in the last week about Apple's
release policy because they shipped the problematic version). I
personally developed a script relying on %{time_total} on a distro
that shipped 7.74.0 without realizing that version had the wrong time
units. When I ran the script on another system with a newer curl, it
mysteriously failed because of it.
With an installation base of 1e+10 even a "minor" regression can still
affect a huge number of people. Not all users have a distro
maintainer to curate patches and ensure they have a stable version.
I would like to see patch releases more often that include fixes that
target regressions. That's easy for me to say since I'm not the one
doing the work of making those extra releases, but that
