Re: A new release process?
On Tue, Jan 3, 2017 at 1:32 PM, Jacob Champion wrote: > On 12/29/2016 08:16 PM, David Zuelke wrote: >> >> The tl;dr of this approach is that >> >> - any x.y.z release only introduces bugfixes. These releases are done >> every four weeks, like clockwork. If a fix doesn't make the cut for a >> release, it'll end up in the next one; - x.y.0 releases, on the other >> hand, may introduce new features, fixes, and deprecations, but no >> breaking changes; - x.0.0 releases are the big ones (think PHP 7.0.0 >> in late 2015). where backward compatibility may be changed, etc. > > My favorite pieces here are > - the introduction of a release cadence > - the separation of bugfix releases from feature releases > > Both make life easier -- for us, for users, and for intermediate maintainers > -- for the reasons you've mentioned. Consider also our consumers, module authors who rely on us to provide an API they can design to. All of our subsidiary components, be it apr, or expat, or pcre, or openssl, or libxml2, or lua, or nghttp2... and most of the expansions to httpd, be it python or perl or whatnot, all follow essentially this model. That isn't how httpd evolved; if you look far far back into the pre-1.0 history and even through the evolution up to 1.3.12, there wasn't a particular rhyme or reason other than what the major patch submitters wanted to have happen. So when pools are introduced, minor version bump - a subversion bump was obviously not going to cut it. But there was not necessarily any compatibility at all with previously compiled modules. This all changed radically with 1.3.14 and 2.0 later on. But the aversion from back in the 1.2 and 1.3 days to declaring the version minors are precious resources has not changed. In the meantime, nghttp2 just published the 18th version minor by year end last year. And remains solidly compatible due to good design from the get go, it is still major version 1. Will be interesting to watch and see when enough API issues have accumulated to ditch major v1 for v2. We are certainly the enigma, and if there is one reason for developers to disabused themselves from your API, that is likely making shit up as you go along. I know a lot of folks frustrated back in the PHP 3-4 days, and it seems like this solution has worked out very well for the PHP crew. And there was an equal amount of chaos around Perl's versioning and lifecycle evolution, but that also seems to be working out pretty well. >> [...] >> >> There are a bunch of technicalities that would need adjusting to fit >> HTTPD, such as release intervals, release management (for PHP, every >> x.y.* series has two managers who jointly coordinate releases), etc, >> but overall the idea is, IMO, worth considering. > > Our current process is also fairly labor-intensive, so moving to a quick > release cadence without simultaneously fixing that is not likely to end up > well, IMO. There are other discussions on the list for improving our > automated QA, which I think is probably step 1. Releases themselves are not terribly challenging. The investment, outside of the reviews and testing we hope many committers are engaged in, boils down to less than an hour of actual mechanics, and some time to prepare specific announcement details or merge back in some sequestered security patches if those have to be applied. >> As a, more or less, "outside observer", I happen to think that the >> current method of voting on finals, instead of a practice of rolling >> out RCs (that are then left up for testing for at least a week), is >> fundamentally broken. The 2.4 changelog in particular is littered >> with releases that were never officially published. For users, that's >> really confusing. > > To a certain extent I think this is something that people get used to > easily... but for security patches in *particular*, I agree. It's > disconcerting to hear that CVE 2099-BLAH was fixed in 2.4.Z and then find > out that 2.4.Z was never released. We say this, that subversion numbers are cheap, and the lack of 2.4.24 really isn't confusing. But I have to answer questions from at least 15 individuals over the course of the following month when that happens :) So apparently it might not confuse us, but it does confuse the rest of the world more than we might expect. >> For maintainers, it's painful to start over the >> process each time, and it sometimes leads to months and months >> without a release that contains certain fixes. > > The pain of discarding and restarting your manual tests because someone else > found a regression on another platform two hours into the release vote comes > to mind, yes. On that, adding features to a security release 4 days before tag and roll is a sure fire path to a bad tag. But features-features-features, gotta collect them all :) Looking at a long term solution to the specific quirk that occurred in 2.4.24 and 2.4.25, but the general truth of throwing in one more late last feature usually turns out how
Re: A new release process?
On 12/29/2016 08:16 PM, David Zuelke wrote: The tl;dr of this approach is that - any x.y.z release only introduces bugfixes. These releases are done every four weeks, like clockwork. If a fix doesn't make the cut for a release, it'll end up in the next one; - x.y.0 releases, on the other hand, may introduce new features, fixes, and deprecations, but no breaking changes; - x.0.0 releases are the big ones (think PHP 7.0.0 in late 2015). where backward compatibility may be changed, etc. My favorite pieces here are - the introduction of a release cadence - the separation of bugfix releases from feature releases Both make life easier -- for us, for users, and for intermediate maintainers -- for the reasons you've mentioned. [...] There are a bunch of technicalities that would need adjusting to fit HTTPD, such as release intervals, release management (for PHP, every x.y.* series has two managers who jointly coordinate releases), etc, but overall the idea is, IMO, worth considering. Our current process is also fairly labor-intensive, so moving to a quick release cadence without simultaneously fixing that is not likely to end up well, IMO. There are other discussions on the list for improving our automated QA, which I think is probably step 1. As a, more or less, "outside observer", I happen to think that the current method of voting on finals, instead of a practice of rolling out RCs (that are then left up for testing for at least a week), is fundamentally broken. The 2.4 changelog in particular is littered with releases that were never officially published. For users, that's really confusing. To a certain extent I think this is something that people get used to easily... but for security patches in *particular*, I agree. It's disconcerting to hear that CVE 2099-BLAH was fixed in 2.4.Z and then find out that 2.4.Z was never released. For maintainers, it's painful to start over the process each time, and it sometimes leads to months and months without a release that contains certain fixes. The pain of discarding and restarting your manual tests because someone else found a regression on another platform two hours into the release vote comes to mind, yes. For the record, though, I think having strong automated QA processes will help both of these issues more than the release cadence will. It's just that the cadence reinforces the QA and the QA reinforces the cadence; they go hand-in-hand in a very nice positive feedback loop. Then a backport goes wrong (still using SVN, in my opinion, does not help there, but that's a whole different discussion :)), I've started some discussions on how we might introduce feature- and fix-branches, and hopefully better backport workflows, into our current use of SVN. (I'm a git user myself, and the current backport process is more error-prone than I'm used to as well. But I have no desire to start a holy war, and I'm confident we can find an SVN workflow that works well.) and a regression is in the latest release until someone eventually picks up a fix. Much of this, and many of the "what do we backport from trunk" and "I'd like to squeeze in a change I've had sitting around locally, please wait with the new release, because who knows when the next one after that will be" are, from what I can tell, a significant source of discord on this mailing list. All these unnecessary distractions that deteriorate personal relationships, while at the same time slowing down the pace of the project (several people have already pointed out Nginx's rate of innovation in comparison) and raising the threshold for contributions, can be fixed. PHP is the perfect example, and I think HTTPD would be wise to at least consider following this example. Happy New Year! Thanks for your input! --Jacob
Re: A new release process?
On Thu, Dec 29, 2016 at 11:16 PM, David Zuelke wrote: > As a, more or less, "outside observer", I happen to think that the current > method of voting on finals, instead of a practice of rolling out RCs (that > are then left up for testing for at least a week), is fundamentally broken. > The 2.4 changelog in particular is littered with releases that were never > officially published. For users, that's really confusing. For maintainers, > it's painful to start over the process each time, and it sometimes leads to > months and months without a release that contains certain fixes. Then a > backport goes wrong (still using SVN, in my opinion, does not help there, but > that's a whole different discussion :)), and a regression is in the latest > release until someone eventually picks up a fix. It seems to me that most "not released" versions are immediately superseded. The practice of throwing away the number instead of calling the initial contents an RC shouldn't contribute meaningfully to a fix being delayed. New blood for release managers would be nice, and smaller/more frequent /more predictable releases would probably help that. -- Eric Covener cove...@gmail.com
A new release process?
Hi everyone, Given the several current threads where there's arguing about what and how and when to release features, backported or not, I'd like to offer a tale of a project that was, more or less, in the same dire spot, and pulled itself out of that misery with great success and universal acclaim. That project is PHP. Back in 2010, it was stuck at version 5.3.something, there was no standards around what feature goes into a minor release, how a feature is even accepted for inclusion, how security updates were treated, and so forth. Enter: https://wiki.php.net/rfc/releaseprocess The tl;dr of this approach is that - any x.y.z release only introduces bugfixes. These releases are done every four weeks, like clockwork. If a fix doesn't make the cut for a release, it'll end up in the next one; - x.y.0 releases, on the other hand, may introduce new features, fixes, and deprecations, but no breaking changes; - x.0.0 releases are the big ones (think PHP 7.0.0 in late 2015). where backward compatibility may be changed, etc. This makes for very predictable update cycles, and for pretty easy updating, as it's very very unusual that a x.y.z release breaks existing behavior. At the same time, since x.y.0 releases are made roughly once a year, bigger changes do not exist in a limbo branch forever, but will see the light of day eventually (and, again, after a predictable amount of time). There is a defined roadmap with active/security maintenance status and EOL dates for all versions; see http://php.net/supported-versions.php From a user perspective, this change has been fantastic. Once virtually stagnant, development of the language itself has been much more active; users have a much better idea up front around when fixes or features will land; upgrades have become easy as there are clear definitions of what kind of changes a version may contain. But most importantly, the language itself has seen a steady flow of new features as a result from this discipline. For features and bigger changes, an RFC is written with rationale, analysis, suggestion of implementation, and then it can be voted on. Once accepted, it can be committed, reviewed, and merged. Even for "outside" contributors, this process is transparent, and the mental threshold towards contribution has been lowered greatly, as acceptance of an RFC basically means that the feature will go into a new release anywhere between within a few months and a year and a bit (depending on when in the release cycle it happens). There are a bunch of technicalities that would need adjusting to fit HTTPD, such as release intervals, release management (for PHP, every x.y.* series has two managers who jointly coordinate releases), etc, but overall the idea is, IMO, worth considering. As a, more or less, "outside observer", I happen to think that the current method of voting on finals, instead of a practice of rolling out RCs (that are then left up for testing for at least a week), is fundamentally broken. The 2.4 changelog in particular is littered with releases that were never officially published. For users, that's really confusing. For maintainers, it's painful to start over the process each time, and it sometimes leads to months and months without a release that contains certain fixes. Then a backport goes wrong (still using SVN, in my opinion, does not help there, but that's a whole different discussion :)), and a regression is in the latest release until someone eventually picks up a fix. Much of this, and many of the "what do we backport from trunk" and "I'd like to squeeze in a change I've had sitting around locally, please wait with the new release, because who knows when the next one after that will be" are, from what I can tell, a significant source of discord on this mailing list. All these unnecessary distractions that deteriorate personal relationships, while at the same time slowing down the pace of the project (several people have already pointed out Nginx's rate of innovation in comparison) and raising the threshold for contributions, can be fixed. PHP is the perfect example, and I think HTTPD would be wise to at least consider following this example. Happy New Year! David