On Fri, 18 Dec 2015, Rainer Gerhards wrote:
Date: Fri, 18 Dec 2015 14:46:34 +0100
From: Rainer Gerhards <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] release cycle - was: rsyslog 8.15.0 (v8-stable) released
2015-12-17 22:52 GMT+01:00 Michael Biebl <[email protected]>:
2015-12-17 16:51 GMT+01:00 Rainer Gerhards <[email protected]>:
2015-12-17 16:47 GMT+01:00 Thomas D. <[email protected]>:
Hi,
I agree with Michael.
While I understand Rainers concerns in general this is different: For
you there are only test files missing. But for distributions there is no
working v8.15 release (tests are really important for us).
Can't you apply a patch? I remeber well in that long discussion over a
year ago that you were on of the strong proponents of "it's easy to
patch if something is a small nit"?
I just want to understand that change of position, if it is one.
We run the test-suite as part of the Debian build (that's why I
noticed the failure). So this is a serious issue for the Debian build.
But sure, if it's too much of a hassle, I'll just add the missing
files as a downstream patch.
I agree that it would be almost no work for me to do a re-release.
It's more a policy argument. Let me explain, especially as I consider
changing policy:
Up until ~15 month ago, we released when there was need to. Need was defined as
- important enough (set of bugfixes)
- new functionality
This resulted in various releases. We had the stable/devel releases.
Stable releases were rare, devel frequent.
Now, we have scheduled releases. Actually, a release is triggered when
we hit a certain calender date, irrelevant of whether or not there is
need to release (there is always one or two minor fixes, so we will
probably never exprience a totally blank release). We also have
switched to stable releases only, and done so without grief (basically
because a) we have improved testing and b) users didn't use devel at
all).
I just dug into the old discussion. A good entry point is probably
this here, where we talk about patches:
http://lists.adiscon.net/pipermail/rsyslog/2014-October/038796.html
The new system works reasonably well. It has it's quircks, though.
Let's look at a concrete example:
8.14.0, to me, was an absolutely horrible release. The worst we have
done in the past 2 to 3 years. I worked hard on fixing some real bad
race issues with JSON variables. Friday before the release I was ready
to release that work, which would be really useful for folks that make
heavy use of those variables. Then, over the weekend and Monday, it
turned out that we may get unwanted regressions that weren't detected
earlier (NO testbench can mimic a heavy-used production system, so
let's not get into "we need better tests" blurb). The end result was
that I pulled the plug on release day, and what we finally released
was 8.13.0 plus a few small things. All problems with variables
persisted. If I had have half a week to a week (don't remember
exactly) more, we could have done a real release instead of the 8.13
re-incarnation. But, hey, we run on a schedule.
Now 8.15.0 fixes these problems (except for the json-c induced
segfault, which we cannot fix in rsyslog). I also has all other "8.14"
enhancements and fixes and so is actually worth 3 month of work. It is
a *very heavy* release. Usually, I'd never released such a fat release
shortly before the holiday period. Not that I distrust it, and we
really got some new testing capabilites (really, really much better),
so it is probably the most solid release for a longer time (besides
the small quirk with the missing testbench files). But in general I
don't like to do releases when I know there is very limited resources
available to deal with problems. That's the old datacenter guy in me.
But, again, hey, we run on a schedule.
There have similiar occasions in the past 14 month. That's the
downside. And due to the 6-week cycle things usually do not get really
bad.
The scheduled model has a lot of good things as well. First of all,
everyone (users and contributors) know when the next release will be.
This also means you can promise to include something into a specific
release. However, usually users know when the release happens, but not
what will be part of it, so in a sense it's not much better than
before IMO. The new model has advantages for me: less releases mean
less work. Also, I do not longer really need to think about when to do
a release, which feature is important engouh and so. I just look at
the calender and know that, for example, in 2016, November 15th we
will have a release, no matter if I am present, no matter what is done
code-wise etc (we actually had, for the first time everm a release
while I was in vacation and it went really well as I learned later).
That really eases my task.
the other big change was the change in mindset from "Ok, we made a stable
release, we can break things now and fix them over time before the next stable
release" to "master should be kept in a releasable state at all times"
The big adantage I see of the schedules release is that it (almost) eliminates
the pressure to get something that's not quite ready into _this_ release because
if it doesn't make this release, you know that the next one isn't far away.
I see 8.14 as an example of the new approach working perfectly. Instead of
agonizing over things for weeks as we tried to work things out (and it did take
weeks to finally track things down in this case), you reverted the problem (just
about everything, since we couldn't identify the problem) made a release and we
kept going working on the problem.
All of this bases on the "we release every 6 weeks, interim releases
happen only for emergencies and anything else may be pulled as
patches" policy. If we now begin to say "this problem is inconvenient
to ..{pick somebody}", we need to do a re-release we get into trouble.
I wonder which groups of "sombody" are important enough to grant
non-emergency releases. Are only distro maintainers important enough?
Probably not. So enterprise users? Mmmm.. maybe small enterprises as
well? Who judges this? So let's assume every user is as important as
every other (an idea I really like). If I then look at my change logs,
I think I would need to release more frequent. In essence, I would
need to release again when it is needed, which is, surprise, the
as-needed schedule).
Rsyslog is not a project big enough to do an even more complex release
schedule. To keep things managable to me, I need to release either
a) as-needed
b) on schedule (except for *true* emergencies)
And *that all* is the reason for my reluctance to break the release
policy because this time distro maintainers experience the bug versus
end users.
I am currently tempted to switch back to "as-needed" mode, even though
this means more work for me.
I would not like to see us move back to the "as-needed" mode the way things were
before, it was WAY too long between releases. Every 6 weeks is great, if
anything we could probably bump this out a couple weeks if it made a noticable
difference to your workload.
That said, there are two slight tweaks I think we should consider.
If we are on 8 week schedule, at week 7 we should look at outstanding bug,
especially on new features, and try agressively reverting changes. most of the
time this will just be 'remove the new feature', but once in a great while it
will end up with the 8.14 situation where you revert just about everything,
keeping only 'obviously correct, trivial bugfixes'. It wouldn't hurt to have a
standard e-mail go out to the list at the start of week 7 reminding people that
the release is only a week away, test the master branch.
If there is a regression (including build problems) reported within the first
couple days after a new release, and the fix is available within a day or two,
and the fix is either small (<10 lines), or the fix is reverting patches applied
during this release cycle , then I think releasing a .1 within week 9 is
reasonable. After about the midpoint of week 9, only critical things would
trigger a new release. So a .1, .2, etc would only come out during week 9 except
for a critical bug.
If this makes releases enough more complicated that we need to push out to a 8
week, or even 10 week release cycle, I think it would be worth it.
I'd rather do a 'brown paper bag' release every year or so than have a release
with a fix that is trivial to fix out there. Based on how things have gone over
the last year, I think such releases would be pretty rare.
I think the releases over the last year have worked out well. The pace of
releases has drastically increased, the effort of releases has decreased, and
the reliability of releases and rsyslog overall has increased.
going to a pure 'as needed' release is a problem, a 'small' bugfix that isn't
worth a release to most people, is very much needed by someone tripping over
that bug in production. Having a guarantee that the bug will be in a release
withing a known period is worth a lot.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
