On Tue, Dec 06, 2022 at 06:59:30PM +0100, Tim Düsterhus wrote:
> William,
>
> On 12/6/22 15:37, William Lallemand wrote:
> > As I already mentionned, I don't really like the "latest" keyword for
> > the OpenSSL version as it prevent us to have reproducible builds.
> > It updates versions without warning, even major ones.
>
> I agree and also was not really happy with the 'latest' back when it was
> introduced in the first place, but didn't care strongly enough to speak up
> then.
No problem, it's always easier to see what we've missed after it reaches
its limits :-)
> > What I suggest is to stop using "latest" for the "git push" CI, but
> > using it only in a separate CI (once a day/week I don't know). And only
> > use fixed version of the libraries on the CI so builds are not broken by
> > external components. Because in my opinion the "git push" CI is to test
> > our code, not the libraries.
> >
>
> I don't even think such a weekly job is necessary [1]. Add an item to the
> release checklist "check if any new SSL versions are available and add them
> to matrix.py" and this should be fine, all SSL versions will then be updated
> every 6 months and can also be updated on demand for important releases.
> It's similar to how I simply rerun the Coccinelle patches from time to time
> to fix whatever crept in since the last release.
I'm seeing a slightly different approach here. We need to keep in mind
that:
- dev must be as forward-thinking as possible; as such, being aware
that something is going to break soon is useful, particularly when
it comes to forthcoming changes in third party dependencies. I.e.
openssl 3 breakage was reported upfront and that was fortunate
because a lot of changes were required to adapt to it, long before
it was even released. So I don't know if the weekly job is the best
option or not, but right now, watching form time to time how it goes
with openssl 3.1-dev is useful. Here I'm speaking about a branch, it
thus means that we're following what significant changes may bother
us. It's purely integration testing, and we could decide that one
major upgrade will not be the target for the current version because
it requires too much work. But if we focus on such a target we should
be as up-to-date as possible (no need to alarm on past breakage if
the issue is since fixed). That's where following a -latest from a
lib can be useful.
- dev must also tell us if we're going to cause breakage on what we
expect to support. E.g. once we see that 3.1-dev looks good and we
decide to enable it, we'll also need to be able to quickly follow it,
because until it's released, it may revert some changes or change its
API and cause breakage. It's not like a stable version we can trust,
and we need to know as well if we finally prefer to give up because
going back-and-forth indicates a third-party dependency is not stable
enough for example, or we could mention before the release that it's
only experimental for now.
- even when adopting a stable release, we'll suffer from bugs in third
party dependencies just like we suffer from our own bugs. If OpenSSL
3.1.0 is released and we decide to follow its stable branch, maybe
some tests will randomly fail from time to time. And we probably
don't want to manually change its version every week to follow the
first post-release fixes that we may need to stabilize the CI. In
this case again, following the latest from a given branch would be
useful, but it becomes less critical. Indeed, generally the most
annoying issues should stabilize during the very first versions, and
we don't need to care as much about updating the lib once the CI works
fine again.
- however, once we release a version, the most important is to make sure
there are as few moving parts as possible. We should not change any
dependency automatically for the CI. Triggering the build on the same
release two weeks apart should execute the same tests, otherwise we
can think that we broke something during a backport while it's not
the case (that's in fact how we first noticed the problem).
Thus I think that:
- for dev, adopting the latest version of a chosen branch (released or
dev) if possible would be desirable so that we're informed ASAP that
the branch we're expecting to support is experiencing/causing some
trouble ;
- for dev, watching weekly or so the level of support of future versions
that we have not yet planed to support would help prepare internals
where needed ;
- for stable, we'd just stick to the version that was in effect that the
moment of the release (i.e. the version the tests were run on during
dev).
And that would make sense considering that a release is nothing more than
freezing changes between dev and stable.
I'm particularly conscious that there may be technical limitations for
the -latest stuff and am not firmly sold on that. I'm just explaining
some principles that should IMHO help us orient our choices when facing
constraints (and I think that what we have right now is already quite
good for -dev, though it could indeed possibly be improved).
Just my two cents,
Willy
