Re: [ANNOUNCE] haproxy-1.6.16
Hi Tim, On Tue, Mar 23, 2021 at 08:46:31PM +0100, Tim Düsterhus wrote: > Willy, > > On 3/19/21 6:32 PM, Willy Tarreau wrote: > >> Am 19.03.21 um 18:29 schrieb Willy Tarreau: > >>> [...] I know Christopher will harrass me if I > >>> forget anyway :-) > >> > >> Be assured, I will as well :-) > > > > No doubt :-) And that offloads my brain quite a lot! > > > > After the update haproxy.org shows a stray '>' above the table. This is > caused by an additional '>' after the '' element for the 1.8 row. > You might want to remove that. Wow, good catch, I didn't notice it and even with the indications it took me a moment to locate it! Now fixed, thanks! Willy
Re: [ANNOUNCE] haproxy-1.6.16
Willy, On 3/19/21 6:32 PM, Willy Tarreau wrote: >> Am 19.03.21 um 18:29 schrieb Willy Tarreau: >>> [...] I know Christopher will harrass me if I >>> forget anyway :-) >> >> Be assured, I will as well :-) > > No doubt :-) And that offloads my brain quite a lot! > After the update haproxy.org shows a stray '>' above the table. This is caused by an additional '>' after the '' element for the 1.8 row. You might want to remove that. Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-1.6.16
Hi Lukas,
On Tue, Mar 23, 2021 at 12:21:02AM +0100, Lukas Tribus wrote:
> I agree that finding the sweet spot can be difficult, but I have to
> say I share Vincent's concerns. I do feel like currently we backport
> *a lot*, especially on those near-EOLed trees. When looking at the
> list of backported patches, I don't feel like the age and remaining
> lifetime is taken into consideration.
It usually is, but let's face it, we're all humans, and when you spend
a whole day doing backports from 2.3 to 1.6, it's not always trivial
to decide whether it's better to drop certain fixes, especially when
you remember having worked on them and know the trouble the bug they
fix can cause.
> I don't want to be the monday morning quarterback,
No, feel free to!
> but in 1.7 we have
> 853926a9ac ("BUG/MEDIUM: ebtree: use a byte-per-byte memcmp() to
> compare memory blocks") and I quote from the commit message:
>
> > This is not correct and definitely confuses address sanitizers. In real
> > life the problem doesn't have visible consequences.
> > [...]
> > there is no way such a multi-byte access will cross a page boundary
> > and end up reading from an unallocated zone. This is why it was
> > never noticed before.
>
> This sounds like a backport candidate for "warm" stable branches
> (maybe), but 1.7 and 1.8 feel too "cold" for this, even 8 - 9 months
> ago.
I agree in principle, but I'm pretty sure we've later met a situation
where it resulted in a read past the end of a node. I mean, given that
commits cannot be amended after they're merged, it's quite common to
later figure that a fix for a somewhat innocent bug used to fix a more
important one. This is exactly one of the problems that GregKH explained
in his talk about issues caused by CVE (https://youtu.be/HeeoTE9jLjM).
Normally we address this by adding a personal comment to the commit
during the backport.
> This backport specifically causes a build failure of 1.7.13 on musl
> (#760) - because of a missing backport, but that's just an example.
Yes but this is a good example. We are particularly careful about
such issues and sometimes wait longer to pick a series at once in
order to get a fix and its own fixes. But this does not always work
that well :-/ And we're trying to force ourselves not to backport
too fast to older versions. However we know that backporting a patch
to 5 versions is roughly the same amount of work as doing it for 2
as long as it's still hot in the developer's head, while it would
be 5 if done with a delay, so there's a sweet spot to find there.
We've even thought about using the -next branch in stable repos to
queue delayed fixes and wait for their own fixes, but that implies
doing other non-trivial changes to our workflow.
> 39
> "MINOR" patches made it into 1.6.16, 62 patches in 1.7.13. While it is
> true that a lot of "MINOR" tagged patches are actually important, this
> is still a large number for a tree that is supposed to die so soon.
I agree. But on the other hand many times we've found that we've left a
nasty bug because we didn't backport something. The problem here is in
fact the frequency of such releases. If we managed to emit them more
often with colder fixes, this would be much less of a concern. Right
now, patches are "ejected" once a year or so, and while some of them
are totally cold and save, others are much more recent and may present
a problem. I'm well accustomed to this issue, I was facing the same with
the extended LTS kernels. In theory everyone would love to see a sliding
window of patches, in practice some fixes are quite important and may
rely on other ones so it's not that easy to keep some out of the way.
> Very rarely do users build from source from such old trees anyway (and
> those users would be especially conservative, definitely not
> interested in generic, non-important improvements).
Sure but the goal is not to bring improvements there but to fix real
issues for which they upgrade. On the other hand we also know that such
users will not update their prod unless they face an issue. And when
you face an issue after 3-4 years of operations, it's rarely one tagged
major because it would have been spotted much earlier. Often it's a
corner case of a minor one that was not identified during its fixing
(e.g. a crash every 3 months because the memcmp() above crosses a page
boundary when comparing a string past its 8ths byte and lands into an
unallocated area).
> > But with this in mind, there were two options:
> > - not releasing the latest fixes
>
> You are talking about whether to publish a release or not for tree
> X.Y, with the backports that are already in the tree. I don't think
> that's the issue.
That's the concern Vincent initially brought, this is why I'm speaking
about this option.
> I think the discussion should be about what commits land in those old
> trees in the first place.
Yes I agree.
> And I don't believe it is scalable to make
> those decisions during your backporting sess
Re: [ANNOUNCE] haproxy-1.6.16
Hello Willy,
On Sat, 20 Mar 2021 at 10:09, Willy Tarreau wrote:
> > 1.6 was EOL last year, I don't understand why there is a last release.
>
> There were some demands late last year and early this year to issue a
> last one with pending fixes to "flush the pipe" but it was terribly
> difficult to find enough time to go through the whole chain with the
> other nasty bugs that kept us busy.
>
> > Both 1.6 and 1.7 are marked for critical fixes but many fixes are pushed
> > in it. The risk is to introduce a late regression in this last version.
>
> There's always such a risk when doing backports unfortunately and it's
> always difficult to set the boundary between what is needed and what
> not. A lot of the issues I'm seeing there are crash-related, and
> others address not-so-funny recent changes in compilers behaviors
> leading to bad code generation. There are also some that were possibly
> not strictly necessary, but then they're obvious fixes (like the one
> on the timer that's incorrectly compared), and whose possible
> consequences are not always trivial to imagine (such as checks looping
> at 100% CPU every 24 days maybe depending on the tick's sign).
I agree that finding the sweet spot can be difficult, but I have to
say I share Vincent's concerns. I do feel like currently we backport
*a lot*, especially on those near-EOLed trees. When looking at the
list of backported patches, I don't feel like the age and remaining
lifetime is taken into consideration.
I don't want to be the monday morning quarterback, but in 1.7 we have
853926a9ac ("BUG/MEDIUM: ebtree: use a byte-per-byte memcmp() to
compare memory blocks") and I quote from the commit message:
> This is not correct and definitely confuses address sanitizers. In real
> life the problem doesn't have visible consequences.
> [...]
> there is no way such a multi-byte access will cross a page boundary
> and end up reading from an unallocated zone. This is why it was
> never noticed before.
This sounds like a backport candidate for "warm" stable branches
(maybe), but 1.7 and 1.8 feel too "cold" for this, even 8 - 9 months
ago.
This backport specifically causes a build failure of 1.7.13 on musl
(#760) - because of a missing backport, but that's just an example. 39
"MINOR" patches made it into 1.6.16, 62 patches in 1.7.13. While it is
true that a lot of "MINOR" tagged patches are actually important, this
is still a large number for a tree that is supposed to die so soon.
Very rarely do users build from source from such old trees anyway (and
those users would be especially conservative, definitely not
interested in generic, non-important improvements).
> But with this in mind, there were two options:
> - not releasing the latest fixes
You are talking about whether to publish a release or not for tree
X.Y, with the backports that are already in the tree. I don't think
that's the issue.
I think the discussion should be about what commits land in those old
trees in the first place. And I don't believe it is scalable to make
those decisions during your backporting sessions. Instead I believe we
should be more conservative when suggesting backports in the commit
message. Currently, we say "should/can be backported to X.Y" based on
whether it is *technically* possible to do so for supported trees, not
if it makes sense considering the age and lifetime of the suggested
tree. This is why I'm proposing a commit author should make such
considerations when suggesting backports. Avoiding backports to cold
trees of no-impact improvements and minor fixes for rare corner cases
should be a goal.
Unless we insist every single bug needs to be fixed on every single
supported release branch.
lukas
Re: [ANNOUNCE] haproxy-1.6.16
Hi Vincent, On Fri, Mar 19, 2021 at 11:50:24PM +0100, Vincent Bernat wrote: > ? 19 mars 2021 17:34 +01, Christopher Faulet: > > > HAProxy 1.6.16 was released on 2021/03/19. It added 71 new commits > > after version 1.6.15. > > 1.6 was EOL last year, I don't understand why there is a last release. There were some demands late last year and early this year to issue a last one with pending fixes to "flush the pipe" but it was terribly difficult to find enough time to go through the whole chain with the other nasty bugs that kept us busy. > Both 1.6 and 1.7 are marked for critical fixes but many fixes are pushed > in it. The risk is to introduce a late regression in this last version. There's always such a risk when doing backports unfortunately and it's always difficult to set the boundary between what is needed and what not. A lot of the issues I'm seeing there are crash-related, and others address not-so-funny recent changes in compilers behaviors leading to bad code generation. There are also some that were possibly not strictly necessary, but then they're obvious fixes (like the one on the timer that's incorrectly compared), and whose possible consequences are not always trivial to imagine (such as checks looping at 100% CPU every 24 days maybe depending on the tick's sign). Ideally, with enough time, older versions should be released a bit more often, but we also know that reality doesn't always match the ideal case, and that old versions are rarely upgraded in field :-/ But with this in mind, there were two options: - not releasing the latest fixes, leaving those on 1.6.15 facing occasional issues with their issues ; - releasing 1.6.16 with all latest fixes to grant a bit more time to 1.6.15 users to migrate, knowing that in case of a regression there, they could roll back to 1.6.15 and finish exactly in the same situation as #1 above. As such I think it's always preferable to publish the available fixes because in the best case it helps those facing issues, and in the worst case the situation remains exactly the same in case of regression. Last point, a final version also allows to update the version string to mention that it's the last version (this entry didn't exist in 1.6 but it seems to have helped with 1.9 as we're not seeing reports anymore). It's important when the production management is share between several persons (typically a contractor is in charge of some updates). It then helps those less aware of maintenance cycles to figure that they're using something outdated. With this said, maybe you or anyone else has better suggestions to propose for old versions that we could try for 1.7 and then 1.8. This is a tough job and good ideas are always welcome especially if they result in less effort and better quality. Cheers, Willy
Re: [ANNOUNCE] haproxy-1.6.16
❦ 19 mars 2021 17:34 +01, Christopher Faulet: > HAProxy 1.6.16 was released on 2021/03/19. It added 71 new commits > after version 1.6.15. 1.6 was EOL last year, I don't understand why there is a last release. Both 1.6 and 1.7 are marked for critical fixes but many fixes are pushed in it. The risk is to introduce a late regression in this last version. -- Make sure your code "does nothing" gracefully. - The Elements of Programming Style (Kernighan & Plauger)
Re: [ANNOUNCE] haproxy-1.6.16
On Fri, Mar 19, 2021 at 06:30:27PM +0100, Tim Düsterhus wrote: > Willy, > > Am 19.03.21 um 18:29 schrieb Willy Tarreau: > > [...] I know Christopher will harrass me if I > > forget anyway :-) > > Be assured, I will as well :-) No doubt :-) And that offloads my brain quite a lot! Cheers, Willy
Re: [ANNOUNCE] haproxy-1.6.16
Willy, Am 19.03.21 um 18:29 schrieb Willy Tarreau: > [...] I know Christopher will harrass me if I > forget anyway :-) Be assured, I will as well :-) Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-1.6.16
On Fri, Mar 19, 2021 at 06:25:24PM +0100, Tim Düsterhus wrote: > Christopher, > Willy, > > Am 19.03.21 um 17:34 schrieb Christopher Faulet: > > The 1.6 branch is EOL now. Thus, it is the last 1.6 release. No further > > Don't forget to update the color of the table on haproxy.org :-) Same > probably applies to 2.1, even if you don't rule out critical fixes for > it. And I also believe you wanted to put 1.8 to "critical only" as well. Yes I know, I need to do a pass on all the page (table and statuses). I'll do it later this evening once I'm back home or this week-end if I'm too tired to do it right. I know Christopher will harrass me if I forget anyway :-) Thanks! Willy
Re: [ANNOUNCE] haproxy-1.6.16
Christopher, Willy, Am 19.03.21 um 17:34 schrieb Christopher Faulet: > The 1.6 branch is EOL now. Thus, it is the last 1.6 release. No further Don't forget to update the color of the table on haproxy.org :-) Same probably applies to 2.1, even if you don't rule out critical fixes for it. And I also believe you wanted to put 1.8 to "critical only" as well. Best regards Tim Düsterhus
[ANNOUNCE] haproxy-1.6.16
Hi, HAProxy 1.6.16 was released on 2021/03/19. It added 71 new commits after version 1.6.15. The 1.6 branch is EOL now. Thus, it is the last 1.6 release. No further release should be expected. It contains all pending patches marked to be backported to the 1.6 to leave it in a proper state. Have a look at the changelog below for the complete list of fixes. You should have no reason to deploy it in a production environment. Use the 1.8 or above instead. However, if possible directly move on the 2.0 or the 2.2. No specific support should no longer be expected on the 1.6. This branch will not receive fixes anymore. Thanks everyone for you help and your contributions ! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/1.6/src/ Git repository : http://git.haproxy.org/git/haproxy-1.6.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-1.6.git Changelog: http://www.haproxy.org/download/1.6/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Amaury Denoyelle (1): BUG/MINOR: config: Fix memory leak on config parse listen Baptiste Assmann (1): BUG/MINOR: http_act: don't check capture id in backend Christopher Faulet (24): BUG/MEDIUM: stream: Be sure to release allocated captures for TCP streams BUG/MINOR: http-rules: Remove buggy deinit functions for HTTP rules BUG/MINOR: stick-table: Use MAX_SESS_STKCTR as the max track ID during parsing MINOR: proxy/http-ana: Add support of extra attributes for the cookie directive BUG/MINOR: rules: Increment be_counters if backend is assigned for a silent-drop BUG/MEDIUM: lua: Reset analyse expiration timeout before executing a lua action BUG/MEDIUM: pattern: Add a trailing \0 to match strings only if possible BUG/MEDIUM: channel: Be aware of SHUTW_NOW flag when output data are peeked BUG/MINOR: tcp-rules: Set the inspect-delay when a tcp-response action yields BUG/MEDIUM: map/lua: Return an error if a map is loaded during runtime BUG/MINOR: lua: Check argument type to convert it to IPv4/IPv6 arg validation BUG/MINOR: lua: Check argument type to convert it to IP mask in arg validation BUG/MEDIUM: pattern: Renew the pattern expression revision when it is pruned MINOR: hlua: Display debug messages on stderr only in debug mode BUG/MINOR: http-fetch: Fix calls w/o parentheses of the cookie sample fetches BUG/MINOR: tools: make parse_time_err() more strict on the timer validity BUG/MINOR: tools: Reject size format not starting by a digit BUG/MINOR: server: Fix server-state-file-name directive CLEANUP: deinit: release global and per-proxy server-state variables on deinit BUG/MINOR: server: Don't call fopen() with server-state filepath set to NULL BUG/MINOR: sample: Always consider zero size string samples as unsafe BUG/MINOR: server: Init params before parsing a new server-state line BUG/MINOR: http-ana: Only consider dst address to process originalto option BUG/MINOR: connection: Use the client's dst family for adressless servers David Carlier (1): DOC: email change of the DeviceAtlas maintainer Dragan Dosen (1): BUG/MEDIUM: pattern: fix memory leak in regex pattern functions Emeric Brun (1): CLEANUP: channel: fix comment in ci_putblk. Emmanuel Hocdet (1): BUG/MINOR: ssl: fix crt-list neg filter for openssl < 1.1.1 Jerome Magnin (2): BUG/MINOR: stream: don't mistake match rules for store-request rules BUG/MINOR: pattern: handle errors from fgets when trying to load patterns Joao Morais (1): BUG/MINOR: config: Update cookie domain warn to RFC6265 Maciej Zdeb (1): BUG/MINOR: http-fetch: Extract cookie value even when no cookie name Mathias Weiersmueller (1): DOC: clarify matching strings on binary fetches Thierry Fournier (1): MINOR: common: mask conversion Tim Duesterhus (4): BUG/MINOR: dns: Make dns_query_id_seed unsigned BUG/MAJOR: proxy_protocol: Properly validate TLV lengths BUG/MEDIUM: fetch: Fix hdr_ip misparsing IPv4 addresses due to missing NUL BUG/MINOR: http_act: don't check capture id in backend (2) William Dauchy (3): BUG/MINOR: dns: allow 63 char in hostname BUG/MINOR: namespace: avoid closing fd when socket failed in my_socketat DOC: agent-check: fix typo in "fail" word expected reply William Lallemand (2): BUG/MINOR: ssl: verifyhost is case sensitive DOC: ssl: crt-list negative filters are only a hint Willy Tarreau (26): BUG/MINOR: listener: also clear the error flag on a paused listener
