Re: OpenSSL Security Advisory
Kevin, On 3/25/21 7:40 PM, Fox, Kevin M wrote: > That would be unfortunate. Some clusters won't be able to distinguish if > there is an update or not. If your toolchain is unable to correctly handle changing tags for Docker Images then it is completely broken. This is happening all the time with the 'latest' tag or the branch specific specific aliases (e.g. 'haproxy:2.2' tracking 2.2.x). If you want to ensure that you receive a specific tested version then you could either mirror the tested image onto your infrastructure with a tag of your choice or reference them by digest. Tianon from DOI Team (also put into Cc) directed me to this list of commits that tracks all changes to the 'haproxy' image: https://github.com/docker-library/repo-info/commits/master/repos/haproxy/remote/latest.md > That's one reason I typically follow the distro convention of packaging, of > tacking onto the tag a -1, so if I need to bump them, it can be. -2 when a > new release of the same version comes out. -3 next, etc. Could something like > that be adopted? Unless you are consuming the 'haproxytech/*' images your request is best directed here: https://github.com/docker-library/official-images/ However I don't expect this to be implemented. It seems to introduce much work for very little benefit and might prevent users that accidentally use these tags without understanding them from receiving security fixes. Tianon might or might not authoritatively comment on this :-) Best regards Tim Düsterhus PS: For the folks interested in the rebuild. The update of the 'debian' base image is happening here: https://github.com/docker-library/official-images/pull/9862
Re: OpenSSL Security Advisory
That would be unfortunate. Some clusters won't be able to distinguish if there is an update or not. That's one reason I typically follow the distro convention of packaging, of tacking onto the tag a -1, so if I need to bump them, it can be. -2 when a new release of the same version comes out. -3 next, etc. Could something like that be adopted? Thanks, Kevin From: Tim Düsterhus Sent: Thursday, March 25, 2021 11:32 AM To: Paul Lockaby Cc: Lukas Tribus; haproxy Subject: Re: OpenSSL Security Advisory Check twice before you click! This email originated from outside PNNL. Paul, On 3/25/21 7:31 PM, Paul Lockaby wrote: > Thanks for all of the responses! So the image version number for HAProxy > stays the same but the hash will update? > Yes exactly. Best regards Tim Düsterhus
Re: OpenSSL Security Advisory
Paul, On 3/25/21 7:31 PM, Paul Lockaby wrote: > Thanks for all of the responses! So the image version number for HAProxy > stays the same but the hash will update? > Yes exactly. Best regards Tim Düsterhus
Re: OpenSSL Security Advisory
Thanks for all of the responses! So the image version number for HAProxy stays the same but the hash will update? > On Mar 25, 2021, at 9:03 AM, Tim Düsterhus wrote: > > Paul, > > On 3/25/21 4:46 PM, Paul Lockaby wrote: >> As a lurker on this list I've always kind of wondered what the policy > is for releasing new containers to address security patches on > dependencies like this. I'm not sure who maintains the "official" > containers for HAProxy but would they do a re-release of the latest > versions to include a patch on a dependency like OpenSSL? >> > > The 'haproxy' image for Docker is maintained by the Docker Official > Images Team [1] [2]. They also handle the necessary rebuilds when the > base image changes. I maintain 2 images as part of the Official Images > program and also contribute to the HAProxy image via Pull Requests. I am > not part of the DOI Team, though. > > Independently from your email I already asked in their IRC whether the > 'debian' base image is going to be rebuilt due to the OpenSSL update. > This would then cause a rebuild of the 'haproxy' image. > > For the images that contain a username (e.g. timwolla/haproxy) the > authors are responsible to trigger a rebuild. > > Best regards > Tim Düsterhus > > [1] https://github.com/docker-library/haproxy/ > [2] https://github.com/docker-library/official-images/
Re: OpenSSL Security Advisory
[…] > On 25.03.2021., at 17:03, Tim Düsterhus wrote: > […] > > The 'haproxy' image for Docker is maintained by the Docker Official > Images Team [1] [2]. They also handle the necessary rebuilds when the > base image changes. I maintain 2 images as part of the Official Images > program and also contribute to the HAProxy image via Pull Requests. I am > not part of the DOI Team, though. > > Independently from your email I already asked in their IRC whether the > 'debian' base image is going to be rebuilt due to the OpenSSL update. > This would then cause a rebuild of the 'haproxy' image. > > For the images that contain a username (e.g. timwolla/haproxy) the > authors are responsible to trigger a rebuild. > Just to follow-up on this: As Tim has already kindly summarised, the same thing also applies for haproxytech (https://hub.docker.com/u/haproxytech) HAProxy CE images as well, they will get rebuilt on official base image (Debian, Ubuntu, Alpine, etc.) being rebuilt. This includes regular HAProxy CE images, Ingress Controller images, etc. Kind regards, D. -- Dinko Korunic ** Standard disclaimer applies ** Sent from OSF1 osf1v4b V4.0 564 alpha
is it possible to disable option httpchk per backend?
Hi, is it possible to disable "option httpchk" in specific backend when it is enabled in defaults block? I have config where basically every backend sans one is http so I'd like to keep that in defaults and just disable it in tcp backend (which is backend for SPOE/A) but it seems to be one of very few options that do not have "no option httpchk". Cheers Mariusz -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: ad...@efigence.com
Re: OpenSSL Security Advisory
Paul, On 3/25/21 4:46 PM, Paul Lockaby wrote: > As a lurker on this list I've always kind of wondered what the policy is for releasing new containers to address security patches on dependencies like this. I'm not sure who maintains the "official" containers for HAProxy but would they do a re-release of the latest versions to include a patch on a dependency like OpenSSL? > The 'haproxy' image for Docker is maintained by the Docker Official Images Team [1] [2]. They also handle the necessary rebuilds when the base image changes. I maintain 2 images as part of the Official Images program and also contribute to the HAProxy image via Pull Requests. I am not part of the DOI Team, though. Independently from your email I already asked in their IRC whether the 'debian' base image is going to be rebuilt due to the OpenSSL update. This would then cause a rebuild of the 'haproxy' image. For the images that contain a username (e.g. timwolla/haproxy) the authors are responsible to trigger a rebuild. Best regards Tim Düsterhus [1] https://github.com/docker-library/haproxy/ [2] https://github.com/docker-library/official-images/
[ANNOUNCE] haproxy-2.3.8
Hi, HAProxy 2.3.8 was released on 2021/03/25. It added 22 new commits after version 2.3.7. This version focuses on fixing a few annoying bugs that were reported on 2.3.7 and older, some of which were recent regressions. - Lua: Maciej hit a situation where the watchdog occasionally triggers while one of his threads is in the Lua's memory allocator. While this is particuarly strange and needs to be investigated, it had the bad taste of trying to dump a Lua backtrace which itself required a memory allocation, and used to deadlock inside the libc's allocator! Christopher fixed this by using a different Lua trace function that does not require any memory allocation. - a fix for a locking bug in the FD migration code unveiled an AB/BA situation with the listeners, that is more visible in 2.2: when reloading processes under a high connection rate traffic, it was possible for one thread to try to stop the listeners while another one had the listener lock, and both would wait for each other, until the watchdog reconciliated them the hard way. As this would require very high loads and would only affect the old process, I guess most users didn't notice, but Christopher triggered it on a reg-test and since then we could only see it. In 2.3 and 2.4 it will be harder to trigger it but it's definitely possible while resuming a temporarily paused listener. This was fixed with Olivier's help by deferring the close to the last user of the FD. - when I fixed the freq_ctr issue (multiple rotation and occasional livelock), I addressed both second-based counters and period-based ones. However my fix on the latter only worked for sub-second periods which were the ones I used in my tests :-/ The visible effect is that some rate counters in stick-tables averaged over more than one second will indefinitely grow. That's now fixed and I could verify that any delay now works. And I still have no idea how to reliably test rate counters automatically. On the other hand if we break them once every 10 years, I'm wondering if it's really worth worrying about them :-/ - during the backport to 2.3 of the patch to relieve the contention on the idle_conns_lock had significant contextual changes, and I apparently misplaced an unlock there that Christopher fixed. - when using DEBUG_UAF (to detect use-after-free), we use mmap/munmap to allocate/free memory. Since this can take a lot of time, the thread entering releases there marks itself harmless around the calls. But Christopher found one case where if that's done under thread_isolate() as happens during pool_gc(), the return back to normal operation can deadlock with the thread itself. - we were contacted by BitMEX security team and @melardev who found that haproxy would accept garbage after an IP address that was improperly formatted by their first level proxy resulting in an issue not being detected for a while. For example if the first layer incorrectly uses semi-colons instead of colons to delimit addresses, the real field would read "1.2.3.4; 5.6.7.8" and haproxy would return 1.2.3.4. The reason is that the parser was also used to parse delimited addresses in URLs. So now we comply more strictly to the format in RFC7239, but still accept a colon and a port because at least Azure's application gateway is documented for appending this, but we reject everything else. This will be easier to detect than an apparently valid address that does not correspond to what the first level proxy meant. - the log-forward section would not support forwarding from UNIX dgram sockets just because the protocol definition was missing! Emeric added it. - Rémi addressed an issue affecting the "add ssl crt-list" command on the CLI: if both a ctr-list and a ca-file were fed from the CLI, the openssl functions used to build them could perform FS accesses. That's nasty because in the unlucky case that one entry matches one present during, say, a config check or something like this, it would not necessarily work later after a chroot or a drop of privileges. Now these functions were adjusted to only use the in-memory API and to return errors directly on the CLI. - Ilya secured another case of OpenSSL version comparison that's more reliably handled by testing a macro. - Eric's previously mentionned fix for the unix-bind-prefix that used to mistakenly affect the master socket location was integrated. I would appreciate it if those who experienced issues with latest 2.2 could give this one a try on a single server and confirm it addresses their concerns as well, so that we're more confident in what goes into 2.2.12, probably early next week to still give some time for last issues to be reported. For all versions there's still an issue with ssl_sock_shutw() crashing under certain circumstances but it was already affecting 2.0.13 a year
Re: OpenSSL Security Advisory
Hello! As a lurker on this list I've always kind of wondered what the policy is for releasing new containers to address security patches on dependencies like this. I'm not sure who maintains the "official" containers for HAProxy but would they do a re-release of the latest versions to include a patch on a dependency like OpenSSL? Thanks, -Paul > On Mar 25, 2021, at 7:28 AM, Lukas Tribus wrote: > > FYI > > -- Forwarded message - > From: OpenSSL > Date: Thu, 25 Mar 2021 at 15:03 > Subject: OpenSSL Security Advisory > To: , OpenSSL User Support ML > , OpenSSL Announce ML > > > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > OpenSSL Security Advisory [25 March 2021] > = > > CA certificate check bypass with X509_V_FLAG_X509_STRICT (CVE-2021-3450) > > > Severity: High > > The X509_V_FLAG_X509_STRICT flag enables additional security checks of the > certificates present in a certificate chain. It is not set by default. > > Starting from OpenSSL version 1.1.1h a check to disallow certificates in > the chain that have explicitly encoded elliptic curve parameters was added > as an additional strict check. > > An error in the implementation of this check meant that the result of a > previous check to confirm that certificates in the chain are valid CA > certificates was overwritten. This effectively bypasses the check > that non-CA certificates must not be able to issue other certificates. > > If a "purpose" has been configured then there is a subsequent opportunity > for checks that the certificate is a valid CA. All of the named "purpose" > values implemented in libcrypto perform this check. Therefore, where > a purpose is set the certificate chain will still be rejected even when the > strict flag has been used. A purpose is set by default in libssl client and > server certificate verification routines, but it can be overridden or > removed by an application. > > In order to be affected, an application must explicitly set the > X509_V_FLAG_X509_STRICT verification flag and either not set a purpose > for the certificate verification or, in the case of TLS client or server > applications, override the default purpose. > > OpenSSL versions 1.1.1h and newer are affected by this issue. Users of these > versions should upgrade to OpenSSL 1.1.1k. > > OpenSSL 1.0.2 is not impacted by this issue. > > This issue was reported to OpenSSL on 18th March 2021 by Benjamin Kaduk > from Akamai and was discovered by Xiang Ding and others at Akamai. The fix was > developed by Tomáš Mráz. > > > NULL pointer deref in signature_algorithms processing (CVE-2021-3449) > = > > Severity: High > > An OpenSSL TLS server may crash if sent a maliciously crafted renegotiation > ClientHello message from a client. If a TLSv1.2 renegotiation ClientHello > omits > the signature_algorithms extension (where it was present in the initial > ClientHello), but includes a signature_algorithms_cert extension then a NULL > pointer dereference will result, leading to a crash and a denial of service > attack. > > A server is only vulnerable if it has TLSv1.2 and renegotiation enabled (which > is the default configuration). OpenSSL TLS clients are not impacted by this > issue. > > All OpenSSL 1.1.1 versions are affected by this issue. Users of these versions > should upgrade to OpenSSL 1.1.1k. > > OpenSSL 1.0.2 is not impacted by this issue. > > This issue was reported to OpenSSL on 17th March 2021 by Nokia. The fix was > developed by Peter Kästle and Samuel Sapalski from Nokia. > > Note > > > OpenSSL 1.0.2 is out of support and no longer receiving public updates. > Extended > support is available for premium support customers: > https://www.openssl.org/support/contracts.html > > OpenSSL 1.1.0 is out of support and no longer receiving updates of any kind. > The impact of these issues on OpenSSL 1.1.0 has not been analysed. > > Users of these versions should upgrade to OpenSSL 1.1.1. > > References > == > > URL for this Security Advisory: > https://www.openssl.org/news/secadv/20210325.txt > > Note: the online version of the advisory may be updated with additional > details > over time. > > For details of OpenSSL severity classifications please see: > https://www.openssl.org/policies/secpolicy.html > -BEGIN PGP SIGNATURE- > > iQEzBAEBCAAdFiEEhlersmDwVrHlGQg52cTSbQ5gRJEFAmBcl6sACgkQ2cTSbQ5g > RJGvnAgAtG6I7rfokDC9E5yB26KC3k0Vasfq5iH/aZz0CNRyOokWJBUyyNIVjqr0 > 2eZP7VsQT7zRM+tgh9c8MwH3FIghtpwJRJls4qZDHKoXts7JH4Ul4NLPd546x7xA > GcKNwTD4NkZbTqtZ72NTgliInzrj0MCC8jqQrIIkcAIleGNzvZ0f64jdE+vBXoqX > M2FOhWiA/JkAKtB3W7pthIt25qkOwHbrpTy+UUp/S5QD779NJ/EOYcsOFBRfLZiP > gA6QILuW2L55lhG6Y2u+nVE3UI2hqd2hGgSAvDIPr2lVJxq0LQpgHca7Gj5bfIRo > GLDz7n0FhN6n7NBqetP+nlHmYivcSg== > =XIXK > -END PGP SIGNATURE- >
Fwd: OpenSSL Security Advisory
FYI -- Forwarded message - From: OpenSSL Date: Thu, 25 Mar 2021 at 15:03 Subject: OpenSSL Security Advisory To: , OpenSSL User Support ML , OpenSSL Announce ML -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 OpenSSL Security Advisory [25 March 2021] = CA certificate check bypass with X509_V_FLAG_X509_STRICT (CVE-2021-3450) Severity: High The X509_V_FLAG_X509_STRICT flag enables additional security checks of the certificates present in a certificate chain. It is not set by default. Starting from OpenSSL version 1.1.1h a check to disallow certificates in the chain that have explicitly encoded elliptic curve parameters was added as an additional strict check. An error in the implementation of this check meant that the result of a previous check to confirm that certificates in the chain are valid CA certificates was overwritten. This effectively bypasses the check that non-CA certificates must not be able to issue other certificates. If a "purpose" has been configured then there is a subsequent opportunity for checks that the certificate is a valid CA. All of the named "purpose" values implemented in libcrypto perform this check. Therefore, where a purpose is set the certificate chain will still be rejected even when the strict flag has been used. A purpose is set by default in libssl client and server certificate verification routines, but it can be overridden or removed by an application. In order to be affected, an application must explicitly set the X509_V_FLAG_X509_STRICT verification flag and either not set a purpose for the certificate verification or, in the case of TLS client or server applications, override the default purpose. OpenSSL versions 1.1.1h and newer are affected by this issue. Users of these versions should upgrade to OpenSSL 1.1.1k. OpenSSL 1.0.2 is not impacted by this issue. This issue was reported to OpenSSL on 18th March 2021 by Benjamin Kaduk from Akamai and was discovered by Xiang Ding and others at Akamai. The fix was developed by Tomáš Mráz. NULL pointer deref in signature_algorithms processing (CVE-2021-3449) = Severity: High An OpenSSL TLS server may crash if sent a maliciously crafted renegotiation ClientHello message from a client. If a TLSv1.2 renegotiation ClientHello omits the signature_algorithms extension (where it was present in the initial ClientHello), but includes a signature_algorithms_cert extension then a NULL pointer dereference will result, leading to a crash and a denial of service attack. A server is only vulnerable if it has TLSv1.2 and renegotiation enabled (which is the default configuration). OpenSSL TLS clients are not impacted by this issue. All OpenSSL 1.1.1 versions are affected by this issue. Users of these versions should upgrade to OpenSSL 1.1.1k. OpenSSL 1.0.2 is not impacted by this issue. This issue was reported to OpenSSL on 17th March 2021 by Nokia. The fix was developed by Peter Kästle and Samuel Sapalski from Nokia. Note OpenSSL 1.0.2 is out of support and no longer receiving public updates. Extended support is available for premium support customers: https://www.openssl.org/support/contracts.html OpenSSL 1.1.0 is out of support and no longer receiving updates of any kind. The impact of these issues on OpenSSL 1.1.0 has not been analysed. Users of these versions should upgrade to OpenSSL 1.1.1. References == URL for this Security Advisory: https://www.openssl.org/news/secadv/20210325.txt Note: the online version of the advisory may be updated with additional details over time. For details of OpenSSL severity classifications please see: https://www.openssl.org/policies/secpolicy.html -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEEhlersmDwVrHlGQg52cTSbQ5gRJEFAmBcl6sACgkQ2cTSbQ5g RJGvnAgAtG6I7rfokDC9E5yB26KC3k0Vasfq5iH/aZz0CNRyOokWJBUyyNIVjqr0 2eZP7VsQT7zRM+tgh9c8MwH3FIghtpwJRJls4qZDHKoXts7JH4Ul4NLPd546x7xA GcKNwTD4NkZbTqtZ72NTgliInzrj0MCC8jqQrIIkcAIleGNzvZ0f64jdE+vBXoqX M2FOhWiA/JkAKtB3W7pthIt25qkOwHbrpTy+UUp/S5QD779NJ/EOYcsOFBRfLZiP gA6QILuW2L55lhG6Y2u+nVE3UI2hqd2hGgSAvDIPr2lVJxq0LQpgHca7Gj5bfIRo GLDz7n0FhN6n7NBqetP+nlHmYivcSg== =XIXK -END PGP SIGNATURE-
Re: [2.2.9] 100% CPU usage
Le 25/03/2021 à 13:38, Maciej Zdeb a écrit : Hi, I deployed a patched (with volatile hlua_not_dumpable) HAProxy and so far so good, no looping. Christopher I saw new patches with hlua_traceback used instead, looks much cleaner to me, should I verify them instead? :) Christopher & Willy I've forgotten to thank you for help! Yes please, try the last 2.2 snapshot. It is a really a better way to fix this issue because the Lua traceback is never ignored. And it is really safer to not allocate memory in the debugger. So now, we should be able to figure out why the Lua fires the watchdog. Because, under the hood, it is the true issue :) -- Christopher Faulet
Re: [2.2.9] 100% CPU usage
Hi, I deployed a patched (with volatile hlua_not_dumpable) HAProxy and so far so good, no looping. Christopher I saw new patches with hlua_traceback used instead, looks much cleaner to me, should I verify them instead? :) Christopher & Willy I've forgotten to thank you for help! Kind regards, śr., 24 mar 2021 o 10:51 Maciej Zdeb napisał(a): > śr., 24 mar 2021 o 10:37 Christopher Faulet > napisał(a): > >> However, reading the other trace Maciej sent (bussy_thread_peers.txt), it >> seems >> possible to stop a memory allocation from other places. Thus, I guess we >> must >> find a more global way to prevent the lua stack dump. >> > > I'm not sure which part of trace you're referring to but I need to clarify > that both "bussy_thread_peers.txt" and "free_thread_spoe_lua.txt" occurred > at the same time but on different threads (free_thread_spoe_lua on thread > 10 and bussy_thread_peers on other threads). So If I understand it > correctly thread 10 locked itself and other threads looping to acquire lock. >
