Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Thu, 2019-02-07 at 22:55 +0100, Jean-Marc wrote: > On Mon, 26 Nov 2018 23:41:13 +0100 Sebastian Andrzej Siewior a...@breakpoint.cc> wrote: > > On 2018-11-04 22:15:04 [+0100], Kurt Roeckx wrote: > > > > You're implying openvpn doesn't pick up the openssl.cnf changes > > > > so I have to set tls-version-min 1.0 in the server side > > > > configuration? OK, that works too. > > > > > > Your client doesn't support the settings in the openssl.cfg file. > > > Your openvpn client by defaults does TLS 1.0 only. The only way > > > for your client to do something other than TLS 1.0 is set the > > > tls-version-min variable to something. If you set it to 1.0, it > > > will do any version supported by the openssl library higher than > > > 1.0. > > > > James, is everything okay/clear? The tls-version-min option for the > > older OpenVPN version should have fixed things. Is there anything > > else or can this be considered done? > > > > > Kurt > > > > Sebastian > > Hi James, > > May I ask you if you got all the answers you needed and if it fixed > the problem. Yes, I said that in the initial quote: setting tls-version-min in openssl.cnf works, and that's what I've done. It's just unexpected that you have to update your openvpn config files. James signature.asc Description: This is a digitally signed message part
Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Mon, 26 Nov 2018 23:41:13 +0100 Sebastian Andrzej Siewior wrote: > On 2018-11-04 22:15:04 [+0100], Kurt Roeckx wrote: > > > You're implying openvpn doesn't pick up the openssl.cnf changes so I > > > have to set tls-version-min 1.0 in the server side configuration? OK, > > > that works too. > > > > Your client doesn't support the settings in the openssl.cfg file. Your > > openvpn client by defaults does TLS 1.0 only. The only way for your client > > to do something other than TLS 1.0 is set the tls-version-min variable > > to something. If you set it to 1.0, it will do any version > > supported by the openssl library higher than 1.0. > > James, is everything okay/clear? > The tls-version-min option for the older OpenVPN version should have > fixed things. > Is there anything else or can this be considered done? > > > Kurt > > Sebastian Hi James, May I ask you if you got all the answers you needed and if it fixed the problem. Thank you so much. Regards, Jean-Marc https://6jf.be/keys/ED863AD1.txt pgpbKgFaTfbY4.pgp Description: PGP signature
Bug#912864: openssl: new version of openssl breaks some openvpn clients
On 2018-11-04 22:15:04 [+0100], Kurt Roeckx wrote: > > You're implying openvpn doesn't pick up the openssl.cnf changes so I > > have to set tls-version-min 1.0 in the server side configuration? OK, > > that works too. > > Your client doesn't support the settings in the openssl.cfg file. Your > openvpn client by defaults does TLS 1.0 only. The only way for your client > to do something other than TLS 1.0 is set the tls-version-min variable > to something. If you set it to 1.0, it will do any version > supported by the openssl library higher than 1.0. James, is everything okay/clear? The tls-version-min option for the older OpenVPN version should have fixed things. Is there anything else or can this be considered done? > Kurt Sebastian
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, Nov 04, 2018 at 12:49:48PM -0800, James Bottomley wrote: > On Sun, 2018-11-04 at 21:30 +0100, Kurt Roeckx wrote: > > On Sun, Nov 04, 2018 at 12:13:43PM -0800, James Bottomley wrote: > > > > > > No, I'm saying with no client tls-version-min specified at all (the > > > usual default openvpn config) it fails in 1.1.1 and works with > > > 1.1.0 > > > > > > With client tls-version-min set to 1.0 it works with both. > > > > Yes, and that's totally what I expected, and have been explaining. > > The 2.3.X version only want to do TLS 1.0 unless you specify > > "tls-version-min 1.0", in which case they also do TLS 1.2. > > You're implying openvpn doesn't pick up the openssl.cnf changes so I > have to set tls-version-min 1.0 in the server side configuration? OK, > that works too. Your client doesn't support the settings in the openssl.cfg file. Your openvpn client by defaults does TLS 1.0 only. The only way for your client to do something other than TLS 1.0 is set the tls-version-min variable to something. If you set it to 1.0, it will do any version supported by the openssl library higher than 1.0. > > So I'm failing to see what this bug report is about. > > When you upgrade from openssl 1.1.0 to 1.1.1 causes an openvpn > connection failure which the upgrade instructions don't fix. It also > seems to me there are probably quite a few other openssl.cnf blind > applications in the system which will fail in a similar fashion. This is on the server side. As far as I know, changing the openssl.cnf file should just work. openvpn in testing takes the minimum of the openssl.cfg value and TLS 1.0. So if you set None, it should set TLS 1.0 as minimum. I assume you don't set a minimum tls version in your openvpn config file or on the command line. Kurt
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, 2018-11-04 at 21:30 +0100, Kurt Roeckx wrote: > On Sun, Nov 04, 2018 at 12:13:43PM -0800, James Bottomley wrote: > > > > No, I'm saying with no client tls-version-min specified at all (the > > usual default openvpn config) it fails in 1.1.1 and works with > > 1.1.0 > > > > With client tls-version-min set to 1.0 it works with both. > > Yes, and that's totally what I expected, and have been explaining. > The 2.3.X version only want to do TLS 1.0 unless you specify > "tls-version-min 1.0", in which case they also do TLS 1.2. You're implying openvpn doesn't pick up the openssl.cnf changes so I have to set tls-version-min 1.0 in the server side configuration? OK, that works too. > So I'm failing to see what this bug report is about. When you upgrade from openssl 1.1.0 to 1.1.1 causes an openvpn connection failure which the upgrade instructions don't fix. It also seems to me there are probably quite a few other openssl.cnf blind applications in the system which will fail in a similar fashion. James
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, Nov 04, 2018 at 12:13:43PM -0800, James Bottomley wrote: > > No, I'm saying with no client tls-version-min specified at all (the > usual default openvpn config) it fails in 1.1.1 and works with 1.1.0 > > With client tls-version-min set to 1.0 it works with both. Yes, and that's totally what I expected, and have been explaining. The 2.3.X version only want to do TLS 1.0 unless you specify "tls-version-min 1.0", in which case they also do TLS 1.2. So I'm failing to see what this bug report is about. Kurt
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, 2018-11-04 at 21:10 +0100, Kurt Roeckx wrote: > On Sun, Nov 04, 2018 at 11:39:59AM -0800, James Bottomley wrote: > > > > > > On which side do you use tls-version-min? > > > > client > > > > > Can you please give the version of both openvpn and openssl on > > > both > > > sides. > > > > Client is openwrt, server is debian testing. The package of the > > server > > was already provided in the bug report, but again it's > > > > openssl 1.1.1-2 > > openvpn 2.4.6-1 > > > > Packages on the openwrt client are > > > > libopenssl 1.0.2g-1 > > openvpn-openssl 2.3.6-5 > > So you're saying that even with tls-version-min 1.0 on your > client side and with openssl.cnf changed on the server it's still > not working? No, I'm saying with no client tls-version-min specified at all (the usual default openvpn config) it fails in 1.1.1 and works with 1.1.0 With client tls-version-min set to 1.0 it works with both. James
Bug#912864: [Pkg-openssl-devel] Bug#912864: Bug#912864: openssl: new version of openssl breaks some openvpn clients
On 2018-11-04 11:39:59 [-0800], James Bottomley wrote: > > > OK, so I'm weary of trying to construct a theory of what the bug > > > actually is, why don't you try to come up with one. The symptoms > > > are > > > that openvpn in openwrt works with server 1.1.0 and fails with > > > server > > > 1.1.1 if you don't specify tls-version-min 1.0 on the command line. > > > > On which side do you use tls-version-min? > > client > > > Can you please give the version of both openvpn and openssl on both > > sides. > > Client is openwrt, server is debian testing. The package of the server > was already provided in the bug report, but again it's > > openssl 1.1.1-2 > openvpn 2.4.6-1 > > Packages on the openwrt client are > > libopenssl 1.0.2g-1 > openvpn-openssl 2.3.6-5 That matches what I see. The Jessie version (which matches your openwrt client) does TLSv1.0 only by default. If you specify tls-version-min (even 1.0) then it tries 1.0+ and does the best possible protocol which is TLS1.2 in my case. Stretch seems to do the best possible version by default. Buster/Testing has TLS1.0 disabled by default. So in your environment the client tries TLS1.0 only and server does TLS1.2 only. Adding tls-version-min on the client side enables TLS1.0+ and handshakes with TLS1.2. This behaviour has been reported as #912650 on the Debian side. > James Sebastian
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, Nov 04, 2018 at 11:39:59AM -0800, James Bottomley wrote: > > > > On which side do you use tls-version-min? > > client > > > Can you please give the version of both openvpn and openssl on both > > sides. > > Client is openwrt, server is debian testing. The package of the server > was already provided in the bug report, but again it's > > openssl 1.1.1-2 > openvpn 2.4.6-1 > > Packages on the openwrt client are > > libopenssl 1.0.2g-1 > openvpn-openssl 2.3.6-5 So you're saying that even with tls-version-min 1.0 on your client side and with openssl.cnf changed on the server it's still not working? Either of those changes should be enough to get it working as far as I understand. I have almost the reverse in my setup, where the server is 2.3.4 and the client runs testing. On the server I've set the tls-version-min 1.0 and everything works for me. I will try to look at this in a few days. Kurt
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, 2018-11-04 at 20:32 +0100, Kurt Roeckx wrote: > On Sun, Nov 04, 2018 at 11:19:41AM -0800, James Bottomley wrote: > > On Sun, 2018-11-04 at 20:15 +0100, Kurt Roeckx wrote: > > > This is not at all how the version negiotation in TLS 1.2 and > > > below works. The client just indicates the highest version it > > > supports, so for instance TLS 1.2. It's then up to the server to > > > pick a version that the client supports, so one that is smaller > > > than > > > TLS 1.2, and it might pick TLS 1.0 or 1.2. It will then send a > > > server > > > hello with that version. > > > > OK, so I'm weary of trying to construct a theory of what the bug > > actually is, why don't you try to come up with one. The symptoms > > are > > that openvpn in openwrt works with server 1.1.0 and fails with > > server > > 1.1.1 if you don't specify tls-version-min 1.0 on the command line. > > On which side do you use tls-version-min? client > Can you please give the version of both openvpn and openssl on both > sides. Client is openwrt, server is debian testing. The package of the server was already provided in the bug report, but again it's openssl 1.1.1-2 openvpn 2.4.6-1 Packages on the openwrt client are libopenssl 1.0.2g-1 openvpn-openssl 2.3.6-5 James
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, Nov 04, 2018 at 11:19:41AM -0800, James Bottomley wrote: > On Sun, 2018-11-04 at 20:15 +0100, Kurt Roeckx wrote: > > This is not at all how the version negiotation in TLS 1.2 and > > below works. The client just indicates the highest version it > > supports, so for instance TLS 1.2. It's then up to the server to > > pick a version that the client supports, so one that is smaller than > > TLS 1.2, and it might pick TLS 1.0 or 1.2. It will then send a server > > hello with that version. > > OK, so I'm weary of trying to construct a theory of what the bug > actually is, why don't you try to come up with one. The symptoms are > that openvpn in openwrt works with server 1.1.0 and fails with server > 1.1.1 if you don't specify tls-version-min 1.0 on the command line. On which side do you use tls-version-min? Can you please give the version of both openvpn and openssl on both sides. Kurt
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, 2018-11-04 at 20:15 +0100, Kurt Roeckx wrote: > This is not at all how the version negiotation in TLS 1.2 and > below works. The client just indicates the highest version it > supports, so for instance TLS 1.2. It's then up to the server to > pick a version that the client supports, so one that is smaller than > TLS 1.2, and it might pick TLS 1.0 or 1.2. It will then send a server > hello with that version. OK, so I'm weary of trying to construct a theory of what the bug actually is, why don't you try to come up with one. The symptoms are that openvpn in openwrt works with server 1.1.0 and fails with server 1.1.1 if you don't specify tls-version-min 1.0 on the command line. > So there are normally 2 cases that can be a problem: > - The client sends TLS 1.0 and the server has 1.2 as minimum, so > the server say it's not supported. > - The client sends TLS 1.2, the server answers with 1.0, the > client says 1.0 is too low. > > The error message you showed says that it's the server that is > rejecting the client's version, and that the server is running a > 1.1.1 version. Are you sure you've actually restarted the server > after changing the config file? Yes, the server got rebooted after the upgrade. James
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, Nov 04, 2018 at 10:19:00AM -0800, James Bottomley wrote: > On Sun, 2018-11-04 at 18:43 +0100, Kurt Roeckx wrote: > > Older versions of openvpn only support TLS 1.0 because they told > > OpenSSL to only use TLS 1.0. Adding the --tls-version-min 1.0 > > should make it support all TLS versions since openvpn 2.3.4 or > > something like that, and I think 2.4 or newer should just work. > > There's a difference: if you don't specify the command line tls- > version-min, it actually asks openssl for the minimum version. If you > do specify, it takes what you tell it. There is no API in OpenSSL to ask the minimum supported version. What 2.3.4 does is that if you don't specify anything, it tells OpenSSL to use TLS 1.0 only. What 2.4 does it just tell OpenSSL to use any version it supports. You can also specify that minimum version in the config file. > > But if you changed the openssl.cfg to say all versions are > > supported, it should work too, I'm not sure why you say otherwise. > > Well, obviously because it doesn't work as the log attached in the bug > report shows. > > The values I have in openssl.cnf are the recommended > > MinProtocol = None > CipherString = DEFAULT > > And it definitely works because imap has an android client at 0.9.8 > which didn't work before the addition of that. > > The openssl code looks to use SSL_CTX_get_min_proto_version() if you > don't specify a version, so it finds a protocol below tls 1.0 to > present which causes the error. From the ordering in openssl, this is > likely to be SSLv3, isn't it? With the above config SSL_CTX_get_min_proto_version() will return 0 indicating that all version supported at compile time are supported. The minium at compile time is TLS 1.0. > The bug here is that you shouldn't kill the negotiation just because > the client offers to support SSLv3, you should move on to negotiate a > more secure cipher and only error out if the client can't support any > protocols openssl is told to consider secure. This is not at all how the version negiotation in TLS 1.2 and below works. The client just indicates the highest version it supports, so for instance TLS 1.2. It's then up to the server to pick a version that the client supports, so one that is smaller than TLS 1.2, and it might pick TLS 1.0 or 1.2. It will then send a server hello with that version. So there are normally 2 cases that can be a problem: - The client sends TLS 1.0 and the server has 1.2 as minimum, so the server say it's not supported. - The client sends TLS 1.2, the server answers with 1.0, the client says 1.0 is too low. The error message you showed says that it's the server that is rejecting the client's version, and that the server is running a 1.1.1 version. Are you sure you've actually restarted the server after changing the config file? Kurt
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, 2018-11-04 at 18:43 +0100, Kurt Roeckx wrote: > Older versions of openvpn only support TLS 1.0 because they told > OpenSSL to only use TLS 1.0. Adding the --tls-version-min 1.0 > should make it support all TLS versions since openvpn 2.3.4 or > something like that, and I think 2.4 or newer should just work. There's a difference: if you don't specify the command line tls- version-min, it actually asks openssl for the minimum version. If you do specify, it takes what you tell it. > But if you changed the openssl.cfg to say all versions are > supported, it should work too, I'm not sure why you say otherwise. Well, obviously because it doesn't work as the log attached in the bug report shows. The values I have in openssl.cnf are the recommended MinProtocol = None CipherString = DEFAULT And it definitely works because imap has an android client at 0.9.8 which didn't work before the addition of that. The openssl code looks to use SSL_CTX_get_min_proto_version() if you don't specify a version, so it finds a protocol below tls 1.0 to present which causes the error. From the ordering in openssl, this is likely to be SSLv3, isn't it? The bug here is that you shouldn't kill the negotiation just because the client offers to support SSLv3, you should move on to negotiate a more secure cipher and only error out if the client can't support any protocols openssl is told to consider secure. James
Bug#912864: [Pkg-openssl-devel] Bug#912864: openssl: new version of openssl breaks some openvpn clients
On Sun, Nov 04, 2018 at 08:59:05AM -0800, James Bottomley wrote: > Package: openssl > Version: 1.1.1-2 > Severity: important > > I've applied all the downgrades recommended to the openssl.cnf file > and most services are now working again with the exception of openvpn. > > The only failure seems to be a VPN connection to an openwrt router. > The router is running Chaos Calmer 15.05 and has an upgraded openssl > (to 1.0.2g-1). > > The error is on the debian server side and only shows up at openvpn > extreme verbosity: > > Sun Nov 4 08:40:04 2018 us=149552 50.35.68.20:56038 OpenSSL: > error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported > protocol > > The full verbose negotiation is > > Sun Nov 4 08:40:04 2018 us=116122 50.35.68.20:56038 Control Channel MTU > parms [ L:1621 D:1212 EF:38 EB:0 ET:0 EL:3 ] > Sun Nov 4 08:40:04 2018 us=116160 50.35.68.20:56038 Data Channel MTU parms [ > L:1621 D:1450 EF:121 EB:406 ET:0 EL:3 ] > Sun Nov 4 08:40:04 2018 us=116243 50.35.68.20:56038 Local Options String > (VER=V4): 'V4,dev-type tun,link-mtu 1557,tun-mtu 1500,proto UDPv4,cipher > AES-128-CBC,auth SHA1,keysize 128,key-method 2,tls-server' > Sun Nov 4 08:40:04 2018 us=116263 50.35.68.20:56038 Expected Remote Options > String (VER=V4): 'V4,dev-type tun,link-mtu 1557,tun-mtu 1500,proto > UDPv4,cipher AES-128-CBC,auth SHA1,keysize 128,key-method 2,tls-client' > RSun Nov 4 08:40:04 2018 us=116336 50.35.68.20:56038 TLS: Initial packet > from [AF_INET]50.35.68.20:56038, sid=812b650a 26613bfb > WRRWRSun Nov 4 08:40:04 2018 us=149552 50.35.68.20:56038 OpenSSL: > error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported > protocol > Sun Nov 4 08:40:04 2018 us=150331 50.35.68.20:56038 TLS_ERROR: BIO read > tls_read_plaintext error > Sun Nov 4 08:40:04 2018 us=150984 50.35.68.20:56038 TLS Error: TLS object -> > incoming plaintext read error > Sun Nov 4 08:40:04 2018 us=151598 50.35.68.20:56038 TLS Error: TLS handshake > failed > Sun Nov 4 08:40:04 2018 us=152357 50.35.68.20:56038 SIGUSR1[soft,tls-error] > received, client-instance restarting > > Obviously this was all working with 1.1.0 so something seems to have > changed in the tls negotiation routines. > > I can fix this in the client by adding the openssl command > --tls-version-min 1.0 so it probably means, the way openvpn works that > the openssl version installed in openwrt has some strange TLS version > < 1.0 but clearly openssl shouldn't error out when presented with > lower ciphers it should negotiate the mutually acceptable version. Older versions of openvpn only support TLS 1.0 because they told OpenSSL to only use TLS 1.0. Adding the --tls-version-min 1.0 should make it support all TLS versions since openvpn 2.3.4 or something like that, and I think 2.4 or newer should just work. But if you changed the openssl.cfg to say all versions are supported, it should work too, I'm not sure why you say otherwise. Kurt
Bug#912864: openssl: new version of openssl breaks some openvpn clients
Package: openssl Version: 1.1.1-2 Severity: important I've applied all the downgrades recommended to the openssl.cnf file and most services are now working again with the exception of openvpn. The only failure seems to be a VPN connection to an openwrt router. The router is running Chaos Calmer 15.05 and has an upgraded openssl (to 1.0.2g-1). The error is on the debian server side and only shows up at openvpn extreme verbosity: Sun Nov 4 08:40:04 2018 us=149552 50.35.68.20:56038 OpenSSL: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol The full verbose negotiation is Sun Nov 4 08:40:04 2018 us=116122 50.35.68.20:56038 Control Channel MTU parms [ L:1621 D:1212 EF:38 EB:0 ET:0 EL:3 ] Sun Nov 4 08:40:04 2018 us=116160 50.35.68.20:56038 Data Channel MTU parms [ L:1621 D:1450 EF:121 EB:406 ET:0 EL:3 ] Sun Nov 4 08:40:04 2018 us=116243 50.35.68.20:56038 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1557,tun-mtu 1500,proto UDPv4,cipher AES-128-CBC,auth SHA1,keysize 128,key-method 2,tls-server' Sun Nov 4 08:40:04 2018 us=116263 50.35.68.20:56038 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1557,tun-mtu 1500,proto UDPv4,cipher AES-128-CBC,auth SHA1,keysize 128,key-method 2,tls-client' RSun Nov 4 08:40:04 2018 us=116336 50.35.68.20:56038 TLS: Initial packet from [AF_INET]50.35.68.20:56038, sid=812b650a 26613bfb WRRWRSun Nov 4 08:40:04 2018 us=149552 50.35.68.20:56038 OpenSSL: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol Sun Nov 4 08:40:04 2018 us=150331 50.35.68.20:56038 TLS_ERROR: BIO read tls_read_plaintext error Sun Nov 4 08:40:04 2018 us=150984 50.35.68.20:56038 TLS Error: TLS object -> incoming plaintext read error Sun Nov 4 08:40:04 2018 us=151598 50.35.68.20:56038 TLS Error: TLS handshake failed Sun Nov 4 08:40:04 2018 us=152357 50.35.68.20:56038 SIGUSR1[soft,tls-error] received, client-instance restarting Obviously this was all working with 1.1.0 so something seems to have changed in the tls negotiation routines. I can fix this in the client by adding the openssl command --tls-version-min 1.0 so it probably means, the way openvpn works that the openssl version installed in openwrt has some strange TLS version < 1.0 but clearly openssl shouldn't error out when presented with lower ciphers it should negotiate the mutually acceptable version. -- System Information: Debian Release: buster/sid APT prefers testing APT policy: (500, 'testing'), (500, 'stable') Architecture: i386 (i686) Kernel: Linux 4.18.0-2-686 (SMP w/1 CPU core) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages openssl depends on: ii libc6 2.27-8 ii libssl1.1 1.1.1-2 openssl recommends no packages. Versions of packages openssl suggests: ii ca-certificates 20170717 -- Configuration Files: /etc/ssl/openssl.cnf changed [not included] -- no debconf information
