** Description changed: - Apache is lacking proper support for post-handhake-auth in TLSv1.3 POST - requests using certificate authentication. + [Impact] + Apache is lacking proper support for post-handhake-auth in TLSv1.3 POST requests using certificate authentication. This is used by freeipa, but any client doing a TLSv1.3 POST with certificate authentication is impacted and would need to downgrade the protocol to TLSv1.2. This was fixed in debian[1] via patches from upstream[2]. There is an upstream bug report[3] requesting the backport of these patches from trunk. - Test case: + It's also being shipped in Fedora[4] already. + + + [Test Case] $ lxc launch ubuntu-daily:focal ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this other bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key These certs are luckily still valid until june 2020, so they can be used for this bug as well. Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: cd /root wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c> <VirtualHost _default_:443> LogLevel info ssl:warn ServerAdmin webmaster@localhost DocumentRoot /var/www/html LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl ErrorLog \${APACHE_LOG_DIR}/error.log CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl SSLEngine on SSLCertificateFile /etc/apache2/ubuntu.pem SSLCertificateKeyFile /etc/apache2/ubuntu.key SSLCACertificateFile /etc/apache2/cacert.pem <FilesMatch "\.(cgi|shtml|phtml|php)$"> SSLOptions +StdEnvVars </FilesMatch> <Directory /usr/lib/cgi-bin> SSLOptions +StdEnvVars </Directory> <Location /> SSLRenegBufferSize 1024 SSLVerifyClient require Require ssl-verify-client </Location> </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 Verify that cert authentication is required: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --tlsv1.3 -f -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0 Verify that a GET request with the client certificate works: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 969k 0 --:--:-- --:--:-- --:--:-- 969k Verify that POST request with the client certificate fails: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -F bug=true -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 142 0 0 100 142 0 12909 --:--:-- --:--:-- --:--:-- 14200 curl: (22) The requested URL returned error: 403 Forbidden Server logs for the above: ==> /var/log/apache2/error.log <== [Mon Apr 13 14:49:03.357183 2020] [ssl:error] [pid 12248:tid 139782462109440] [client 10.0.100.38:41002] AH02263: Re-negotiation handshake failed: Client certificate missing ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:14:49:03 +0000] "POST / HTTP/1.1" 403 3798 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=- With the fixed packages, the POST request works: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -F bug=true -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11060 100 10918 100 142 101k 1352 --:--:-- --:--:-- --:--:-- 102k And the server log confirms it was a POST request, using certificates, and TLSv1.3: ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:14:52:26 +0000] "POST / HTTP/1.1" 200 17118 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=client-auth To test the error message changed by tlsv13-add-logno.diff, submit a slightly bigger POST request: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -H "Expect:" -F file=@/bin/ls % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 139k 0 0 100 139k 0 27.1M --:--:-- --:--:-- --:--:-- 27.1M curl: (22) The requested URL returned error: 413 Request Entity Too Large And in the server log: ==> /var/log/apache2/error.log <== [Mon Apr 13 15:00:37.446562 2020] [ssl:error] [pid 13415:tid 140391466624768] [client 10.0.100.38:41272] AH02018: request body exceeds maximum size (1024) for SSL buffer [Mon Apr 13 15:00:37.446620 2020] [ssl:error] [pid 13415:tid 140391466624768] [client 10.0.100.38:41272] AH10228: could not buffer message body to allow TLS Post-Handshake Authentication to proceed ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:15:00:37 +0000] "POST / HTTP/1.1" 413 3721 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=- + + [Regression Potential] + + [Other Info] + 1. https://bugs.debian.org/955348 2. https://svn.apache.org/viewvc?view=revision&revision=1870095 and https://svn.apache.org/viewvc?view=revision&revision=1870097 3. https://bz.apache.org/bugzilla/show_bug.cgi?id=64242 + 4. https://bugzilla.redhat.com/show_bug.cgi?id=1775146
** Description changed: [Impact] Apache is lacking proper support for post-handhake-auth in TLSv1.3 POST requests using certificate authentication. This is used by freeipa, but any client doing a TLSv1.3 POST with certificate authentication is impacted and would need to downgrade the protocol to TLSv1.2. This was fixed in debian[1] via patches from upstream[2]. There is an upstream bug report[3] requesting the backport of these patches from trunk. It's also being shipped in Fedora[4] already. - [Test Case] $ lxc launch ubuntu-daily:focal ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this other bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key These certs are luckily still valid until june 2020, so they can be used for this bug as well. Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: cd /root wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c> <VirtualHost _default_:443> LogLevel info ssl:warn ServerAdmin webmaster@localhost DocumentRoot /var/www/html LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl ErrorLog \${APACHE_LOG_DIR}/error.log CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl SSLEngine on SSLCertificateFile /etc/apache2/ubuntu.pem SSLCertificateKeyFile /etc/apache2/ubuntu.key SSLCACertificateFile /etc/apache2/cacert.pem <FilesMatch "\.(cgi|shtml|phtml|php)$"> SSLOptions +StdEnvVars </FilesMatch> <Directory /usr/lib/cgi-bin> SSLOptions +StdEnvVars </Directory> <Location /> SSLRenegBufferSize 1024 SSLVerifyClient require Require ssl-verify-client </Location> </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 Verify that cert authentication is required: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --tlsv1.3 -f -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0 Verify that a GET request with the client certificate works: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 969k 0 --:--:-- --:--:-- --:--:-- 969k Verify that POST request with the client certificate fails: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -F bug=true -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 142 0 0 100 142 0 12909 --:--:-- --:--:-- --:--:-- 14200 curl: (22) The requested URL returned error: 403 Forbidden Server logs for the above: ==> /var/log/apache2/error.log <== [Mon Apr 13 14:49:03.357183 2020] [ssl:error] [pid 12248:tid 139782462109440] [client 10.0.100.38:41002] AH02263: Re-negotiation handshake failed: Client certificate missing ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:14:49:03 +0000] "POST / HTTP/1.1" 403 3798 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=- With the fixed packages, the POST request works: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -F bug=true -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11060 100 10918 100 142 101k 1352 --:--:-- --:--:-- --:--:-- 102k And the server log confirms it was a POST request, using certificates, and TLSv1.3: ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:14:52:26 +0000] "POST / HTTP/1.1" 200 17118 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=client-auth To test the error message changed by tlsv13-add-logno.diff, submit a slightly bigger POST request: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -H "Expect:" -F file=@/bin/ls % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 139k 0 0 100 139k 0 27.1M --:--:-- --:--:-- --:--:-- 27.1M curl: (22) The requested URL returned error: 413 Request Entity Too Large And in the server log: ==> /var/log/apache2/error.log <== [Mon Apr 13 15:00:37.446562 2020] [ssl:error] [pid 13415:tid 140391466624768] [client 10.0.100.38:41272] AH02018: request body exceeds maximum size (1024) for SSL buffer [Mon Apr 13 15:00:37.446620 2020] [ssl:error] [pid 13415:tid 140391466624768] [client 10.0.100.38:41272] AH10228: could not buffer message body to allow TLS Post-Handshake Authentication to proceed ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:15:00:37 +0000] "POST / HTTP/1.1" 413 3721 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=- - - [Regression Potential] + [Regression Potential] + TLSv1.3 has introduced changes that generated bugs in the past. The PHA change in particular is still to this day impacting many clients. Clients who claim to support TLSv1.3, negotiate this version of the protocol, but don't implement it fully and lack PHA. + Regressions can happen, but we should be able to back this change out in that case. It also gives some comfort knowing that this change is already applied upstream (but not backported to 2.4), and in other distributions (debian and fedora). It's also good that we have a simple test case. [Other Info] + If this can't make it into focal prior to release, it can become an SRU, but the versioning might have to be changed then (i.e., become 2.4.41-4ubuntu2.1 instead of 2.4.41-4ubuntu3). 1. https://bugs.debian.org/955348 2. https://svn.apache.org/viewvc?view=revision&revision=1870095 and https://svn.apache.org/viewvc?view=revision&revision=1870097 3. https://bz.apache.org/bugzilla/show_bug.cgi?id=64242 4. https://bugzilla.redhat.com/show_bug.cgi?id=1775146 ** Description changed: [Impact] Apache is lacking proper support for post-handhake-auth in TLSv1.3 POST requests using certificate authentication. This is used by freeipa, but any client doing a TLSv1.3 POST with certificate authentication is impacted and would need to downgrade the protocol to TLSv1.2. This was fixed in debian[1] via patches from upstream[2]. There is an upstream bug report[3] requesting the backport of these patches from trunk. It's also being shipped in Fedora[4] already. [Test Case] $ lxc launch ubuntu-daily:focal ubuntu Enter the container as root: $ lxc exec ubuntu bash Verify hostname is "ubuntu": # hostname ubuntu Install apache2: apt update && apt install apache2 Download the following files from this other bug report and place them in /etc/apache2: cd /etc/apache2 wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key These certs are luckily still valid until june 2020, so they can be used for this bug as well. Adjust permissions of the key file: chmod 0640 /etc/apache2/ubuntu.key chgrp www-data /etc/apache2/ubuntu.key Download the client certificate and key files and place them in /root: cd /root wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line): cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF <IfModule mod_ssl.c> <VirtualHost _default_:443> LogLevel info ssl:warn ServerAdmin webmaster@localhost DocumentRoot /var/www/html LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl ErrorLog \${APACHE_LOG_DIR}/error.log CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl SSLEngine on SSLCertificateFile /etc/apache2/ubuntu.pem SSLCertificateKeyFile /etc/apache2/ubuntu.key SSLCACertificateFile /etc/apache2/cacert.pem <FilesMatch "\.(cgi|shtml|phtml|php)$"> SSLOptions +StdEnvVars </FilesMatch> <Directory /usr/lib/cgi-bin> SSLOptions +StdEnvVars </Directory> <Location /> SSLRenegBufferSize 1024 SSLVerifyClient require Require ssl-verify-client </Location> </VirtualHost> </IfModule> EOF Enable the ssl module and this new vhost we just created: a2enmod ssl && a2ensite cert-auth-test.conf Restart apache2: systemctl restart apache2 Verify that cert authentication is required: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --tlsv1.3 -f -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0 Verify that a GET request with the client certificate works: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10918 100 10918 0 0 969k 0 --:--:-- --:--:-- --:--:-- 969k Verify that POST request with the client certificate fails: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -F bug=true -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 142 0 0 100 142 0 12909 --:--:-- --:--:-- --:--:-- 14200 curl: (22) The requested URL returned error: 403 Forbidden Server logs for the above: ==> /var/log/apache2/error.log <== [Mon Apr 13 14:49:03.357183 2020] [ssl:error] [pid 12248:tid 139782462109440] [client 10.0.100.38:41002] AH02263: Re-negotiation handshake failed: Client certificate missing ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:14:49:03 +0000] "POST / HTTP/1.1" 403 3798 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=- With the fixed packages, the POST request works: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -F bug=true -H "Expect:" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11060 100 10918 100 142 101k 1352 --:--:-- --:--:-- --:--:-- 102k And the server log confirms it was a POST request, using certificates, and TLSv1.3: ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:14:52:26 +0000] "POST / HTTP/1.1" 200 17118 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=client-auth To test the error message changed by tlsv13-add-logno.diff, submit a slightly bigger POST request: root@ubuntu:~# curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.3 -f -H "Expect:" -F file=@/bin/ls % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 139k 0 0 100 139k 0 27.1M --:--:-- --:--:-- --:--:-- 27.1M curl: (22) The requested URL returned error: 413 Request Entity Too Large And in the server log: ==> /var/log/apache2/error.log <== [Mon Apr 13 15:00:37.446562 2020] [ssl:error] [pid 13415:tid 140391466624768] [client 10.0.100.38:41272] AH02018: request body exceeds maximum size (1024) for SSL buffer [Mon Apr 13 15:00:37.446620 2020] [ssl:error] [pid 13415:tid 140391466624768] [client 10.0.100.38:41272] AH10228: could not buffer message body to allow TLS Post-Handshake Authentication to proceed ==> /var/log/apache2/access.log <== 10.0.100.38 - - [13/Apr/2020:15:00:37 +0000] "POST / HTTP/1.1" 413 3721 "-" "curl/7.68.0" protocol=TLSv1.3 commonName=- [Regression Potential] TLSv1.3 has introduced changes that generated bugs in the past. The PHA change in particular is still to this day impacting many clients. Clients who claim to support TLSv1.3, negotiate this version of the protocol, but don't implement it fully and lack PHA. Regressions can happen, but we should be able to back this change out in that case. It also gives some comfort knowing that this change is already applied upstream (but not backported to 2.4), and in other distributions (debian and fedora). It's also good that we have a simple test case. [Other Info] If this can't make it into focal prior to release, it can become an SRU, but the versioning might have to be changed then (i.e., become 2.4.41-4ubuntu2.1 instead of 2.4.41-4ubuntu3). - 1. https://bugs.debian.org/955348 + 1. https://bugs.debian.org/955348 and https://salsa.debian.org/apache-team/apache2/-/commit/86b49fbd189484353d8462f2eb694cd6c9a53342 2. https://svn.apache.org/viewvc?view=revision&revision=1870095 and https://svn.apache.org/viewvc?view=revision&revision=1870097 3. https://bz.apache.org/bugzilla/show_bug.cgi?id=64242 4. https://bugzilla.redhat.com/show_bug.cgi?id=1775146 -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1872478 Title: Support TLSv1.3 PHA in POST requests with cert authentication To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1872478/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
