Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu
As the bug is now fixed upstream [1], can we get a patch for the current stable (bullseye)? Because it's impossible to have SSL on apache without the mod-gnutls I know maybe it's impossible to re-insert a removed package to stable, but, I beg the Debian people to let this package come back directly to stable, because it is pretty much needed. [1] https://lists.gnupg.org/pipermail/mod_gnutls-devel/2023-February/000221.html El dom, 5 jun 2022 a la(s) 15:04, Félix Arreola Rodríguez ( fgatuno@gmail.com) escribió: > Finally managed to write a patch. This patch applies ok on mod-gnutls > 0.9.0 and not sure if it will work for buster. > > -- > Atte. Félix Arreola > Firmado con GPG 0x1e249ee4 > -- Atte. Félix Arreola Rodríguez, «Sin firma GPG»
Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu
Finally managed to write a patch. This patch applies ok on mod-gnutls
0.9.0 and not sure if it will work for buster.
--
Atte. Félix Arreola
Firmado con GPG 0x1e249ee4
Author: Félix Arreola RodrÃguez
Date: Thu, 5 Jun 2022 10:42:46 -0500
Subject: Fix a loop caused by timeout if mod_reqtimeout is enabled
Bug-Debian: https://bugs.debian.org/942737
When mod_reqtimeout is enabled, when mod_gnutls tries to read from
the previous apr_bucket, it returns a timeout. Mod_gnutls handles
incorrectly this timeout as a EAGAIN, causing to forever loop
and consumes 100% CPU on the current apache process.
---
diff --git a/src/gnutls_io.c b/src/gnutls_io.c
===
--- a/src/gnutls_io.c
+++ b/src/gnutls_io.c
@@ -269,6 +269,17 @@ static apr_status_t gnutls_io_input_read
ap_log_cerror(APLOG_MARK, APLOG_TRACE2, ctxt->input_rc, ctxt->c,
"%s: looping recv after '%s' (%d)",
__func__, gnutls_strerror(rc), rc);
+if (APR_STATUS_IS_TIMEUP (ctxt->input_rc)) {
+ /* Timeout by mod_reqtimeout, bye bye */
+ ap_log_cerror(APLOG_MARK,
+APLOG_INFO,
+ctxt->input_rc,
+ctxt->c,
+"GnuTLS: Timeout reading data. (%d) '%s'",
+rc,
+gnutls_strerror(rc));
+break;
+}
/* For a blocking read, loop and try again
* immediately. Otherwise just notify the caller. */
if (ctxt->input_block != APR_NONBLOCK_READ)
pgpktdURikVpl.pgp
Description: Firma digital OpenPGP
Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu
Tags 942737 security thanks -- Atte. Félix Arreola Firmado con GPG 0x1e249ee4 pgpJSeaWennzx.pgp Description: Firma digital OpenPGP
Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu
Now I think this bug, this could be used as DOS, should we call the security team to handle this? -- Atte. Félix Arreola Firmado con GPG 0x1e249ee4 pgpV3xZ_Wj95S.pgp Description: Firma digital OpenPGP
Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu
On Mon, 16 Dec 2019 02:37:50 +0100 =?UTF-8?Q?Bernhard_=c3=9cbelacker?= wrote: > Dear Maintainer, > tried to reconstruct the given backtrace with debug symbols > in a gdb session and came to following, maybe it could be > of some help. > (Still a proper backtrace with dbgsym packages > installed would be better.) > > Kind regards, > Bernhard > > Dear Maintainer: I came across this bugs too. The problem is a infinite loop between mod-reqtimeout and mod-gnutls. The mod-gnutls (and underlaying gnutls) tries to read some bytes when the client hasn't sent any (like an empty tcp conn) and the mod-reqtimeout returns a timeout, causing mod-gnutls to loop endless trying to read more. Either mod-reqtimeout is broken, or either mod-gnutls doesn't handle correctly the timeouts. After debugging a lot with gdb, I can reconstruct the loop as follow: The code starts at function gnutls_io_input_read from gnutls_io.c from mod_gnutls, line 246, the "while(1)" loop. The code calls gnutls_record_recv, from gnutls, which in turns calls _gnutls_recv_int. gnutls_recv_int is called with type=GNUTLS_APPLICATION_DATA asking for 8192 bytes. Next it calls check_session_status, which checks if there is an EOF, and there is NO EOF. Also, check_session_status says that the session->internals.recv_state is RECV_STATE_0, whichs makes check_session_status return 1. Next, it calls get_data_from_buffers, which returns 0, which means there is no data in the buffer to consume. Next, it calls _gnutls_recv_in_buffers. _gnutls_recv_in_buffers calls recv_headers, which tries to read the header (which is 5). it calls _gnutls_io_read_buffered for 5 bytes. _gnutls_io_read_buffered ask for 5 bytes to _gnutls_read, which in turn uses _gnutls_stream_read to complete the request. _gnutls_stream_read returns back to mod-gnutls using the "pull_func" mgs_transport_read. msg_transport_read is located at gnutls_io.c:824 mgs_transport_read tries to use ap_get_brigade to get some bytes from the underlaying socket. ap_get_brigade tries to read from the "next" ap_filter_t, which (in my case) is "reqtimeout". reqtimeout_filter function calls check_time_left, which returns APR_TIMEUP. When it returns APR_TIMEUP, it logs "Request %s read timeout". So, back to mgs_transport_read, it reads APR_TIMEUP from the ap_get_brigade. BUT, it handles wrong this timeout and converts it to EAGAIN, which makes this whole stack run back. This is a big, 'nice' nasty bug, I have to say. As a workout around, I disabled module reqtimeout. Seems to solve the cpu usage issue. But, the bug is way more big than 'disabling' reqtimeout. Because, the problem is between mod-gnutls and mod-reqtimeout Steps to reproduce: Enable apache2 with the modules mod-gnutls, and mod-reqtimeout. Setup a reqtimeout like: RequestReadTimeout header=20-40,minrate=500 and open an openssl s_client: openssl s_client -connect IP:port Don't send any data over the openssl connect. Just wait for the timeout to happen. After the timeout, the CPU usage will increase. Also, you can quit the openssl s_client and the apache process will be stuck in the endless loop. -- Atte. Félix Arreola Firmado con GPG 0x1e249ee4 pgpRhuwCqo6hD.pgp Description: Firma digital OpenPGP
Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu
Dear Maintainer, tried to reconstruct the given backtrace with debug symbols in a gdb session and came to following, maybe it could be of some help. (Still a proper backtrace with dbgsym packages installed would be better.) Kind regards, Bernhard Reconstructed: #0 0x7f78b4cfb92f in gnutls_assert_val_int at ../../lib/errors.h:139 from libgnutls.so.30 #1 0x7f78b4cfdf7c in _gnutls_recv_int at ../../lib/record.c:1773 from libgnutls.so.30 in gnutls_record_recv at ../../lib/record.c:2281 #2 0x7f78b4e90f38 in gnutls_io_input_read at gnutls_io.c:246 from mod_gnutls.so #3 0x7f78b4e91ad2 in gnutls_io_input_getline at gnutls_io.c:342 from mod_gnutls.so in mgs_filter_input at gnutls_io.c:595 in ap_get_brigade at util_filter.c:553 #4 0x55c220cd08e1 in ap_rgetline_core at protocol.c:246 #5 0x55c220cd336c in read_request_line at protocol.c:682 in ap_read_request at protocol.c:1322 #6 0x55c220cfe7a8 in ap_process_http_sync_connection at http_core.c:192 #7 0x55c220cf38b0 in ap_run_process_connection at connection.c:42 in ap_process_connection at connection.c:219 #8 0x7f78b3bd23df in child_main at prefork.c:615 from mod_mpm_prefork.so #9 0x7f78b3bd26d4 in make_child at prefork.c:717 from mod_mpm_prefork.so #10 0x7f78b3bd272f in startup_children at prefork.c:735 from mod_mpm_prefork.so #11 0x7f78b3bd32f3 in prefork_run at prefork.c:897 from mod_mpm_prefork.so #12 0x55c220ccc67e in ap_run_mpm at mpm_common.c:94 #13 0x55c220cc4f57 in main at main.c:819
Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu
Package: libapache2-mod-gnutls Version: 0.9.0-1 Severity: grave Tags: upstream Justification: renders package unusable Dear Maintainer, I have updated apache2 & mod gnutls application. In testing with a basic html page, I notice that apache2 process loops infinitly, "ps" output: www-data 6103 92.8 0.1 29696 9708 ?R19:37 28:24 /usr/sbin/apache2 -k start I try with ssl module: SSLEngine On SSLCertificateFile /etc/apache2/ssl/domain.com.crt SSLCertificateKeyFile /etc/apache2/ssl/domain.com.key GnuTLSEnable on GnuTLSCertificateFile /etc/apache2/ssl/domain.com.crt GnuTLSKeyFile /etc/apache2/ssl/domain.com.key GnuTLSPriorities PFS:-VERS-ALL:+VERS-TLS1.3:+VERS-TLS1.2:%SERVER_PRECEDENCE I have the issue only with gnutls module. To check, I attach a gdb session: (gdb) bt #0 0x7f78b4cfb92f in ?? () from target:/usr/lib/x86_64-linux-gnu/libgnutls.so.30 #1 0x7f78b4cfdf7c in ?? () from target:/usr/lib/x86_64-linux-gnu/libgnutls.so.30 #2 0x7f78b4e90f38 in ?? () from target:/usr/lib/apache2/modules/mod_gnutls.so #3 0x7f78b4e91ad2 in mgs_filter_input () from target:/usr/lib/apache2/modules/mod_gnutls.so #4 0x55c220cd08e1 in ap_rgetline_core () #5 0x55c220cd336c in ap_read_request () #6 0x55c220cfe7a8 in ?? () #7 0x55c220cf38b0 in ap_run_process_connection () #8 0x7f78b3bd23df in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so #9 0x7f78b3bd26d4 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so #10 0x7f78b3bd272f in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so #11 0x7f78b3bd32f3 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so #12 0x55c220ccc67e in ap_run_mpm () #13 0x55c220cc4f57 in main () I don't know how to fix the issue and how to help you. -- System Information: Debian Release: 10.1 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.19.0-6-amd64 (SMP w/4 CPU cores) Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages libapache2-mod-gnutls depends on: ii apache2-bin [apache2-api-20120211] 2.4.38-3+deb10u3 ii libc6 2.28-10 ii libgnutls30 3.6.7-4 ii libmsv1 1.1.1-3 libapache2-mod-gnutls recommends no packages. libapache2-mod-gnutls suggests no packages. -- no debconf information Available to do tests and more. Nico
