Package: squid
Version: 4.13-10+deb11u3
Severity: important
X-Debbugs-Cc: christophe.beaureg...@ec.gc.ca
Dear Maintainer,
I have a CGI script which serves up some larger (60+MB seems to be the
sweet spot) files via Apache through a squid caching reverse proxy. We've
recently been seeing those files being truncated (even when the content
doesn't change).
My hypothesis is the patch related to CVE-2023-46846 has either
broken chunked transfer encoding, or Apache is generating "bad" chunked
transfers which previous versions of squid were lax about handling.
I've reliably replicated the problem in a bullseye VM using fairly
minimal configurations:
/usr/lib/cgi-bin/bigfile.cgi:
#!/bin/sh
BS=16384
BC=4000
CL=`expr ${BS} \* ${BC}`
TF=/tmp/bigfile.out
if ! -f ${TF}
then
/usr/bin/dd bs=${BS} if=/dev/urandom of=${TF} count=${BC} 2>/dev/null
fi
/usr/bin/echo -e -n "Status: 200 OK\r\n"
/usr/bin/echo -e -n "Content-type: application/octet-stream\r\n"
/usr/bin/echo -e -n "Content-length: ${CL}\r\n"
/usr/bin/echo -e -n "Date: " `date -R -u` "\r\n"
/usr/bin/echo -e -n "\r\n"
exec /usr/bin/cat ${TF}
/etc/apache2/sites-enabled/test.conf:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Require all granted
</Directory>
</VirtualHost>
cd /etc/apache2/mods-enabled/ && ln -s ../mods-available/cgi.load
/etc/squid/squid.conf:
access_log /var/log/squid/access.log squid
cache_log /var/log/squid/cache.log squid
cache_store_log none
icp_port 3130
coredump_dir /var/spool/squid
cache_mem 768 MB
cache_dir ufs /var/spool/squid 1024 16 256
memory_pools on
request_header_access X-Forwarded-Proto allow all
request_header_access Other deny all
cache_peer http-test parent 80 0 no-query originserver name=localaccel
acl our_sites dstdomain http-test
http_access allow our_sites
cache_peer_access localaccel allow our_sites
http_port 8080 accel vhost
restart apache2 and squid.
A normal request directly to Apache looks like:
test@http-test:~$ wget -Ox -S http://http-test:80/cgi-bin/bigfile.cgi
--2024-07-25 12:04:31-- http://http-test/cgi-bin/bigfile.cgi
Resolving http-test (http-test)... 127.0.1.1
Connecting to http-test (http-test)|127.0.1.1|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Thu, 25 Jul 2024 16:04:31 GMT
Server: Apache/2.4.61 (Debian)
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/octet-stream
Length: unspecified [application/octet-stream]
Saving to: ‘x’
x [ <=> ] 62.50M 220MB/s in 0.3s
2024-07-25 12:04:32 (220 MB/s) - ‘x’ saved [65536000]
Same request through the reverse proxy:
test@http-test:~$ wget -Ox -S http://http-test:8080/cgi-bin/bigfile.cgi
--2024-07-25 12:05:01-- http://http-test:8080/cgi-bin/bigfile.cgi
Resolving http-test (http-test)... 127.0.1.1
Connecting to http-test (http-test)|127.0.1.1|:8080... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Thu, 25 Jul 2024 16:05:01 GMT
Server: Apache/2.4.61 (Debian)
Content-Type: application/octet-stream
X-Cache: MISS from http-test
X-Cache-Lookup: MISS from http-test:8080
Transfer-Encoding: chunked
Via: 1.1 http-test (squid/4.13)
Connection: keep-alive
Length: unspecified [application/octet-stream]
Saving to: ‘x’
x [ <=> ] 62.50M 98.0MB/s in 0.6s
2024-07-25 12:05:02 (98.0 MB/s) - ‘x’ saved [65536000]
But sometimes, the request gets silently truncated:
test@http-test:~$ wget -Ox -S http://http-test:8080/cgi-bin/bigfile.cgi
--2024-07-25 12:05:52-- http://http-test:8080/cgi-bin/bigfile.cgi
Resolving http-test (http-test)... 127.0.1.1
Connecting to http-test (http-test)|127.0.1.1|:8080... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Thu, 25 Jul 2024 16:05:52 GMT
Server: Apache/2.4.61 (Debian)
Content-Type: application/octet-stream
X-Cache: MISS from http-test
X-Cache-Lookup: MISS from http-test:8080
Transfer-Encoding: chunked
Via: 1.1 http-test (squid/4.13)
Connection: keep-alive
Length: unspecified [application/octet-stream]
Saving to: ‘x’
x [ <=> ] 10.33M --.-KB/s in 0.1s
2024-07-25 12:05:54 (89.1 MB/s) - ‘x’ saved [10835159]
The rate of truncated requests is wildly variable. In my test VM it
might be 20% of the time, on our operational systems it's more like 80%.
When the request truncation happens, Apache sees it as a client disconnect
(add a %X to the Apache LogFormat). More interestingly, if you add a:
debug_options ALL,1 05,3 11,3
to squid.conf, the truncated downloads will start to spit out:
2024/07/25 12:10:48.122 kid1| Exception error:garbage instead of CRLF line
terminator
exception location: Parser.cc(73) skipLineTerminator
at which point they close down the connection.
To make matters worse, if your CGI script has decent cache control,
squid appears to be treating these truncated files as "good", and will
serve them up as a cache hit.
The problem can be mitigated by adding:
SetEnv ap_trust_cgilike_cl 1
to the Apache cgi-bin Directory section, assuming the CGI script in
question generated a Content-length header. This prevents Apache from treating
it as chunked, and the resulting request is:
test@http-test:~$ wget -Ox -S http://http-test:8080/cgi-bin/bigfile.cgi
--2024-07-25 12:18:13-- http://http-test:8080/cgi-bin/bigfile.cgi
Resolving http-test (http-test)... 127.0.1.1
Connecting to http-test (http-test)|127.0.1.1|:8080... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Thu, 25 Jul 2024 16:18:13 GMT
Server: Apache/2.4.61 (Debian)
Content-Length: 65536000
Content-Type: application/octet-stream
X-Cache: MISS from http-test
X-Cache-Lookup: MISS from http-test:8080
Via: 1.1 http-test (squid/4.13)
Connection: keep-alive
Length: 65536000 (62M) [application/octet-stream]
Saving to: ‘x’
x 100%[===================>] 62.50M 29.9MB/s in 2.1s
2024-07-25 12:18:15 (29.9 MB/s) - ‘x’ saved [65536000/65536000]
-- System Information:
Debian Release: 11.10
APT prefers oldstable-updates
APT policy: (500, 'oldstable-updates'), (500, 'oldstable-security'), (500,
'oldstable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-31-amd64 (SMP w/4 CPU threads)
Locale: LANG=en_CA.UTF-8, LC_CTYPE=en_CA.UTF-8 (charmap=UTF-8),
LANGUAGE=en_CA:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages squid depends on:
ii adduser 3.118+deb11u1
ii init-system-helpers 1.60
ii libc6 2.31-13+deb11u10
ii libcap2 1:2.44-1
ii libcom-err2 1.46.2-2
ii libcrypt1 1:4.4.18-4
ii libdb5.3 5.3.28+dfsg1-0.8
ii libdbi-perl 1.643-3+b1
ii libecap3 1.0.1-3.2+b1
ii libexpat1 2.2.10-2+deb11u5
ii libgcc-s1 10.2.1-6
ii libgnutls30 3.7.1-5+deb11u5
ii libgssapi-krb5-2 1.18.3-6+deb11u5
ii libkrb5-3 1.18.3-6+deb11u5
ii libldap-2.4-2 2.4.57+dfsg-3+deb11u1
ii libltdl7 2.4.6-15
ii libnetfilter-conntrack3 1.0.8-3
ii libnettle8 3.7.3-1
ii libnsl2 1.3.0-2
ii libpam0g 1.4.0-9+deb11u1
ii libsasl2-2 2.1.27+dfsg-2.1+deb11u1
ii libstdc++6 10.2.1-6
ii libsystemd0 247.3-7+deb11u5
ii libxml2 2.9.10+dfsg-6.7+deb11u4
ii logrotate 3.18.0-2+deb11u2
ii lsb-base 11.1.0
ii netbase 6.3
ii squid-common 4.13-10+deb11u3
Versions of packages squid recommends:
ii ca-certificates 20210119
ii libcap2-bin 1:2.44-1
Versions of packages squid suggests:
ii apparmor 2.13.6-10
pn resolvconf <none>
pn smbclient <none>
pn squid-cgi <none>
pn squid-purge <none>
pn squidclient <none>
pn ufw <none>
pn winbind <none>
-- no debconf information
