Processed: Re: Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Processing control commands: > found -1 4.9.88-1+deb9u1 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as found in versions linux/4.9.88-1+deb9u1. > notfound -1 4.9.88-1 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba No longer marked as found in versions linux/4.9.88-1. -- 900821: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900821 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Control: found -1 4.9.88-1+deb9u1
Control: notfound -1 4.9.88-1
Hi Salvatore,
On zondag 20 november 2022 16:38:25 CET Salvatore Bonaccorso wrote:
> On Sun, Nov 20, 2022 at 04:26:45PM +0100, Diederik de Haas wrote:
> > Control: notfound -1 4.9.88-1+deb9u1
> > Control: found -1 4.9.88-1
>
> Hmm this one I do not understand, as 4.9.88-1+deb9u1 was a very
> targetted fix for two CVEs and reverting the "random: fix crng_ready()
> test" changes re-opening CVE-2018-1108.
Oh, I thought that the +debXYZ thing confuses the bug tracker and therefor
shouldn't be there*. All the notfound/found versions I reported were tested
with a backports kernel, so +debXYZ, but I marked them without that suffix.
I thought that the 4.9.88-1+deb9u1 should therefor be corrected to be without
the suffix.
(I don't see how that affects the CVEs though as that has nothing to do with
this bug? But it's (very) possible I'm too tired atm)
*) The dependency graph at the top doesn't show the 4.9 version; later I
realized that another 4.9 version was already there and that wasn't reflected
in that graph either, so I reverted the change.
I don't know if the other found/notfound versions should be corrected.
> > IOW: that's your educated guess a git bisect could turn up?
>
> Not really. I was more looking at between versions you are not able to
> reproduce the issue, looking through the upstream changes commits and
> noticing that dacb5d8875cc ("tcp: fix page frag corruption on page
> fault") mentions:
>
> [...]
> Steffen reported a TCP stream corruption for HTTP requests
> served by the apache web-server using a cifs mount-point
> and memory mapping the relevant file.
> [...]
>
> and then noticing that the upstrema commit was backported to 5.10.84
> an 5.15.7, which fall exactly in the ranges you have the switch of
> result.
That was what I actually meant, but I now realize that git bisect finds the
cause while your educated guess is about the solution :-)
> > I can try that*, although I'm not clear onto what I should apply it.
> > Should I apply it to linux/5.10.70-1 or 5.10.46-4 f.e.? Or onto an
> > entirely
> > different version?
>
> Basically I wonder if c6f340a331fb72e5ac23a083de9c780e132ca3ae in
> 5.10.84 fixes the issue, and
> c6f340a331fb72e5ac23a083de9c780e132ca3ae~1 still would show the
> problem.
>
> Alterntively if 5.10.70-1 + commit fixes the issue.
It won't be tonight, but I'll likely try that :-)
Cheers,
Diederik
signature.asc
Description: This is a digitally signed message part.
Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Hi Diederik,
On Sun, Nov 20, 2022 at 04:26:45PM +0100, Diederik de Haas wrote:
> Control: notfound -1 4.9.88-1+deb9u1
> Control: found -1 4.9.88-1
Hmm this one I do not understand, as 4.9.88-1+deb9u1 was a very
targetted fix for two CVEs and reverting the "random: fix crng_ready()
test" changes re-opening CVE-2018-1108.
> On zondag 20 november 2022 13:55:09 CET Salvatore Bonaccorso wrote:
> > Seems the BSP was productive :).
>
> Yeah, once I set up the VM and created the script, it was actually quite easy.
:-)
> > If you have spare cycles, might you
> > check if dacb5d8875cc ("tcp: fix page frag corruption on page fault")
> > in 5.16-rc4 is the commit we are searching?
> > That one was backported to the 5.10.y series in 5.10.84 and in 5.15.y series
> > in 5.15.7 which would fall into your found ranges as well.
>
> IOW: that's your educated guess a git bisect could turn up?
Not really. I was more looking at between versions you are not able to
reproduce the issue, looking through the upstream changes commits and
noticing that dacb5d8875cc ("tcp: fix page frag corruption on page
fault") mentions:
[...]
Steffen reported a TCP stream corruption for HTTP requests
served by the apache web-server using a cifs mount-point
and memory mapping the relevant file.
[...]
and then noticing that the upstrema commit was backported to 5.10.84
an 5.15.7, which fall exactly in the ranges you have the switch of
result.
> I can try that*, although I'm not clear onto what I should apply it.
> Should I apply it to linux/5.10.70-1 or 5.10.46-4 f.e.? Or onto an entirely
> different version?
Basically I wonder if c6f340a331fb72e5ac23a083de9c780e132ca3ae in
5.10.84 fixes the issue, and
c6f340a331fb72e5ac23a083de9c780e132ca3ae~1 still would show the
problem.
Alterntively if 5.10.70-1 + commit fixes the issue.
Regards,
Salvatore
Processed: Re: Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Processing control commands: > notfound -1 4.9.88-1+deb9u1 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba No longer marked as found in versions linux/4.9.88-1+deb9u1. > found -1 4.9.88-1 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as found in versions linux/4.9.88-1. -- 900821: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900821 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Control: notfound -1 4.9.88-1+deb9u1
Control: found -1 4.9.88-1
On zondag 20 november 2022 13:55:09 CET Salvatore Bonaccorso wrote:
> Seems the BSP was productive :).
Yeah, once I set up the VM and created the script, it was actually quite easy.
> If you have spare cycles, might you
> check if dacb5d8875cc ("tcp: fix page frag corruption on page fault")
> in 5.16-rc4 is the commit we are searching?
> That one was backported to the 5.10.y series in 5.10.84 and in 5.15.y series
> in 5.15.7 which would fall into your found ranges as well.
IOW: that's your educated guess a git bisect could turn up?
I can try that*, although I'm not clear onto what I should apply it.
Should I apply it to linux/5.10.70-1 or 5.10.46-4 f.e.? Or onto an entirely
different version?
> OTOH, it fixes 5640f7685831 ("net: use a per task frag allocator")
> which is way much longer back as where the people noticed to be
> introduced.
While the focus has been on the kernel (possibly rightly so), there were 2
other packages (apache and samba) involved. It could be that on Buster f.e.
the problem still exists (even with a backports kernel).
I actually did look for an oldstable (net-)installer, but I couldn't find one.
I later realized that maybe there are DVDs of old releases, but I doubt I'll
do sth with that (at least wrt this bug).
*) I had plenty of cycles when I started this reply, but less so when I'm
sending it. I may do this some other day though.
signature.asc
Description: This is a digitally signed message part.
Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Hi,
On Sun, Nov 20, 2022 at 12:30:53PM +0100, Diederik de Haas wrote:
> Control: found -1 5.14.9-2
> Control: found -1 5.15.5-2
> Control: fixed -1 5.15.15-2
> Control: fixed -1 5.16.11-1
> Control: fixed -1 5.18.16-1
> Control: fixed -1 6.0.3-1
>
> On zondag 20 november 2022 12:18:50 CET you wrote:
> > Control: found -1 5.14.9-2 5.15.5-2
> > Control: fixed -1 5.15.15-2 5.16.11-1 5.18.16-1 6.0.3-1
>
> That doesn't seem to work and I probably should just use the last version
> for 'found' and the first one for 'fixed', but I wanted to make sure (and
> log)
> that the latter ones *all* are fixed.
Seems the BSP was productive :). If you have spare cycles, might you
check if dacb5d8875cc ("tcp: fix page frag corruption on page fault")
in 5.16-rc4 is the commit we are searching? That one was backported to
the 5.10.y series in 5.10.84 and in 5.15.y series in 5.15.7 which
would fall into your found ranges as well.
OTOH, it fixes 5640f7685831 ("net: use a per task frag allocator")
which is way much longer back as where the people noticed to be
introduced.
Regards,
Salvatore
Processed: Re: Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Processing control commands: > found -1 5.14.9-2 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as found in versions linux/5.14.9-2. > found -1 5.15.5-2 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as found in versions linux/5.15.5-2. > fixed -1 5.15.15-2 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as fixed in versions linux/5.15.15-2. > fixed -1 5.16.11-1 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as fixed in versions linux/5.16.11-1. > fixed -1 5.18.16-1 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as fixed in versions linux/5.18.16-1. > fixed -1 6.0.3-1 Bug #900821 [src:linux] apache reads wrong data over cifs filesystems served by samba Marked as fixed in versions linux/6.0.3-1. -- 900821: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900821 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Control: found -1 5.14.9-2 Control: found -1 5.15.5-2 Control: fixed -1 5.15.15-2 Control: fixed -1 5.16.11-1 Control: fixed -1 5.18.16-1 Control: fixed -1 6.0.3-1 On zondag 20 november 2022 12:18:50 CET you wrote: > Control: found -1 5.14.9-2 5.15.5-2 > Control: fixed -1 5.15.15-2 5.16.11-1 5.18.16-1 6.0.3-1 That doesn't seem to work and I probably should just use the last version for 'found' and the first one for 'fixed', but I wanted to make sure (and log) that the latter ones *all* are fixed. signature.asc Description: This is a digitally signed message part.
Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
Control: found -1 5.14.9-2 5.15.5-2 Control: fixed -1 5.15.15-2 5.16.11-1 5.18.16-1 6.0.3-1 On zaterdag 19 november 2022 21:22:46 CET Diederik de Haas wrote: > found 900821 5.10.70-1 > fixed 900821 5.10.84-1 I continued testing with kernels from backports and updated metadata accordingly. Interesting to note is that the 2 kernels that failed, did NOT show any message either in my ssh session nor in `dmesg`. Also interesting is that when it failed, it *always* failed at the first run. So in conclusion, in *current* Stable and Testing and Unstable the issue is fixed!diederik@prancing-pony:~/dev/debian/bugs/900821$ ssh vm-bullseye Linux debian-bullseye 5.14.0-0.bpo.2-amd64 #1 SMP Debian 5.14.9-2~bpo11+1 (2021-10-10) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sun Nov 20 11:22:02 2022 from 192.168.122.1 debian@debian-bullseye:~$ dpkg -l | grep linux-image ii linux-image-5.10.0-10-amd645.10.84-1 amd64Linux 5.10 for 64-bit PCs (signed) ii linux-image-5.10.0-19-amd645.10.149-2 amd64Linux 5.10 for 64-bit PCs (signed) ii linux-image-5.10.0-9-amd64 5.10.70-1 amd64Linux 5.10 for 64-bit PCs (signed) ii linux-image-5.14.0-0.bpo.2-amd64 5.14.9-2~bpo11+1 amd64Linux 5.14 for 64-bit PCs (signed) ii linux-image-5.15.0-0.bpo.2-amd64 5.15.5-2~bpo11+1 amd64Linux 5.15 for 64-bit PCs (signed) ii linux-image-5.15.0-0.bpo.3-amd64 5.15.15-2~bpo11+1 amd64Linux 5.15 for 64-bit PCs (signed) ii linux-image-5.16.0-0.bpo.3-amd64 5.16.11-1~bpo11+1 amd64Linux 5.16 for 64-bit PCs (signed) ii linux-image-5.16.0-0.bpo.4-amd64 5.16.12-1~bpo11+1 amd64Linux 5.16 for 64-bit PCs (signed) ii linux-image-5.18.0-0.bpo.1-amd64 5.18.2-1~bpo11+1 amd64Linux 5.18 for 64-bit PCs (signed) ii linux-image-5.18.0-0.deb11.4-amd64 5.18.16-1~bpo11+1 amd64Linux 5.18 for 64-bit PCs (signed) ii linux-image-6.0.0-0.deb11.2-amd64-unsigned 6.0.3-1~bpo11+1 amd64Linux 6.0 for 64-bit PCs ii linux-image-amd64 5.10.149-2 amd64Linux for 64-bit PCs (meta-package) debian@debian-bullseye:~$ mount /var/www/html/ debian@debian-bullseye:~$ sha256sum /var/www/html/100Mzero 20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e /var/www/html/100Mzero debian@debian-bullseye:~$ ./bug900821 sha256sum should be: 20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e . Bug 900821 triggered! Calculated SHA256: 5d9654b5f3258475e95313f7228fe142ad59f860e6d58ff4f47394bdc0b791ce debian@debian-bullseye:~$ su -l Password: root@debian-bullseye:~# reboot Connection to 192.168.122.2 closed by remote host. Connection to 192.168.122.2 closed. diederik@prancing-pony:~/dev/debian/bugs/900821$ ssh vm-bullseye Linux debian-bullseye 5.15.0-0.bpo.2-amd64 #1 SMP Debian 5.15.5-2~bpo11+1 (2022-01-02) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sun Nov 20 11:24:06 2022 from 192.168.122.1 debian@debian-bullseye:~$ mount /var/www/html/ debian@debian-bullseye:~$ sha256sum /var/www/html/100Mzero 20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e /var/www/html/100Mzero debian@debian-bullseye:~$ ./bug900821 sha256sum should be: 20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e . Bug 900821 triggered! Calculated SHA256: 7f7320e116b21b9867e42d8317dfd28a7c102ad7f5d28903debea699a144 debian@debian-bullseye:~$ su -l Password: root@debian-bullseye:~# reboot Connection to 192.168.122.2 closed by remote host. Connection to 192.168.122.2 closed. diederik@prancing-pony:~/dev/debian/bugs/900821$ ssh vm-bullseye Linux debian-bullseye 5.15.0-0.bpo.3-amd64 #1 SMP Debian 5.15.15-2~bpo11+1 (2022-02-03) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sun Nov 20 11:30:30 2022 from 192.168.122.1 debian@debian-bullseye:~$ mount /var/www/html/ debian@debian-bullseye:~$ sha256sum /var/www/html/100Mzero
Bug#900821: Found working and failing 5.10 versions and got kernel crash, report from BSP Tilburg (https://deb.li/iiOID)
user debian-rele...@lists.debian.org
usertag 900821 + bsp-2022-11-nl-tilburg
found 900821 5.10.70-1
fixed 900821 5.10.84-1
thanks
We have/had a BSP in Tilburg today and I went to work on this bug.
With success :-)
I first created a new Bullseye VM with virt-manager using the
debian-11.5.0-amd64-netinst.iso installation media and installed only
the Standard System Utilities and SSH Server.
Upon reboot I installed vim (ofc) and followed the excellent instructions
from OP and installed the packages:
"apt-get install samba apache2 cifs-utils"
Then I added the `[ftp]` block to /etc/samba/smb.conf and created the
`/srv/ftp/100Mzero` file consisting of only zero's.
As I'd be rebooting often, I made mounting easy by adding to `/etc/fstab/`:
//localhost/ftp /var/www/html cifs
username=debian,password=root,noauto,user 0 0
Then I created the following script:
```
debian@debian-bullseye:~$ cat bug900821
#!/bin/sh
#/srv/ftp/100Mzero
SHA256_STORED="20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e"
echo "sha256sum should be:"
echo " $SHA256_STORED"
echo ""
i=0
while [ $i -ne 100 ]
do
i=$((i + 1))
#printf "$i "
printf ". "
SHA256_CALC="$(wget http://localhost/100Mzero -O - 2>/dev/null |
sha256sum | awk '{ print $1 }')"
if [ "$SHA256_CALC" != "$SHA256_STORED" ] ; then
printf "\nBug 900821 triggered! Calculated SHA256: %s\n"
"$SHA256_CALC"
exit 1
fi
done
printf "\nTest completed\n"
```
I tried it out on the installed kernel, 5.10.149-2 aka 5.10.0-19-amd64, and
found out it all worked as expected. Idem ditto for 5.10.140-1/5.10.0-18-amd64.
I then tried 5.10.28-1/5.10.0-6-amd64 and got a kernel crash \o/
I then went on to narrow down which version worked and which next lower version
did fail and it turned out 5.10.70-1 failed, while 5.10.84-1 succeeded.
What was both odd and interesting was what happened during a crash.
With 5.10.28-1 I got this:
debian@debian-bullseye:~$ ./bug900821
sha256sum should be:
20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e
.
Message from syslogd@debian-bullseye at Nov 19 18:43:22 ...
kernel:[ 40.266711] usercopy: Kernel memory exposure attempt detected from
SLUB object 'vm_area_struct' (offset 0, size 117)!
That killed my SSH session. I don't know if it was the case each time, but
at least twice I was able to directly log in to the VM ... and I was able
to secure `dmesg` which shows the kernel crash :-)
I have attached both that dmesg output as the Konsole output from my
BSP session wrt this bug.
I don't know if it's useful to do a (lengthy!) git-bisect session as I'm quite
sure the issue is fixed.
I've updated the metadata, but I haven't closed it (yet?).
Cheers,
Diederik[0.00] Linux version 5.10.0-9-amd64 (debian-ker...@lists.debian.org)
(gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian)
2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)
[0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-5.10.0-9-amd64
root=UUID=a7893f33-27c0-4b56-8d49-b0e98db7f84a ro quiet
[0.00] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point
registers'
[0.00] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[0.00] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[0.00] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[0.00] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[0.00] x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64
[0.00] x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64
[0.00] x86/fpu: Enabled xstate features 0x1f, context size is 960
bytes, using 'compacted' format.
[0.00] BIOS-provided physical RAM map:
[0.00] BIOS-e820: [mem 0x-0x0009fbff] usable
[0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved
[0.00] BIOS-e820: [mem 0x000f-0x000f] reserved
[0.00] BIOS-e820: [mem 0x0010-0x3ffdbfff] usable
[0.00] BIOS-e820: [mem 0x3ffdc000-0x3fff] reserved
[0.00] BIOS-e820: [mem 0xb000-0xbfff] reserved
[0.00] BIOS-e820: [mem 0xfed1c000-0xfed1] reserved
[0.00] BIOS-e820: [mem 0xfeffc000-0xfeff] reserved
[0.00] BIOS-e820: [mem 0xfffc-0x] reserved
[0.00] NX (Execute Disable) protection: active
[0.00] SMBIOS 2.8 present.
[0.00] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.16.0-debian-1.16.0-4 04/01/2014
[0.00] Hypervisor detected: KVM
[0.00] kvm-clock: Using msrs 4b564d01 and 4b564d00
[0.00] kvm-clock: cpu 0, msr 2c4b3001, primary cpu clock
[0.00] kvm-clock: using sched offset of 1463003892121 cycles
[
