Bug#1032104:
Root cause found, merge request here: https://salsa.debian.org/kernel-team/linux/-/merge_requests/917
Bug#1032104: Status update
I have traced this bug to a missing memory barrier in the powerpc IPI handling code. io_uring uses task_work_add() to schedule I/O worker creation, which in turn issues an IPI, and when precise timing conditions are met the inconsistent state between the two CPU cores can lead to corruption of userspace data in RAM. I have sent a patch upstream, and created a merge request for Debian here: https://salsa.debian.org/kernel-team/linux/-/merge_requests/907
Bug#1032104: Status update
I have traced this to a regression in the Linux kernel. The issue appears to be a type of data race that is more likely to occur on ppc64el than on other architectures, but is also likely to affect other architectures. The issue remains in the latest GIT version of the Linux kernel, and I am working with both upstream and our internal resources to try to isolate the root cause and generate a fix. In the interim, disabling the io_uring subsystem will allow mariadb to function normally. Given the nature of the kernel bug, I would recommend disabling io_uring entirely in the kernel configuration for affected systems, as other applications may also be impacted by the data corruption.
Bug#1051355: Processed: your mail
For 16 to work we'll need the Debian clang team to include this patchset: https://reviews.llvm.org/D158066 Any chance of that happening? - Original Message - > From: "Andres Salomon" > To: "Leandro Cunha" , 1051...@bugs.debian.org > Cc: "Timothy Pearson" > Sent: Sunday, September 10, 2023 11:43:18 PM > Subject: Re: Bug#1051355: Processed: your mail > Alright, I built 117 w/ clang-16 on sid and it doesn't segfault. Same > exact build but with clang-14 segfaults. > > Timothy, did you ever get the ppc64 issues with clang >= 15 squared > away? It's looking like I'm going to need to upload a build with > clang-16. > > On Sun, Sep 10 2023 at 03:07:29 PM -03:00:00, Leandro Cunha > wrote: >> Hi, >> >> Em dom., 10 de set. de 2023 15:01, Andres Salomon >> mailto:dilin...@queued.net>> escreveu: >>> Unfortunately 117 *also* segfaults on sid. >>> >>> I'm tempted to try a newer clang, but probably not 15 since debian's >>> planning to remove it. 16, I guess? >> >> Arch is already with Clang 16 and I tested Chromium 117 in a vm that >> I installed here and it was working normally.
Bug#1032104: Still present in Bookworm
We've started hitting this on a busy server after upgrading to Bookworm: 2023-09-07 17:00:31 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work. 2023-09-07 17:00:31 0 [Note] Server socket created on IP: '127.0.0.1'. 2023-09-07 17:00:31 0 [Note] /usr/sbin/mariadbd: ready for connections. Version: '10.11.3-MariaDB-1' socket: '/run/mysqld/mysqld.sock' port: 3306 Debian 12 2023-09-07 17:00:31 0 [Note] InnoDB: Buffer pool(s) load completed at 230907 17:00:31 2023-09-07 20:35:06 8630 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './database/data_table.ibd' page [page id: space=393, page number=1534]. You may have to recover from a backup. 2023-09-07 20:35:06 8630 [Note] InnoDB: Page dump (8192 bytes): 2023-09-07 20:35:06 8630 [Note] InnoDB: 86c518ee05fe05fd05ff00067ee5a53145bf 2023-09-07 20:35:06 8630 [Note] InnoDB: 0189000f3eff803d3e030002003a003b 2023-09-07 20:35:06 8630 [Note] InnoDB: 045a6881 2023-09-07 20:35:06 8630 [Note] InnoDB: 12d4ae6314accb64e0a8630400b55a4b8f6557796d5eb10dee577503 2023-09-07 20:35:06 8630 [Note] InnoDB: c64ee090070c90e9fd7e2014256e140c31c2b84d193051d88fefebbe72b9caaa 2023-09-07 20:35:06 8630 [Note] InnoDB: aa36b4985494699461a42852fe41a48ca248f91b5186496a1009f10320bc42d6 2023-09-07 20:35:06 8630 [Note] InnoDB: bef79c7b6fd53d7546a73df0b5caf5d53d6b7dafb5f63ed7ba3bdddbd7ceaeb5 2023-09-07 20:35:06 8630 [Note] InnoDB: 7fbefdb3d5e7b58ff2e2804eeedd3fa674ba789fee9547e9f8f4e4deebc7f4ee 2023-09-07 20:35:06 8630 [Note] InnoDB: e2f1bb2fef53393d3a7ef96be5e8f0e5d716f9381d3fb9f7069da6c5c1f26727 2023-09-07 20:35:06 8630 [Note] InnoDB: f71eec7ff5de57d2f13bf71e3c3a7aefbdc5e1c3ee95f4b013f28b27ef3f54c6 2023-09-07 20:35:06 8630 [Note] InnoDB: a44aac65a0646a544255978c7554b3d05652ff28ff3d12da3fdd5efff9cceaf3 2023-09-07 20:35:06 8630 [Note] InnoDB: f9bf6a9f7ff9853ffef2f667ff3bd7b2d4e43c9134ae70a69088490b1b6d0a2a 2023-09-07 20:35:06 8630 [Note] InnoDB: 7852f8bd97ae751feb1e0c1cfc7c6e0ebe4e3fa483e3270d8074ce09a77dd245 2023-09-07 20:35:06 8630 [Note] InnoDB: 7b321c284b67524ec12525580ed8b742c631dfbc3f85d9daa0133952d6280ab5 2023-09-07 20:35:06 8630 [Note] InnoDB: b028926a9456a6500d8b15e6dbdd7707ccfffbd4f27e1f7fa0c1705e0ac79c95 2023-09-07 20:35:06 8630 [Note] InnoDB: cb1c44c93269ca8695676d4b60dec9fa10388efff6fff4b8bfb4fd39e0a72cbc 2023-09-07 20:35:06 8630 [Note] InnoDB: 2c26daacaab2c20a5d3c809baa6355a29415feadbaffc5dcf8df4a070774daa5 2023-09-07 20:35:06 8630 [Note] InnoDB: c3da80d40684a565112db9a26a55896ba254bda7ec8a14b6e4818191d0710eae 2023-09-07 20:35:06 8630 [Note] InnoDB: ffd31407316a8a3a3b67854825e69482cd5278958b032f3d077bddf7060e7e39 2023-09-07 20:35:06 8630 [Note] InnoDB: 3707f7d3c9a3ee0de2c7402254c39102f08aaa751445db2475cd35e8685cd241 2023-09-07 20:35:06 8630 [Note] InnoDB: ca5a060a7623c719f8d4c15417184e5c925146061b0c33eace07e3654ac58283 2023-09-07 20:35:06 8630 [Note] InnoDB: 9e81e7bb3707067e357f172c96c5bc4c3fb28c311738666b63b5a1c860220aa0 2023-09-07 20:35:06 8630 [Note] InnoDB: ea628c8f9b06d8c48ca3defbd054de0ba98c660bd12a7c93641f3102285b6f63 2023-09-07 20:35:06 8630 [Note] InnoDB: 492a8715ea9bdd5b03ea5fcf8e3a1d1086f6c93bb46ce0200b5b4c7b2f7d724e 2023-09-07 20:35:06 8630 [Note] InnoDB: 3b5652b28dca5a5b83e5b0067e316c1cfbcdff9cca78100163d44aad35a66921 2023-09-07 20:35:06 8630 [Note] InnoDB: 937c44fb57a525c7487685fd56f79d01fb6fe6c6fe95633a39e9904384371482 2023-09-07 20:35:06 8630 [Note] InnoDB: 6b4e967c102a455d6d515a4b9b4375687965fd00fe72dc38fadb7f3f95791fdb 2023-09-07 20:35:06 8630 [Note] InnoDB: 2a11ca05cc561d42162260b21699bc9414d3d9e54df77f73a37f351d3ea4e325 2023-09-07 20:35:06 8630 [Note] InnoDB: 8050b35656b1e06c6370b93a8995a46b72a8f694f3007c2b641cf38dbda98cd7 2023-09-07 20:35:06 8630 [Note] InnoDB: a48553598548c280670cba5c94773561b4b0952bcccf76af0f987ffb34aa7db9 2023-09-07 20:35:06 8630 [Note] InnoDB: da83ab46d59443abbb1c83748c1fa4429cbc4a7abbcc2726da07a6d092d1de60 2023-09-07 20:35:06 8630 [Note] InnoDB: 60586da9266b52a02a5bad83578c939dbdb63f44ce57df8b9372f4f8f0b47bb3 2023-09-07 20:35:06 8630 [Note] InnoDB: 89b2e5682e3a20d55201371592229a9a527602722e665a36ddaac47743c739d8 2023-09-07 20:35:06 8630 [Note] InnoDB: fbc7290ed890434547634df499547186a3935240d529cb7dc637da66ff037373 2023-09-07 20:35:06 8630 [Note] InnoDB: d06fa7878b93533a6e304cb04a64af183c44e858283cfcc0d79065117993fa9d 2023-09-07 20:35:06 8630 [Note] InnoDB: c071fc2ffc5d8f7bb4cb33a5040e72c56c0f6d7114c8298af83236d5bb7ebedf 2023-09-07 20:35:06 8630 [Note] InnoDB: e886dcef7f706efc5f5df069f7e034adbad689e8bc434f1379a36cc88692a4a2 2023-09-07 20:35:06 8630 [Note] InnoDB: 30dcbca8d10de02f468d237ff1afa7328f0de25451b10a7210b34d356b2f8268 2023-09-07 20:35:06 8630 [Note] InnoDB: d97769691c2e56ff879e6ef52f2d092bacf0a86a4c55845ca3774a90909804d9 2023-09-07 20:35:06 8630 [Note] InnoDB:
Bug#990017: [REGRESSION] Bullseye CD installer images fail to install GRUB on OpenPOWER machines
- Original Message - > From: "Steve McIntyre" > To: "Timothy Pearson" , 990...@bugs.debian.org > Sent: Thursday, June 17, 2021 7:43:15 PM > Subject: Re: Bug#990017: [REGRESSION] Bullseye CD installer images fail to > install GRUB on OpenPOWER machines > Control: reassign -1 grub-ieee1275 > > Hi Timothy, > > Reassigning this to the correct package where it should get more > attention... Thank you, much appreciated. > On Thu, Jun 17, 2021 at 04:20:47PM -0500, Timothy Pearson wrote: >>Package: debian-cd >>Severity: Grave >> >>Attempting to use the latest Bullseye RC2 CD installer on an OpenPOWER machine >>results in a fatal error during bootloader installation: >> >>Jun 17 21:14:45 grub-installer: info: Installing grub on '/dev/sda1' >>Jun 17 21:14:45 grub-installer: info: grub-install does not support >>--no-floppy >>Jun 17 21:14:45 grub-installer: info: Running chroot /target grub-install >>--force "/dev/sda1" >>Jun 17 21:14:45 grub-installer: Installing for powerpc-ieee1275 platform. >>Jun 17 21:14:57 grub-installer: grub-install: error: unknown filesystem. >>Jun 17 21:14:57 grub-installer: error: Running 'grub-install --force >>"/dev/sda1"' failed. >> >>The bootloader installs normally using the Buster CD installers on the same >>hardware. > > Just a quick sanity check - how did you partition the disk? Does it > have the normal boot partition etc. needed for OpenPOWER? I'll admit I > don't know all the details here - I'm not a powerpc expert. All defaults. PReP partition was added automatically by the apparition, it almost looks like Grub doesn't know what to do with it? Layout: PReP: /dev/sda1 System: /dev/sda2 Swap: /dev/sda3
Bug#990017: [REGRESSION] Bullseye CD installer images fail to install GRUB on OpenPOWER machines
- Original Message - > From: "Timothy Pearson" > To: "Steve McIntyre" > Cc: "990017" <990...@bugs.debian.org> > Sent: Thursday, June 17, 2021 7:56:27 PM > Subject: Re: Bug#990017: [REGRESSION] Bullseye CD installer images fail to > install GRUB on OpenPOWER machines > - Original Message - >> From: "Steve McIntyre" >> To: "Timothy Pearson" , >> 990...@bugs.debian.org >> Sent: Thursday, June 17, 2021 7:43:15 PM >> Subject: Re: Bug#990017: [REGRESSION] Bullseye CD installer images fail to >> install GRUB on OpenPOWER machines > >> Control: reassign -1 grub-ieee1275 >> >> Hi Timothy, >> >> Reassigning this to the correct package where it should get more >> attention... > > Thank you, much appreciated. > >> On Thu, Jun 17, 2021 at 04:20:47PM -0500, Timothy Pearson wrote: >>>Package: debian-cd >>>Severity: Grave >>> >>>Attempting to use the latest Bullseye RC2 CD installer on an OpenPOWER >>>machine >>>results in a fatal error during bootloader installation: >>> >>>Jun 17 21:14:45 grub-installer: info: Installing grub on '/dev/sda1' >>>Jun 17 21:14:45 grub-installer: info: grub-install does not support >>>--no-floppy >>>Jun 17 21:14:45 grub-installer: info: Running chroot /target grub-install >>>--force "/dev/sda1" >>>Jun 17 21:14:45 grub-installer: Installing for powerpc-ieee1275 platform. >>>Jun 17 21:14:57 grub-installer: grub-install: error: unknown filesystem. >>>Jun 17 21:14:57 grub-installer: error: Running 'grub-install --force >>>"/dev/sda1"' failed. >>> >>>The bootloader installs normally using the Buster CD installers on the same >>>hardware. >> >> Just a quick sanity check - how did you partition the disk? Does it >> have the normal boot partition etc. needed for OpenPOWER? I'll admit I >> don't know all the details here - I'm not a powerpc expert. > > All defaults. PReP partition was added automatically by the apparition, it > almost looks like Grub doesn't know what to do with it? > > Layout: > PReP: /dev/sda1 > System: /dev/sda2 > Swap: /dev/sda3 Correction: that should read "by the partitioner" above. Autocorrect didn't like that word. :)
Bug#990017: [REGRESSION] Bullseye CD installer images fail to install GRUB on OpenPOWER machines
Package: debian-cd Severity: Grave Attempting to use the latest Bullseye RC2 CD installer on an OpenPOWER machine results in a fatal error during bootloader installation: Jun 17 21:14:45 grub-installer: info: Installing grub on '/dev/sda1' Jun 17 21:14:45 grub-installer: info: grub-install does not support --no-floppy Jun 17 21:14:45 grub-installer: info: Running chroot /target grub-install --force "/dev/sda1" Jun 17 21:14:45 grub-installer: Installing for powerpc-ieee1275 platform. Jun 17 21:14:57 grub-installer: grub-install: error: unknown filesystem. Jun 17 21:14:57 grub-installer: error: Running 'grub-install --force "/dev/sda1"' failed. The bootloader installs normally using the Buster CD installers on the same hardware.
Bug#836881: X509 card-based logins fail with SIGSEGV
Package: libhx509-5-heimdal Version: 1.7~git20160703+dfsg-1 Severity: grave Heimdal's X509 PAM authentication plugin crashes with the following backtrace when attempting to use a cryptographic card to log in: Program received signal SIGSEGV, Segmentation fault. 0x7f5c0f30e38d in _hx509_match_keys (c=0x18e2ed0, key=0x18e0960) at crypto.c:3018 3018crypto.c: No such file or directory. (gdb) bt #0 0x7f5c0f30e38d in _hx509_match_keys (c=0x18e2ed0, key=0x18e0960) at crypto.c:3018 #1 0x7f5c0f30b17b in match_keys (value=0x18e0500, certs=0x18e1620, context=0x18a0430) at collector.c:232 #2 _hx509_collector_collect_certs (context=context@entry=0x18a0430, c=0x18e0b30, ret_certs=ret_certs@entry=0x18e0438) at collector.c:275 #3 0x7f5c0f315669 in p11_list_keys (certs=0x18e0438, lock=, session=26085952, slot=0x18e0410, p=0x18a3740, context=0x18a0430) at ks_p11.c:804 #4 p11_init_slot (context=context@entry=0x18a0430, p=p@entry=0x18a3740, lock=lock@entry=0x18a36a0, id=, num=, slot=0x18e0410) at ks_p11.c:363 #5 0x7f5c0f315963 in p11_init (context=0x18a0430, certs=, data=0x1889f70, flags=, residue=, lock=0x18a36a0) at ks_p11.c:933 #6 0x7f5c0f311b10 in hx509_certs_init (context=0x18a0430, name=name@entry=0x1867910 "PKCS11:/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so", flags=flags@entry=0, lock=0x18a36a0, certs=certs@entry=0x18a3668) at keyset.c:159 #7 0x7f5c0fc72f0f in _krb5_pk_load_id (context=context@entry=0x18a0240, ret_id=0x18a3580, user_id=user_id@entry=0x1867910 "PKCS11:/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so", anchor_id=anchor_id@entry=0x18a3440 "FILE:/etc/trinity/ldap/tde-ca/public/odin.starlink.edu.ldap.crt", chain_list=0x0, revoke_list=0x1889ba0, prompter=0x7f5c0feb30d0, prompter_data=0x1867740, password=0x0) at pkinit.c:1882 #8 0x7f5c0fc74b07 in krb5_get_init_creds_opt_set_pkinit (context=0x18a0240, opt=0x1867810, principal=0x1867880, user_id=0x1867910 "PKCS11:/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so", x509_anchors=0x18a3440 "FILE:/etc/trinity/ldap/tde-ca/public/odin.starlink.edu.ldap.crt", pool=, pki_revoke=, flags=0, prompter=0x7f5c0feb30d0, prompter_data=0x1867740, password=0x0) at pkinit.c:2394 #9 0x7f5c0feb0e49 in ?? () from /lib/x86_64-linux-gnu/security/pam_krb5.so #10 0x7f5c0feb175b in ?? () from /lib/x86_64-linux-gnu/security/pam_krb5.so #11 0x7f5c0feb36b7 in pam_sm_authenticate () from /lib/x86_64-linux-gnu/security/pam_krb5.so #12 0x7f5c123c80a6 in ?? () from /lib/x86_64-linux-gnu/libpam.so.0 #13 0x7f5c123c781d in pam_authenticate () from /lib/x86_64-linux-gnu/libpam.so.0 #14 0x004031b3 in ?? () #15 0x7f5c11c18700 in __libc_start_main (main=0x402a90, argc=3, argv=0x7ffd37fe93f8, init=, fini=, rtld_fini=, stack_end=0x7ffd37fe93e8) at ../csu/libc-start.c:291 #16 0x00404200 in ?? ()
Bug#808246: binutils bug report
I went ahead and filed a bug report upstream at: https://sourceware.org/bugzilla/show_bug.cgi?id=19421 Hopefully this helps prod some action on this serious bug... P,S. I completely agree with the previous commenter; this bug is unreal. This is 2015, not 1998...core libraries breaking modern platforms like this is just not acceptable.
Bug#808246: Stretch fails to install on OpenPOWER systems
Package: linux-image-4.2.0-1-powerpc64le Version: 4.2.6-3 Severity: grave Using the latest debian-installer netboot kernel to install on an OpenPOWER server; install proceeds normally until disk detection when it fails stating no disks can be found. dmesg shows a slew of errors similar to the following: fat: no symbol version for TOC. fat: Unknown symbol TOC. (err -22) ext4: no symbol version for TOC. ext4: Unknown symbol TOC. (err -22) ahci: no symbol version for TOC. ahci: Unknown symbol TOC. (err -22) Forcibly loading the modules (modprobe --force ext4, etc.) and resuming the installer allows the system to install normally, however the same issue shows up on reboot. This causes a boot failure that cannot be overridden as the modprobe version in the initramfs does not support the force parameter. Jessie installs normally on the same system.
