Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Mathias Gibbens wrote on 06/06/2023 at 04:06:14+0200: > [[PGP Signed Part:No public key for 29EEE2D6ECF442F9 created at > 2023-06-06T04:06:14+0200 using RSA]] > On Thu, 2023-06-01 at 18:58 +0200, Pierre-Elliott Bécue wrote: >> > I should have time this weekend when I can spin up a qemu vm to >> > test in, if we're not able to get a root cause identified before >> > then. > > I did try to reproduce the issue over the weekend with qemu. Using > qemu-user-static and systemd-nspawn was insufficient due to lxcfs > needing proper access to the fuse kernel api. After trying and failing > to bootstrap an armhf instance by hand, I grabbed a raspberry pi 2 > bookworm image from raspi.d.n, and got it running under qemu-system- > arm. Within that environment, lxcfs appeared to work correctly > (/var/lib/lxcfs/proc/cpuinfo was correct, no obvious errors or warnings > noticed). I didn't spin up a full lxc container instance within that > environment. > >> I can probably grant you privileges on abel as soon as I get an ack >> from my fellow DSA mates. > > If it's possible to get temporary permissions on abel, that would be > useful. > > One other thing to double check on ci-worker-armhf-01 would be the > contents of /var/lib/lxcfs/proc/cpuinfo, so we can see what lxcfs is > doing from the host side. Unfortunately my teammates have some concerns and I don't have the time for the debate now. What I can do is run any required tests myself. Could you tell me what you wanted to try so that I can try it out for you and report back with the results? -- PEB
Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Hi, On 01-06-2023 13:28, Mathias Gibbens wrote: Some other things we can look at -- are there any errors/warnings in the lxcfs service journal and/or the system's dmesg that is running lxcfs? It might also be useful to start lxcfs with debugging (`-d`) if there's nothing being logged about populating /proc/cpuinfo. This maybe? Jun 06 17:35:55 ci-worker-armhf-01 lxcfs[686]: ../src/proc_cpuview.c: 1055: proc_cpuinfo_read: Write to cache was truncated On 06-06-2023 04:06, Mathias Gibbens wrote: One other thing to double check on ci-worker-armhf-01 would be the contents of /var/lib/lxcfs/proc/cpuinfo, so we can see what lxcfs is doing from the host side. debian@ci-worker-armhf-01:~$ cat /var/lib/lxcfs/proc/cpuinfo processor : 0 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 processor : 1 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 [ etc ] Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
On Thu, 2023-06-01 at 18:58 +0200, Pierre-Elliott Bécue wrote: > > I should have time this weekend when I can spin up a qemu vm to > > test in, if we're not able to get a root cause identified before > > then. I did try to reproduce the issue over the weekend with qemu. Using qemu-user-static and systemd-nspawn was insufficient due to lxcfs needing proper access to the fuse kernel api. After trying and failing to bootstrap an armhf instance by hand, I grabbed a raspberry pi 2 bookworm image from raspi.d.n, and got it running under qemu-system- arm. Within that environment, lxcfs appeared to work correctly (/var/lib/lxcfs/proc/cpuinfo was correct, no obvious errors or warnings noticed). I didn't spin up a full lxc container instance within that environment. > I can probably grant you privileges on abel as soon as I get an ack > from my fellow DSA mates. If it's possible to get temporary permissions on abel, that would be useful. One other thing to double check on ci-worker-armhf-01 would be the contents of /var/lib/lxcfs/proc/cpuinfo, so we can see what lxcfs is doing from the host side. Mathias signature.asc Description: This is a digitally signed message part
Bug#1036818: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
De : Mathias Gibbens À : Paul Gevers ; 1036...@bugs.debian.org Cc : Pierre-Elliott Bécue ; Evgeni Golov ; Jochen Sprickerhof Date : 1 juin 2023 13:33:33 Objet : Re: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat > On Mon, 2023-05-29 at 20:03 +0200, Paul Gevers wrote: >> We have three layers here. The bare metal is arm64. It hosts several >> arm* QEMU VM. Inside the VM we run the lxc. I put the output of all >> three at the bottom. *Maybe* it's relevant that the bare metal still >> runs bullseye while the VM's run bookworm. I'll also share the >> content for an arm64 host (which is a VM where I don't have access to >> the bare metal.) > > [snip] > >> # generating the container and showing inside >> debian@ci-worker-armhf-01:~$ sudo lxc-copy -N elbrus >> autopkgtest-unstable-armhf && sudo lxc-start elbrus && sudo lxc- >> attach elbrus >> root@elbrus:/# cat /proc/cpuinfo >> root@elbrus:/# ls -al /proc/cpuinfo >> -r--r--r-- 1 root root 3878 May 29 17:48 /proc/cpuinfo > > Yeah, that's definitely not right! I don't currently have an > armel/armhf system to test on, but did try using the abel.d.o > porterbox. lxcfs requires elevated permissions to start, which I don't > have on that box. > > Some other things we can look at -- are there any errors/warnings in > the lxcfs service journal and/or the system's dmesg that is running > lxcfs? It might also be useful to start lxcfs with debugging (`-d`) if > there's nothing being logged about populating /proc/cpuinfo. > > I should have time this weekend when I can spin up a qemu vm to test > in, if we're not able to get a root cause identified before then. > > Mathias I can probably grant you privileges on abel as soon as I get an ack from my fellow DSA mates. -- Pierre-Elliott Bécue
Bug#1036818: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
On Mon, 2023-05-29 at 20:03 +0200, Paul Gevers wrote: > We have three layers here. The bare metal is arm64. It hosts several > arm* QEMU VM. Inside the VM we run the lxc. I put the output of all > three at the bottom. *Maybe* it's relevant that the bare metal still > runs bullseye while the VM's run bookworm. I'll also share the > content for an arm64 host (which is a VM where I don't have access to > the bare metal.) [snip] > # generating the container and showing inside > debian@ci-worker-armhf-01:~$ sudo lxc-copy -N elbrus > autopkgtest-unstable-armhf && sudo lxc-start elbrus && sudo lxc- > attach elbrus > root@elbrus:/# cat /proc/cpuinfo > root@elbrus:/# ls -al /proc/cpuinfo > -r--r--r-- 1 root root 3878 May 29 17:48 /proc/cpuinfo Yeah, that's definitely not right! I don't currently have an armel/armhf system to test on, but did try using the abel.d.o porterbox. lxcfs requires elevated permissions to start, which I don't have on that box. Some other things we can look at -- are there any errors/warnings in the lxcfs service journal and/or the system's dmesg that is running lxcfs? It might also be useful to start lxcfs with debugging (`-d`) if there's nothing being logged about populating /proc/cpuinfo. I should have time this weekend when I can spin up a qemu vm to test in, if we're not able to get a root cause identified before then. Mathias signature.asc Description: This is a digitally signed message part
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Hi, On 29-05-2023 15:58, Mathias Gibbens wrote: On Mon, 2023-05-29 at 14:26 +0200, Pierre-Elliott Bécue wrote: On Mon, 2023-05-29 at 07:37 +0200, Paul Gevers wrote: Is there more information I can get for you from one of the effected architectures? Can you grab /proc/cpuinfo from a physical CI host as well as from within a container for the armel, armhf, and arm64 architectures? That will let us see what difference(s) lxcfs is presenting and might give a clue for the root issue. Since only the 32bit arm architectures appear to be affected, I'm also curious to see what the arm64 cpuinfo is. We have three layers here. The bare metal is arm64. It hosts several arm* QEMU VM. Inside the VM we run the lxc. I put the output of all three at the bottom. *Maybe* it's relevant that the bare metal still runs bullseye while the VM's run bookworm. I'll also share the content for an arm64 host (which is a VM where I don't have access to the bare metal.) Paul # The armhf VM debian@ci-worker-armhf-01:~$ cat /proc/cpuinfo processor : 0 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 processor : 1 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 [... repeat until ...] processor : 15 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 # generating the container and showing inside debian@ci-worker-armhf-01:~$ sudo lxc-copy -N elbrus autopkgtest-unstable-armhf && sudo lxc-start elbrus && sudo lxc-attach elbrus root@elbrus:/# cat /proc/cpuinfo root@elbrus:/# ls -al /proc/cpuinfo -r--r--r-- 1 root root 3878 May 29 17:48 /proc/cpuinfo # the bare metal machine used for armhf and armel VM's root@altra:~# cat /proc/cpuinfo processor : 0 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 processor : 1 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 [ repeat until ...] processor : 159 BogoMIPS: 50.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x3 CPU part: 0xd0c CPU revision: 1 # the arm64 VM (at Huawei) root@ci-worker-arm64-02:~# cat /proc/cpuinfo processor : 0 BogoMIPS: 200.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm CPU implementer : 0x48 CPU architecture: 8 CPU variant : 0x1 CPU part: 0xd01 CPU revision: 0 processor : 1 BogoMIPS: 200.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm CPU implementer : 0x48 CPU architecture: 8 CPU variant : 0x1 CPU part: 0xd01 CPU revision: 0 processor : 2 BogoMIPS: 200.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm CPU implementer : 0x48 CPU architecture: 8 CPU variant : 0x1 CPU part: 0xd01 CPU revision: 0 processor : 3 BogoMIPS: 200.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm CPU implementer : 0x48 CPU architecture: 8 CPU variant : 0x1 CPU part: 0xd01 CPU revision: 0 root@ci-worker-arm64-02:~# lxc-copy autopkgtest-unstable-arm64 -N elbrus && lxc-start elbrus && lxc-attach elbrus root@elbrus:~# cat /proc/cpuinfo processor : 0 BogoMIPS: 200.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm CPU implementer : 0x48 CPU architecture: 8 CPU variant : 0x1 CPU part: 0xd01 CPU revision: 0 processor : 1 BogoMIPS: 200.00 Features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm CPU imp
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Control: notforwarded -1 On Mon, 2023-05-29 at 14:26 +0200, Pierre-Elliott Bécue wrote: > On Mon, 2023-05-29 at 07:37 +0200, Paul Gevers wrote: > > Hi, > > > > [reducing the people in CC, hope I didn't drop too many and those > > still there are interested] > > > > On Mon, 29 May 2023 00:14:10 + Mathias Gibbens > > wrote: > > > What version of lxcfs is currently installed on the > > > ci.debian.net machines? I'm guessing from the kernel version > > > they've been upgraded to bookworm, so lxcfs should be at 5.0.3-1, > > > but I'd like to clarify that. > > > > I just ran $(apt-cache show lxcfs | grep Version) on all the worker > > hosts and indeed they all run 5.0.3-1. Thanks for confirming that -- for the time being I've unforwarded this bug from upstream issue #553 as it looks like this is a new problem. > > > > > lxcfs 5.0.3-1 does indeed include the referenced fix for > > > upstream issue #553, and has been in testing since 2023-01-22. If > > > the CI machines have that version installed, then I'd lean > > > towards a related but distinct issue than the one identified at > > > the moment. > > > > Is there more information I can get for you from one of the > > effected architectures? Can you grab /proc/cpuinfo from a physical CI host as well as from within a container for the armel, armhf, and arm64 architectures? That will let us see what difference(s) lxcfs is presenting and might give a clue for the root issue. Since only the 32bit arm architectures appear to be affected, I'm also curious to see what the arm64 cpuinfo is. > > > > Paul > > PS: I missed former messages due to a minor mistake in my address, > > but I'm now subscribed. > I think you should keep gibmat in the Cc list. :) +1 :) Mathias signature.asc Description: This is a digitally signed message part
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
De : Paul Gevers À : 1036...@bugs.debian.org Cc : Evgeni Golov ; Pierre-Elliott Bécue ; Jochen Sprickerhof Date : 29 mai 2023 07:41:30 Objet : Re: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat > Hi, > > [reducing the people in CC, hope I didn't drop too many and those still there > are interested] > > On Mon, 29 May 2023 00:14:10 + Mathias Gibbens wrote: >> What version of lxcfs is currently installed on the ci.debian.net >> machines? I'm guessing from the kernel version they've been upgraded to >> bookworm, so lxcfs should be at 5.0.3-1, but I'd like to clarify that. > > I just ran $(apt-cache show lxcfs | grep Version) on all the worker hosts and > indeed they all run 5.0.3-1. > >> lxcfs 5.0.3-1 does indeed include the referenced fix for upstream >> issue #553, and has been in testing since 2023-01-22. If the CI >> machines have that version installed, then I'd lean towards a related >> but distinct issue than the one identified at the moment. > > Is there more information I can get for you from one of the effected > architectures? > > Paul > PS: I missed former messages due to a minor mistake in my address, but I'm > now subscribed. I think you should keep gibmat in the Cc list. :) -- Pierre-Elliott Bécue
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Hi, [reducing the people in CC, hope I didn't drop too many and those still there are interested] On Mon, 29 May 2023 00:14:10 + Mathias Gibbens wrote: What version of lxcfs is currently installed on the ci.debian.net machines? I'm guessing from the kernel version they've been upgraded to bookworm, so lxcfs should be at 5.0.3-1, but I'd like to clarify that. I just ran $(apt-cache show lxcfs | grep Version) on all the worker hosts and indeed they all run 5.0.3-1. lxcfs 5.0.3-1 does indeed include the referenced fix for upstream issue #553, and has been in testing since 2023-01-22. If the CI machines have that version installed, then I'd lean towards a related but distinct issue than the one identified at the moment. Is there more information I can get for you from one of the effected architectures? Paul PS: I missed former messages due to a minor mistake in my address, but I'm now subscribed. OpenPGP_signature Description: OpenPGP digital signature
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
What version of lxcfs is currently installed on the ci.debian.net machines? I'm guessing from the kernel version they've been upgraded to bookworm, so lxcfs should be at 5.0.3-1, but I'd like to clarify that. lxcfs 5.0.3-1 does indeed include the referenced fix for upstream issue #553, and has been in testing since 2023-01-22. If the CI machines have that version installed, then I'd lean towards a related but distinct issue than the one identified at the moment. If not, let's upgrade lxcfs and see if that fixes the issue. Mathias signature.asc Description: This is a digitally signed message part
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Control: reassign -1 src:lxcfs 5.0.3-1 Control: forwarded -1 https://github.com/lxc/lxcfs/issues/553 Control: affects -1 src:mariadb Hi, On Sat, May 27, 2023 at 11:51:26AM +0200, Salvatore Bonaccorso wrote: > Hi, > > On Sat, May 27, 2023 at 11:50:06AM +0200, Salvatore Bonaccorso wrote: > > Hi Helge, hi Otto, > > > > On Sat, May 27, 2023 at 09:26:06AM +0200, Helge Deller wrote: > > > Just wondering / guessing: > > > > > > Are the ARM machines on ci.debian.net (ci-worker-arm??-??) > > > physical machines, or are they running on qemu-user VMs? > > > > > > If they run qemu, this bug report > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653 > > > might be similiar. > > > > > > If so, then qemu probably needs fixing of the output of /proc/cpuinfo > > > for ARM, e.g. like this: > > > https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a > > > > The suspect is that /proc/cpuinfo is empty or not readable, and this > > seems to be a problem with lxcfs after mentioning the issue today to > > Paul and Jochen. > > > > Jochen, understanding you correctly there is already an upstream fix > > which is supposed to addres the issue? > > The upstream issue should be: https://github.com/lxc/lxcfs/issues/553 Now reassigning to the lxcfs package. lxcfs maintainers, can you please adjust the severity as needed. It affects at least mariadb's autopkgtests. Otto, spaking of the issue, I guess Paul will agree, that you can ignore it for now for the mariadb upload to unstable. Regards, Salvatore
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Hi Helge, hi Otto, On Sat, May 27, 2023 at 09:26:06AM +0200, Helge Deller wrote: > Just wondering / guessing: > > Are the ARM machines on ci.debian.net (ci-worker-arm??-??) > physical machines, or are they running on qemu-user VMs? > > If they run qemu, this bug report > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653 > might be similiar. > > If so, then qemu probably needs fixing of the output of /proc/cpuinfo > for ARM, e.g. like this: > https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a The suspect is that /proc/cpuinfo is empty or not readable, and this seems to be a problem with lxcfs after mentioning the issue today to Paul and Jochen. Jochen, understanding you correctly there is already an upstream fix which is supposed to addres the issue? Regards, Salvatore
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Hi, On Sat, May 27, 2023 at 11:50:06AM +0200, Salvatore Bonaccorso wrote: > Hi Helge, hi Otto, > > On Sat, May 27, 2023 at 09:26:06AM +0200, Helge Deller wrote: > > Just wondering / guessing: > > > > Are the ARM machines on ci.debian.net (ci-worker-arm??-??) > > physical machines, or are they running on qemu-user VMs? > > > > If they run qemu, this bug report > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653 > > might be similiar. > > > > If so, then qemu probably needs fixing of the output of /proc/cpuinfo > > for ARM, e.g. like this: > > https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a > > The suspect is that /proc/cpuinfo is empty or not readable, and this > seems to be a problem with lxcfs after mentioning the issue today to > Paul and Jochen. > > Jochen, understanding you correctly there is already an upstream fix > which is supposed to addres the issue? The upstream issue should be: https://github.com/lxc/lxcfs/issues/553 Regards, Salvatore
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Just wondering / guessing: Are the ARM machines on ci.debian.net (ci-worker-arm??-??) physical machines, or are they running on qemu-user VMs? If they run qemu, this bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653 might be similiar. If so, then qemu probably needs fixing of the output of /proc/cpuinfo for ARM, e.g. like this: https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a Helge
Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat
Package: linux Version: 6.1.0 Hi! I noticed that the autopkgtests on Debian on armhf and armel that run the mariadb-test-run have been failing since the Linux kernel was updated from 5.10.0 to 6.1.0. The failure is due to a Perl module not being able to get from /proc/cpu the number of processors: Last passing one: 2023-04-28 https://ci.debian.net/data/autopkgtest/unstable/armel/m/mariadb/33218554/log.gz kernel: Linux 5.10.0-21-arm64 #1 SMP Debian 5.10.162-1 (2023-01-21) perl 5.36.0-7 libdbi-perl armel 1.643-4 libconfig-inifiles-perl all 3.03-2 First failing one: 2023-05-05 https://ci.debian.net/data/autopkgtest/unstable/armel/m/mariadb/33379866/log.gz kernel: Linux 6.1.0-7-arm64 #1 SMP Debian 6.1.20-2 (2023-04-08) perl 5.36.0-7 libdbi-perl armhf 1.643-4 libconfig-inifiles-perl all 3.03-2 Error: starting mysql-test-tun.pl... Logging: ./mysql-test-run.pl --force --testcase-timeout=120 --suite-timeout=540 --retry=3 -- ... Collecting tests... Installing system database... Can't use an undefined value as an ARRAY reference at lib/My/SysInfo.pm line 166. This line 166 in src:mariadb/mysql-test/lib/My/SysInfo.pm has: # Return the number of cpus found sub num_cpus { my ($self)= @_; return int(@{$self->{cpus}}) or confess "INTERNAL ERROR: No cpus in list"; } The cpus is initialized to be an empty list on the line 119: 118 my $self= bless { 119cpus => (), 120 }, $class; Then it tries to fill it from /proc/cpuinfo (line 67) and `kstat` (line 95). If nothing worked it'll create one dummy cpu: 145 push(@{$self->{cpus}}, 146 { 147 bogomips => DEFAULT_BOGO_MIPS, 148 model_name => "unknown", 149 }); See more discussion from MariaDB devs: https://lists.launchpad.net/maria-developers/msg13356.html Thus the primary suspect here is the kernel upgrade. Perl versions have not changed. This only happens on armel/armhf, other archs are fine. Reproducing the environment on ci.debian.net / ci-worker-arm??-?? to study how /proc/cpu etc looks like, so filing this against the Linux package is somewhat of a guess, but at least we get a Bug# to reference for further research.