[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
** Description changed:
I'm assuming the image being used for these deploys is 20230417 or
20230417.1 based on the fact that I saw a 6.2 kernel being used which I
don't believe was part of the 20230319 serial. I don't have access to
the maas server, so I can't directly check any log files.
MAAS Version: 3.3.2
Here's where the serial log indicates it can't download the squashfs. The
full log is attached as scobee-lunar-no-squashfs.log (there are some other
console message intermixed):
no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf
/run/net6
-*.conf
::
root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi
date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes,
fsverity
=yes
shfs
:: mount_squash downloading
http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0
4/lunar/candidate/squashfs to /root.tmp.img
Connecting to 10.229.32.21:5248 (10.229.32.21:5248)
wget: can't connect to remote host (10.229.32.21): Network is unreachable
:: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp'
mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory
done.
+ Still gathering logs and info and will update as I go.
- Still gathering logs and info and will update as I go.
+
+
+ Kernel Bug / Apparmor
+ reproducer
+
+ $ wget
https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-kernel
+ $ wget
https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-initrd
+ $ qemu-system-x86_64 -nographic -m 2G -kernel ./boot-kernel -initrd
./boot-initrd -append 'console=ttyS0 break=modules apparmor=0'
+
+
+ #start the VM
+
+ Starting systemd-udevd version 252.5-2ubuntu3
+ Spawning shell within the initramfs
+
+
+ BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1) built-in shell (ash)
+ Enter 'help' for a list of built-in commands.
+
+ (initramfs) udevadm info --export-db
+ Failed to set death signal: Invalid argument
+
+ Observe that udevadm fails to setup death signal, with in systemd code
+ is this
+
+
https://github.com/systemd/systemd/blob/08c2f9c626e0f0052d505b1b7e52f335c0fbfa1d/src/basic/process-
+ util.c#L1252
+
+ if (flags & (FORK_DEATHSIG|FORK_DEATHSIG_SIGINT))
+ if (prctl(PR_SET_PDEATHSIG, (flags & FORK_DEATHSIG_SIGINT) ?
SIGINT : SIGTERM) < 0) {
+ log_full_errno(prio, errno, "Failed to set death
signal: %m");
+ _exit(EXIT_FAILURE);
+ }
+
+
+
+ MAAS bug
+ Why is maas setting `apparmor=0` ? Ubuntu shouldn't be used without apparmor.
Even for deployment and commisioning.
** Changed in: linux (Ubuntu)
Status: Incomplete => Triaged
** Changed in: maas-images
Status: Incomplete => Invalid
** Changed in: systemd (Ubuntu)
Status: New => Invalid
** Also affects: maas
Importance: Undecided
Status: New
** Summary changed:
- Unable to deploy hosts with lunar images after 20230319 - fails to connect
and download squashfs
+ udev fails to make prctl() syscall with apparmor=0 (as used by maas by
default)
** Description changed:
I'm assuming the image being used for these deploys is 20230417 or
20230417.1 based on the fact that I saw a 6.2 kernel being used which I
don't believe was part of the 20230319 serial. I don't have access to
the maas server, so I can't directly check any log files.
MAAS Version: 3.3.2
Here's where the serial log indicates it can't download the squashfs. The
full log is attached as scobee-lunar-no-squashfs.log (there are some other
console message intermixed):
no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf
/run/net6
-*.conf
::
root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi
date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes,
fsverity
=yes
shfs
:: mount_squash downloading
http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0
4/lunar/candidate/squashfs to /root.tmp.img
Connecting to 10.229.32.21:5248 (10.229.32.21:5248)
wget: can't connect to remote host (10.229.32.21): Network is unreachable
:: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp'
mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory
done.
Still gathering logs and info and will update as I go.
-
Kernel Bug / Apparmor
reproducer
$ wget
https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-kernel
$ wget
https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-initrd
$ qemu-system-x86_64 -nographic -m 2G -kernel ./boot-kernel -initrd
./boot-initrd -append 'console=ttyS0 break=modules apparmor=0'
-
#start the VM
Starting systemd-udevd version 252.5-2ubuntu3
Spawning shell within the initramfs
-
BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1)
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
horay i managed to reproduce it locally. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in linux package in Ubuntu: New Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
We discussed fixes to the maas bootloader builds and as a result, we have a new maas bootloader which should have the latest shim and grub: http://images.maas.io/ephemeral-v3/candidate/bootloaders/uefi/amd64/20230420.0/ I manually copied these to my test maas, it did not resolve the issue (or have any other visible change). Paolo built a kernel to try and debug this from the prctl interface. With this (and udev with -debug), I see these two additional console log entries: [ 18.029479] __do_sys_prctl::2375 [ 18.042840] __do_sys_prctl::2380 There are lots of these, but only the same two lines. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
I am annoyed that i cannot reproduce this locally outside of MAAS. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
Kernel side:
kernel/sys.c::SYSCALL_DEFINE5(prctl, int, option,...):
{
struct task_struct *me = current;
unsigned char comm[sizeof(me->comm)];
long error;
error = security_task_prctl(option, arg2, arg3, arg4, arg5);
if (error != -ENOSYS)
return error;
error = 0;
switch (option) {
case PR_SET_PDEATHSIG:
if (!valid_signal(arg2)) {
error = -EINVAL;
break;
}
me->pdeath_signal = arg2;
break;
...
return error;
}
and include/linux/signal.h::valid_signal():
/* Test if 'sig' is valid signal. Use this instead of testing _NSIG directly */
static inline int valid_signal(unsigned long sig)
{
return sig <= _NSIG ? 1 : 0;
}
and arch/x86/include/asm/signal.h:#define _NSIG 64
I wonder about security_task_prctl() though.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2016908
Title:
Unable to deploy hosts with lunar images after 20230319 - fails to
connect and download squashfs
Status in maas-images:
Incomplete
Status in systemd package in Ubuntu:
New
Bug description:
I'm assuming the image being used for these deploys is 20230417 or
20230417.1 based on the fact that I saw a 6.2 kernel being used which
I don't believe was part of the 20230319 serial. I don't have access
to the maas server, so I can't directly check any log files.
MAAS Version: 3.3.2
Here's where the serial log indicates it can't download the squashfs. The
full log is attached as scobee-lunar-no-squashfs.log (there are some other
console message intermixed):
no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf
/run/net6
-*.conf
::
root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi
date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes,
fsverity
=yes
shfs
:: mount_squash downloading
http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0
4/lunar/candidate/squashfs to /root.tmp.img
Connecting to 10.229.32.21:5248 (10.229.32.21:5248)
wget: can't connect to remote host (10.229.32.21): Network is unreachable
:: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp'
mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory
done.
Still gathering logs and info and will update as I go.
To manage notifications about this bug go to:
https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
** Also affects: systemd (Ubuntu) Importance: Undecided Status: New ** Attachment added: "lunar-udev-debug.log" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2016908/+attachment/5665324/+files/lunar-udev-debug.log -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
