[Bug 2081172] Re: dracut generated initramfs doesn't contain necessary files for iscsi boot
Yes, it is installed. This is from the instance that I used to create the image that later on boots from iscsi: ubuntu@oracular-dracut:~$ sudo dpkg -l dracut-network Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==--- ii dracut-network 103-1ubuntu2 all dracut is an event driven initramfs infrastructure (network modules) I noticed that, when regenerating the initramfs, we get an error that says: dracut[I]: *** Including module: iscsi *** Failed to enable unit: Unit iscsiuio.socket does not exist Not sure if its related. Full output: https://pastebin.ubuntu.com/p/HFmCD9s4WZ/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2081172 Title: dracut generated initramfs doesn't contain necessary files for iscsi boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dracut/+bug/2081172/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2081172] [NEW] dracut generated initramfs doesn't contain necessary files for iscsi boot
Public bug reported: When running Ubuntu 24.10 (Oracular Oriole) and switching from initramfs to dracut (by installing dracut packages and following instructions in /usr/share/doc/dracut-core/README.Debian), the initramfs generated by dracut won't include the necessary files to allow an instance to boot from iscsi. The following packages were installed: ii dracut 103-1ubuntu2 all Initramfs generator using udev ii dracut-core103-1ubuntu2 amd64dracut is an event driven initramfs infrastructure (core tools) ii dracut-install 103-1ubuntu2 amd64dracut is an event driven initramfs infrastructure (dracut-install) ii dracut-network 103-1ubuntu2 all dracut is an event driven initramfs infrastructure (network modules) When trying to boot from iscsi by adding the following cmdline to grub (booting from ISCSI in an Oracle Cloud instance): BOOT_IMAGE=/vmlinuz-6.8.0-1005-oracle root=UUID=f692dbf6-c1a4-4a31-8585-2571fbbd2f7e ro console=tty1 console=ttyS0 nvme.shutdown_timeout=10 libiscsi.debug_libiscsi_eh=1 crash_kexec_post_notifiers rd.luks=0 rd.md=0 rd.dm=0 rd.net.timeout.carrier=5 rd.iscsi.param=node.session.timeo.replacement_timeout=6000 net.ifnames=1 ipmi_si.tryacpi=0 ipmi_si.trydmi=0 libiscsi.debug_libiscsi_eh=1 loglevel=4 rd.net.timeout.dhcp=10 ip=dhcp netroot=iscsi:169.254.0.2:::1:iqn.2015-02.oracle.boot:uefi The boot fails with: [ OK ] Reached target network-online.target - Network is Online. Starting iscsid.service - iSCSI initiator daemon (iscsid)... [FAILED] Failed to start iscsid.service - iSCSI initiator daemon (iscsid). See 'systemctl status iscsid.service' for details. Starting iscsid.service - iSCSI initiator daemon (iscsid)... [FAILED] Failed to start iscsid.service - iSCSI initiator daemon (iscsid). See 'systemctl status iscsid.service' for details. Starting iscsid.service - iSCSI initiator daemon (iscsid)... [FAILED] Failed to start iscsid.service - iSCSI initiator daemon (iscsid). See 'systemctl status iscsid.service' for details. Starting iscsid.service - iSCSI initiator daemon (iscsid)... [FAILED] Failed to start iscsid.service - iSCSI initiator daemon (iscsid). See 'systemctl status iscsid.service' for details. Starting iscsid.service - iSCSI initiator daemon (iscsid)... [FAILED] Failed to start iscsid.service - iSCSI initiator daemon (iscsid). See 'systemctl status iscsid.service' for details. [FAILED] Failed to start iscsid.service - iSCSI initiator daemon (iscsid). See 'systemctl status iscsid.service' for details. [6.891677] dracut-initqueue[582]: iscsiadm: read error (-1/104), daemon died? [ OK ] Listening on iscsid.socket - Open-iSCSI iscsid Socket. [FAILED] Failed to start iscsid.service - iSCSI initiator daemon (iscsid). See 'systemctl status iscsid.service' for details. [7.909211] dracut-initqueue[582]: iscsiadm: read error (-1/104), daemon died? [7.910096] dracut-initqueue[582]: iscsiadm: Cannot perform discovery. Initiatorname required. [7.911164] dracut-initqueue[582]: iscsiadm: Could not perform SendTargets discovery: could not communicate to iscsid [7.913629] dracut-initqueue[521]: Warning: Target discovery to 169.254.0.2:3260 failed Eventually the instance will drop to an emergency shell and we can check the error in details: # journalctl -u iscsid --no-pager Sep 18 19:58:40 localhost systemd[1]: Starting iscsid.service - iSCSI initiator daemon (iscsid)... Sep 18 19:58:40 localhost (hecks.sh)[584]: iscsid.service: Unable to locate executable '/usr/lib/open-iscsi/startup-checks.sh': No such file or directory Sep 18 19:58:40 localhost (hecks.sh)[584]: iscsid.service: Failed at step EXEC spawning /usr/lib/open-iscsi/startup-checks.sh: No such file or directory Sep 18 19:58:40 localhost systemd[1]: iscsid.service: Control process exited, code=exited, status=203/EXEC Sep 18 19:58:40 localhost systemd[1]: iscsid.service: Failed with result 'exit-code'. Sep 18 19:58:40 localhost systemd[1]: Failed to start iscsid.service - iSCSI initiator daemon (iscsid). Extracting the initramfs generated by dracut, we can see that /usr/lib/open-iscsi/startup-checks.sh is not included: ubuntu@oracular-dracut:~/initrd$ ls -l usr/lib/ total 636 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 22:13 dracut -rwxr-xr-x 1 ubuntu ubuntu 8177 Jul 25 03:45 dracut-crypt-lib.sh -rwxr-xr-x 1 ubuntu ubuntu 4078 Jul 25 03:45 dracut-dev-lib.sh -rwxr-xr-x 1 ubuntu ubuntu 29345 Jul 25 03:45 dracut-lib.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 22:13 firmware -rwxr-xr-x 1 ubuntu ubuntu 6552 Dec 6 2023 fs-lib.sh -rw-r--r-- 1 ubuntu ubuntu510 Sep 6 14:20 initrd-release -rw-r--r-- 1 ubuntu ubuntu 14648 Jul 23 13:41 libmpathcmd.so.0 -rw-r--r-- 1 ubuntu ubuntu 39224 Jul
[Bug 2064657] Re: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr
Based on discussions at https://github.com/nfs-ganesha/nfs- ganesha/issues/1123, it seems this is something released with nfs- ganesha V4, so IIUC, this is something that will only be available to an Openstack + Manila + Ceph + NFS Ganesha in Noble Numbat, since it offers nfs-ganesha 4.3-8ubuntu1 ** Changed in: nfs-ganesha (Ubuntu) Status: Incomplete => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2064657 Title: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-ganesha/+bug/2064657/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2064657] Re: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr
This is also being discussed upstream: https://github.com/nfs- ganesha/nfs-ganesha/issues/1123 ** Bug watch added: github.com/nfs-ganesha/nfs-ganesha/issues #1123 https://github.com/nfs-ganesha/nfs-ganesha/issues/1123 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2064657 Title: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-ganesha/+bug/2064657/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2064657] Re: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr
Hi Peter, The NFS client is a Jammy VM running nfs-common 1:2.6.1-1ubuntu1.2 and kernel 5.15.0-102-generic. I'm running a sosreport and will attach it here soon. Regards, Fabio -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2064657 Title: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-ganesha/+bug/2064657/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2064657] Re: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr
** Attachment added: "sosreport-jammy-130612-2024-05-22-dnjwtoi.tar.xz" https://bugs.launchpad.net/ubuntu/+source/nfs-ganesha/+bug/2064657/+attachment/5781410/+files/sosreport-jammy-130612-2024-05-22-dnjwtoi.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2064657 Title: Add support for CEPH ceph.file.layout and ceph.dir.layout xattr To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-ganesha/+bug/2064657/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054343] Re: arm64 build of gcc-10 10.5.0-3ubuntu1 still broken (CVE-2023-4039 still open)
** Tags added: rls-ff-incoming rls-jj-incoming rls-mm-incoming rls-nn- incoming -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054343 Title: arm64 build of gcc-10 10.5.0-3ubuntu1 still broken (CVE-2023-4039 still open) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/gcc-10/+bug/2054343/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056187] Re: fails to configure BOOTIF when using iscsi
It seems 0.142ubuntu19 would be exposed to the bug. To test whether or not my use case would be affected by it, I've booted a Jammy instance with initramfs-tools 0.142ubuntu19, and I can't hit the problem. It is booting well through iscsi. My cmdline is as below, so it might not be exposed to the issue (and hence, please ignore my comment above as a valid test for the patch, given that I wasn't exposed to it to begin with): BOOT_IMAGE=/boot/vmlinuz-6.5.0-1018-oracle root=UUID=2792ceda-655f-4995-bf29-6a714f9a200b ro console=tty1 console=ttyS0 nvme.shutdown_timeout=10 libiscsi.debug_libiscsi_eh=1 crash_kexec_post_notifiers -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056187 Title: fails to configure BOOTIF when using iscsi To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2056187/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056460] [NEW] cloud-init in degraded state on Oracle instance with "Invalid network-config provided"
Public bug reported: When booting a Noble Numbat (Ubuntu 24.04) instance on Oracle Cloud, clout-init ends up in a degraded state reporting "Invalid network-config provided". Details are displayed below: ubuntu@fabio-noble-baremetal-benjaminfix:~$ sudo cloud-init status -l status: done extended_status: degraded done boot_status_code: enabled-by-generator last_update: Thu, 07 Mar 2024 14:42:12 + detail: DataSourceOracle errors: [] recoverable_errors: WARNING: - Invalid network-config provided: Please run 'sudo cloud-init schema --system' to see the schema errors. ubuntu@fabio-noble-baremetal-benjaminfix:~$ sudo cloud-init schema --system Found cloud-config data types: user-data, network-config 1. user-data at /var/lib/cloud/instances/ocid1.instance.oc1.sa-saopaulo-1.antxeljrniwq6syclo77iwfs2fznvqcipuenlp4bkdjw2aqocj65dvmdmnkq/cloud-config.txt: Empty 'cloud-config' found at /var/lib/cloud/instances/ocid1.instance.oc1.sa-saopaulo-1.antxeljrniwq6syclo77iwfs2fznvqcipuenlp4bkdjw2aqocj65dvmdmnkq/cloud-config.txt. Nothing to validate. 2. network-config at /var/lib/cloud/instances/ocid1.instance.oc1.sa-saopaulo-1.antxeljrniwq6syclo77iwfs2fznvqcipuenlp4bkdjw2aqocj65dvmdmnkq/network-config.json: Invalid network-config /var/lib/cloud/instances/ocid1.instance.oc1.sa-saopaulo-1.antxeljrniwq6syclo77iwfs2fznvqcipuenlp4bkdjw2aqocj65dvmdmnkq/network-config.json Error: Cloud config schema errors: config.0.subnets.0: Additional properties are not allowed ('broadcast' was unexpected) Error: Invalid schema: network-config ubuntu@fabio-noble-baremetal-benjaminfix:~$ sudo jq . /var/lib/cloud/instances/ocid1.instance.oc1.sa-saopaulo-1.antxeljrniwq6syclo77iwfs2fznvqcipuenlp4bkdjw2aqocj65dvmdmnkq/network-config.json { "config": [ { "mac_address": "b8:3f:d2:c3:fd:7c", "name": "ens300f0np0", "subnets": [ { "broadcast": "10.0.0.255", "control": "manual", "dns_nameservers": [ "169.254.169.254" ], "gateway": "10.0.0.1", "netmask": "255.255.255.0", "type": "dhcp" } ], "type": "physical" } ], "version": 1 } ubuntu@fabio-noble-baremetal-benjaminfix:~$ sudo cloud-init --version /usr/bin/cloud-init 24.1-0ubuntu1 Attaching the logs. ** Affects: cloud-init (Ubuntu) Importance: High Status: Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056460 Title: cloud-init in degraded state on Oracle instance with "Invalid network- config provided" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2056460/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056460] Re: cloud-init in degraded state on Oracle instance with "Invalid network-config provided"
** Attachment added: "cloud-init.tar.gz" https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2056460/+attachment/5753817/+files/cloud-init.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056460 Title: cloud-init in degraded state on Oracle instance with "Invalid network- config provided" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2056460/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056187] Re: fails to configure BOOTIF when using iscsi
I've tested launching a Oracle Cloud baremetal instance (which boots from iSCSI) using such patch, and all worked well: https://pastebin.ubuntu.com/p/3cdFdYBVFG/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056187 Title: fails to configure BOOTIF when using iscsi To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2056187/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056194] Re: Networking broken in early boot on Oracle Native instances
I've tested the patch and it fixes the issue. I can confirm the MTU settings are now correct and curl works fine. I also confirmed it allowed cloud-init to run and fully complete the boot process: https://pastebin.ubuntu.com/p/KfcP7wmjjV/ There are no cloud-init errors: https://pastebin.ubuntu.com/p/jtkTWkDGVN/ https://pastebin.ubuntu.com/p/vZSyTJ3RsH/ All I noticed was a delay / hang during dhcp probe, followed by a timeout and it worked in the first retry: https://pastebin.ubuntu.com/p/hXWKYCypXM/ Overall, everything worked well. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056194 Title: Networking broken in early boot on Oracle Native instances To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056194] Re: Networking broken in early boot on Oracle Native instances
Between initramfs-tools 0.142ubuntu8 and 0.142ubuntu9, we've Replaced dhclient by dhcpcd (LP: #2024164). When a DHCP server provides MTU settings to dhcpcd, it configures the routes with the appropriate mtu value (due to "option interface_mtu" in /etc/dhcpcd.conf), but it does not configure the MTU setting in the interface. So, we end up having a mismatch between interface and route MTU setting, as observed below: root@(none):/# ip a 1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: ens3: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 02:00:17:05:ee:5a brd ff:ff:ff:ff:ff:ff altname enp0s3 inet 10.0.0.21/24 brd 10.0.0.255 scope global dynamic noprefixroute ens3 valid_lft 86048sec preferred_lft 75248sec inet6 fe80::17ff:fe05:ee5a/64 scope link valid_lft forever preferred_lft forever root@(none):/# ip route show default via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.21 metric 1002 mtu 9000 10.0.0.0/24 dev ens3 proto dhcp scope link src 10.0.0.21 metric 1002 mtu 9000 169.254.0.0/16 dev ens3 proto dhcp scope link src 10.0.0.21 metric 1002 mtu 9000 If we go back to initramfs-tools 0.142ubuntu8 (and, consequently to dhclient), the MTU settings will match: root@(none):/# ip a 1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: ens3: mtu 9000 qdisc mq state UP group default qlen 1000 link/ether 02:00:17:05:ee:5a brd ff:ff:ff:ff:ff:ff altname enp0s3 inet 10.0.0.21/24 brd 10.0.0.255 scope global ens3 valid_lft forever preferred_lft forever inet6 fe80::17ff:fe05:ee5a/64 scope link valid_lft forever preferred_lft forever root@(none):/# ip route show default via 10.0.0.1 dev ens3 10.0.0.0/24 dev ens3 proto kernel scope link src 10.0.0.21 169.254.0.0/16 dev ens3 scope link Due to this mismatch when using dhcpcd, certain network activities might fail. For example, in Oracle instances, a curl to the DataSource will just hang: curl --connect-timeout 10 --max-time 15 -H "Authorization: Bearer Oracle" -L http://169.254.169.254/opc/v2/instance/ If you either reduce the route MTU setting down to 1500 OR increase the interface MTU setting to 9000, curl works: option 1 - Reduce route setting: ip route delete 169.254.0.0/16 dev ens3 proto dhcp scope link src 10.0.0.21 metric 1002 mtu 9000 ip route add 169.254.0.0/16 dev ens3 scope link option 2 - Increase interface setting: ip link set mtu 9000 ens3 The correct MTU setting here is 9000. When you fully boot the instance, the MTU gets configured appropriately (outside of initramfs) I also checked the Paravirtualized instances (uses virtio_net interfaces instead of mlx5_core) and the mismatch is also there, but curl still works, so the Paravirtualized instance is also exposed to the bug but more resistant to it. While investigating this, I noticed that in past releases of dhcpcd, we had a hook 10-mtu, that would automatically configure the mtu setting based on information it received from the dhcp server. This file was removed by the following commit: https://github.com/NetworkConfiguration/dhcpcd/commit/ca6cdf5847cda720b65793ea6827b1b373c62382 There are some discussion around why this was removed here: https://github.com/NetworkConfiguration/dhcpcd/issues/67 Anyway, this causes such mismatch between interface and route settings, which cause curl to the IMDS to hang and cloud-init to fail. ** Bug watch added: github.com/NetworkConfiguration/dhcpcd/issues #67 https://github.com/NetworkConfiguration/dhcpcd/issues/67 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056194 Title: Networking broken in early boot on Oracle Native instances To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056194/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1973114] Re: Key trust verification fails on Ubuntu 22.04
Hi Brian, Thanks for that. I've tested and validated that ec2-instance-connect from jammy-proposed fixes the issue. Here are the evidences: https://pastebin.ubuntu.com/p/dPD6vyS6g4/ Cheers, Fabio Martins -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1973114 Title: Key trust verification fails on Ubuntu 22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ec2-instance-connect/+bug/1973114/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1973114] Re: Key trust verification fails on Ubuntu 22.04
I also tested with mssh to make it works well: When running ec2-instance-connect 1.1.14-0ubuntu1 (from Jammy) and trying to connect with mssh: fabio@fabio-canonical:~/.aws$ mssh ubuntu@i-0af3232b4fb6ed642 ubuntu@3.91.56.142: Permission denied (publickey). Due to the bug, we can see the key being pushed, but then the client denying the connection: https://pastebin.ubuntu.com/p/7StqsfQdJp/ Back in the instance, I've changed the sources.list to point to kinetic repos and upgraded ec2-instance-connect to kinetic's version: ubuntu@ip-10-0-1-226:~$ sudo apt-cache policy ec2-instance-connect ec2-instance-connect: Installed: 1.1.14-0ubuntu2 Candidate: 1.1.14-0ubuntu2 Version table: *** 1.1.14-0ubuntu2 500 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu kinetic/main amd64 Packages 100 /var/lib/dpkg/status And now mssh works fine: fabio@fabio-canonical:~/.aws$ mssh ubuntu@i-0af3232b4fb6ed642 Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-1004-aws x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support:https://ubuntu.com/advantage System information as of Fri May 13 14:16:20 UTC 2022 System load: 0.2421875 Processes: 128 Usage of /: 21.1% of 7.58GB Users logged in: 1 Memory usage: 1%IPv4 address for eth0: 10.0.1.226 Swap usage: 0% 74 updates can be applied immediately. To see these additional updates run: apt list --upgradable Last login: Fri May 13 13:47:05 2022 from 189.19.160.174 ubuntu@ip-10-0-1-226:~$ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1973114 Title: Key trust verification fails on Ubuntu 22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ec2-instance-connect/+bug/1973114/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
I've also updated the [Test Plan] section of the bug description ** Description changed: [Impact] In some cases, ipconfig can take a longer time than the user-specified timeouts, causing unexpected delays. [Test Plan] + + - Check that the ipconfig utility is able to obtain an IP through dhcp: + + # ip l set dev ens9 down + # date; /usr/lib/klibc/bin/ipconfig ens9; date + + - Check if it respects the timeout (i.e. 2 seconds in the example + below): + + # ip l set dev ens9 down + # date; /usr/lib/klibc/bin/ipconfig -t 2 ens9; date + + - The original issue is that timeout is being ignored when not receiving + any reply from a DHCP Server, so for this test you have to stop your + DHCP Server (i.e. dnsmasq) and then test the timeouts: + + # ip l set dev ens9 down + # date; /usr/lib/klibc/bin/ipconfig -t 2 ens9; date + + # ip l set dev ens9 down + # date; /usr/lib/klibc/bin/ipconfig -t 5 ens9; date + + Make sure it times out respectivelly in 2 and 5 seconds. [racb: pending agreement with the SRU team; please see comment 37] Any situation where ipconfig encounters an error sending the DHCP packet, it will automatically set a delay of 10 seconds, which could be longer than the user-specified timeout. It can be reproduced by creating a dummy interface and attempting to run ipconfig on it with a timeout value of less than 10: # ip link add eth1 type dummy # date; /usr/lib/klibc/bin/ipconfig -t 2 eth1; date Thu Nov 18 04:46:13 EST 2021 IP-Config: eth1 hardware address ae:e0:f5:9d:7e:00 mtu 1500 DHCP RARP IP-Config: no response after 2 secs - giving up Thu Nov 18 04:46:23 EST 2021 ^ Notice above, ipconfig thinks that it waited 2 seconds, but the timestamps show an actual delay of 10 seconds. [Where problems could occur] Please see reproduction steps above. We are seeing this in production too (see comment #2). [Other Info] A patch to fix the issue is being proposed here. It is a safe fix - it only checks before going into sleep that the timeout never exceeds the user-requested value. [Original Description] In some cases, ipconfig can take longer than the user-specified timeouts, causing unexpected delays. in main.c, in function loop(), the process can go into process_timeout_event() (or process_receive_event() ) and if it encounters an error situation, will set an attempt to "try again later" at time equal now + 10 seconds by setting s->expire = now + 10; This can happen at any time during the main event loop, which can end up extending the user-specified timeout if "now + 10" is greater than "start_time + user-specified-timeout". I believe a patch like the following is needed to avoid this problem: --- a/usr/kinit/ipconfig/main.c +++ b/usr/kinit/ipconfig/main.c @@ -437,6 +437,13 @@ static int loop(void) if (timeout > s->expire - now.tv_sec) timeout = s->expire - now.tv_sec; + + /* Compensate for already-lost time */ + gettimeofday(&now, NULL); + if (now.tv_sec + timeout > start + loop_timeout) { + timeout = loop_timeout - (now.tv_sec - start); + printf("Lowered timeout to match user request = (%d s) \n", timeout); + } } I believe the current behaviour is buggy. This is confirmed when the following line is executed: if (loop_timeout >= 0 && now.tv_sec - start >= loop_timeout) { printf("IP-Config: no response after %d " "secs - giving up\n", loop_timeout); rc = -1; goto bail; } 'loop_timeout' is the user-specified time-out. With a value of 2, in case of error, this line prints: IP-Config: no response after 2 secs - giving up So it thinks that it waited 2 seconds - however, in reality it had actually waited for 10 seconds. The suggested code-change ensures that the timeout that is actually used never exceeds the user-specified timeout. [ Regression potential ] This change ensures that user-specified timeouts are never exceeded, which is a problem that appears to happen only in case of interface errors. It may be that someone is relying on current behaviour where they receive DHCP offers after their specified timeout (but within the 10-second error timeout). However, 1) that is buggy behaviour and should be exposed. Such a user would need to update their specified timeout to make it long enough to receive the DHCP offer (setting the timeout to 10 would keep the existing behaviour). 2) I think it is unlikely that such a scenario exists at all. The 10-second timeout problem happens w
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
Hello Robie, I've validated that the package from -proposed works well, testing in my VM based environment. I haven't tested it on Oracle bare metal (where the original issue happened) as that is a type of instance hard to get access to. Given that the test packages had proven to fix the original problem that we were targetting to address, I believe the package from -proposed should also work well. Let me know if it is necessary to also test that it addresses the original issue with Oracle BM host and I'll get access to one and validate. Regarding your question below: > ...would it be practical and/or useful to verify that, with a timeout of 2s, a DHCP reply sent after 1.5s works, but a DHCP reply sent after 2.5s does not? That has been addressed in comment #30 (and now also in #39 with the package from -proposed). Regarding your question if upstream had accepted the patch, I would have to defer that to Khaled, but I also agree that it seems a deliberate upstream design decision not to accept it. Regarding your comment below: > In Ubuntu, we might decide to maintain the patch as a delta but then drop that delta in subsequent releases when we no longer need that functionality. That is indeed just needed for Bionic. To confirm/clarify my understanding, if this moves to -updates, in theory this is not going to be reverted (from Bionic) in the future, right? Once it lands there, it is expected that any newer klibc-utils packages that gets released to Bionic will continue to carry this patch? Thank you in advance! Regards, Fabio Martins -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
I've tested the package from -proposed and I can confirm it fixes the problem: Installed from -proposed: root@ubuntu:~# apt-cache policy klibc-utils klibc-utils: Installed: 2.0.4-9ubuntu2.2 Candidate: 2.0.4-9ubuntu2.18.04.1 Version table: 2.0.4-9ubuntu2.18.04.1 500 500 http://ppa.launchpad.net/mfo/lp1947099/ubuntu bionic/main amd64 Packages *** 2.0.4-9ubuntu2.2 500 500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages 100 /var/lib/dpkg/status 2.0.4-9ubuntu2.1 500 500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages 2.0.4-9ubuntu2 500 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages Working well without timeout: root@ubuntu:~# ip l set dev ens9 down root@ubuntu:~# date; /usr/lib/klibc/bin/ipconfig ens9; date Thu May 12 12:54:48 UTC 2022 IP-Config: ens9 hardware address 52:54:00:f3:4e:0e mtu 1500 DHCP RARP IP-Config: ens9 complete (dhcp from 192.168.150.2): address: 192.168.150.105 broadcast: 192.168.150.255 netmask: 255.255.255.0 gateway: 192.168.150.1dns0 : 192.168.150.2dns1 : 0.0.0.0 domain : sts.lab rootserver: 192.168.150.2 rootpath: filename : Thu May 12 12:55:01 UTC 2022 Timeout being honored: root@ubuntu:~# ip l set dev ens9 down root@ubuntu:~# date; /usr/lib/klibc/bin/ipconfig -t 2 ens9; date Thu May 12 12:55:44 UTC 2022 IP-Config: ens9 hardware address 52:54:00:f3:4e:0e mtu 1500 DHCP RARP Lowered timeout to match user request = (2 s) IP-Config: no response after 2 secs - giving up Thu May 12 12:55:46 UTC 2022 Testing the original issue (timeout being ignored when not receiving any reply from a DHCP Server), which is now also getting honored: root@ubuntu:~# ip l set dev ens9 down root@ubuntu:~# date; /usr/lib/klibc/bin/ipconfig -t 2 ens9; date Thu May 12 12:56:32 UTC 2022 IP-Config: ens9 hardware address 52:54:00:f3:4e:0e mtu 1500 DHCP RARP Lowered timeout to match user request = (2 s) IP-Config: no response after 2 secs - giving up Thu May 12 12:56:34 UTC 2022 root@ubuntu:~# ip l set dev ens9 down root@ubuntu:~# date; /usr/lib/klibc/bin/ipconfig -t 5 ens9; date Thu May 12 12:56:43 UTC 2022 IP-Config: ens9 hardware address 52:54:00:f3:4e:0e mtu 1500 DHCP RARP Lowered timeout to match user request = (5 s) IP-Config: no response after 5 secs - giving up Thu May 12 12:56:48 UTC 2022 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
Should this bug be changed to Fix Committed at this point? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
I've tested the new patch from ppa:mfo/lp1947099v2 and I can confirm it resolves the problem: - Without the patch: https://pastebin.ubuntu.com/p/RksNcBGSzn/ It took 396,940865−220,447147 = 176,493718 seconds in the IP-Config section. Total boot time: ubuntu@gpu48-ubuntu18:~$ sudo systemd-analyze time Startup finished in 4min 1.355s (firmware) + 2min 24.433s (loader) + 6min 8.464s (kernel) + 41.466s (userspace) = 13min 15.719s graphical.target reached after 41.068s in userspace - With the patch: https://pastebin.ubuntu.com/p/46nVYCYyDZ/ It took 246,556749−212,019368 = 34,537381 seconds in the IP-Config section. Total boot time: ubuntu@gpu48-ubuntu18:~$ sudo systemd-analyze time Startup finished in 4min 1.246s (firmware) + 2min 24.170s (loader) + 3min 42.915s (kernel) + 39.010s (userspace) = 10min 47.343s graphical.target reached after 38.634s in userspace ubuntu@gpu48-ubuntu18:~$ sudo apt-cache policy klibc-utils klibc-utils: Installed: 2.0.4-9ubuntu2.18.04.1 Candidate: 2.0.4-9ubuntu2.18.04.1 Version table: *** 2.0.4-9ubuntu2.18.04.1 500 500 http://ppa.launchpad.net/mfo/lp1947099v2/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status 2.0.4-9ubuntu2.1 500 500 http://me-dubai-1-ad-1.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages 2.0.4-9ubuntu2 500 500 http://me-dubai-1-ad-1.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
Thank you, Mauricio, for the build process details and for adding the update here. I'm including some evidence of my tests showing that the patch you suggested did work well: Details of the build process: https://pastebin.ubuntu.com/p/dmVWH2fxpy/ Test package installed: https://pastebin.ubuntu.com/p/qxyWBGrm3P/ Working well without specifying a timeout: https://pastebin.ubuntu.com/p/byftnFk33C/ Timeout is also being honored: https://pastebin.ubuntu.com/p/spFsTRhXTz/ Testing the original issue (timeout being ignored when not receiving any reply from a DHCP Server), which is now also getting honored: https://pastebin.ubuntu.com/p/KHVRj7BdRw/ I'm working to get access to another instance to test that the boot time optimiizations are still fine. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
@Łukasz / @Robie, do you think the above comments are enough to proceed with this SRU? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
I've tested the klibc-utils patch using Mauricio's ppa: sudo add-apt-repository ppa:mfo/lp1947099 sudo apt install klibc-utils sudo update-initramfs -u -k all And I can confirm that it does improve the boot time in more than 3 minutes, without causing any noticeable issues. - Without the patch: https://pastebin.ubuntu.com/p/6M7M2FfCGQ/ root@ubuntu1804:~# sudo systemd-analyze time Startup finished in 4min 2.098s (firmware) + 2min 23.684s (loader) + 6min 226ms (kernel) + 38.766s (userspace) = 13min 4.776s graphical.target reached after 38.374s in userspace We can see it spent 6min 226ms in kernel, and looking through the serial console (or dmesg) it was running the ipconfig commands for each of the Mellanox NICs from 211.972259 up to 386.155116 = 174.182857 seconds - With the patch: https://pastebin.ubuntu.com/p/JxDs8WPqc4/ root@ubuntu1804:~# sudo systemd-analyze time Startup finished in 4min 890ms (firmware) + 2min 23.278s (loader) + 3min 46.413s (kernel) + 40.734s (userspace) = 10min 51.317s graphical.target reached after 40.354s in userspace We can see the kernel time decreased to a bit more than 3 minutes and we spent only 41 seconds (as opposed to 174 seconds) in the ipconfig commands: 210.675050 to 252.432441 = 41.757391 seconds That definitely has helped to resolve the issue we are chasing here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
I tried using klibc-utils from the ppa (containing the patch): root@ubuntu:~# sudo apt-cache policy klibc-utils klibc-utils: Installed: 2.0.4-9ubuntu2.18.04.1 Candidate: 2.0.4-9ubuntu2.18.04.1 Version table: *** 2.0.4-9ubuntu2.18.04.1 500 500 http://ppa.launchpad.net/mfo/lp1947099/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status 2.0.4-9ubuntu2 500 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1] But now when I try to obtain an IP using dhcp without specifying any timeout, it dumps lots of "Lowered timeout to match user request" messages. Is that expected? https://pastebin.ubuntu.com/p/5Tpc5Rwdkq/ [2] Other than that, it worked well. We can see from the pastebin below that the timeout is getting respected: https://pastebin.ubuntu.com/p/TJbCzx7hFC/ [3] And also the main issue was the timeout being ignored when not receiving any reply from a DHCP Server, which is now also getting honored: https://pastebin.ubuntu.com/p/BvKk6fJJDc/ I will try to test this package in a OCI BM.GPU4.8 instance to see the improvements with the main optimization, but it would be good to get clarification on [1] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not honour user-requested timeouts in some cases To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1947099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1944574] Re: EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
I believe the "EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary" that was reported on this bug is not what is preventing the VM from booting. If you look into the full log you provided, that message is logged both on the boot that succeeded and in the one that got stuck. I believe whatever was done in the VM in between these reboots, might have caused the problem you are observing. There's also a known problem on linux-oracle kernel that prevents you from seeing the serial console. That is being fixed for Focal images on the linux-oracle 5.13.0.1023.28~20.04.1 kernel, which is currenlty in focal-proposed. If this is something you can reproduce with Focal images, using the kernel from proposed might help you see what is really preventing the VM from starting. We'll review the 'kernel image not aligned on 64k boundary' error anyway to assess what might be causing it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1944574 Title: EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-oracle/+bug/1944574/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
I've setup a Lab with dnsmasq acting as DHCP Server, which I can use the dhcp-reply-delay option to introduce a delay between the DHCPDISCOVER and DHCPOFFER, as in the example below: Mar 30 18:26:34 focal-dhcpsrv dnsmasq-dhcp[2470]: DHCPDISCOVER(ens3) 52:54:00:d7:10:13 Mar 30 18:26:34 focal-dhcpsrv dnsmasq-dhcp[2470]: 4195928115 reply delay: 3 Mar 30 18:26:37 focal-dhcpsrv dnsmasq-dhcp[2470]: DHCPOFFER(ens3) 192.168.150.119 52:54:00:d7:10:13 Just minor note that the delay only happens between DHCPDISCOVER and DHCPOFFER, but not between DHCPREQUEST and DHCPACK: Mar 30 18:22:13 focal-dhcpsrv dnsmasq-dhcp[2470]: DHCPREQUEST(ens3) 192.168.150.119 52:54:00:d7:10:13 Mar 30 18:22:13 focal-dhcpsrv dnsmasq-dhcp[2470]: DHCPACK(ens3) 192.168.150.119 52:54:00:d7:10:13 ubuntu So, if this is a new interface (new mac) looking for an IP, the delay will be added, but if this is an interface which was previously configured and is asking to re-use the same IP (DHCPREQUEST), there won't be any delays. I believe this Lab will help reproducing the issue with and without the proposed patch. 1. In a ideal scenario, where no delays were added (dhcp-reply-delay is commented out), ipconfig will take approximately to do its job: it sends a DHCPREQUEST, then DHCPDISCOVER and then another DHCPREQUEST in this process: - From the ipconfig perspective: root@ubuntu:/etc/dhcp# date; /usr/lib/klibc/bin/ipconfig ens9; date Wed Mar 30 19:25:44 UTC 2022 IP-Config: ens9 hardware address 52:54:00:f3:4e:0e mtu 1500 DHCP RARP IP-Config: ens9 complete (dhcp from 192.168.150.2): address: 192.168.150.105 broadcast: 192.168.150.255 netmask: 255.255.255.0 gateway: 192.168.150.1dns0 : 192.168.150.2dns1 : 0.0.0.0 domain : sts.lab rootserver: 192.168.150.2 rootpath: filename : Wed Mar 30 19:25:54 UTC 2022 - From the dhcp server perspective: Mar 30 19:25:45 focal-dhcpsrv dnsmasq-dhcp[2656]: DHCPREQUEST(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:25:45 focal-dhcpsrv dnsmasq-dhcp[2656]: DHCPACK(ens3) 192.168.150.105 52:54:00:f3:4e:0e ubuntu Mar 30 19:25:54 focal-dhcpsrv dnsmasq-dhcp[2656]: DHCPDISCOVER(ens3) 52:54:00:f3:4e:0e Mar 30 19:25:54 focal-dhcpsrv dnsmasq-dhcp[2656]: DHCPOFFER(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:25:54 focal-dhcpsrv dnsmasq-dhcp[2656]: DHCPREQUEST(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:25:54 focal-dhcpsrv dnsmasq-dhcp[2656]: DHCPACK(ens3) 192.168.150.105 52:54:00:f3:4e:0e 2. Adding a 5 seconds delay in dnsmasq (dhcp-reply-delay=5) and without enforcing a timeout in ipconfig: - From ipconfig perspective: root@ubuntu:/etc/dhcp# date; /usr/lib/klibc/bin/ipconfig ens9; date Wed Mar 30 19:34:09 UTC 2022 IP-Config: ens9 hardware address 52:54:00:f3:4e:0e mtu 1500 DHCP RARP IP-Config: ens9 complete (dhcp from 192.168.150.2): address: 192.168.150.105 broadcast: 192.168.150.255 netmask: 255.255.255.0 gateway: 192.168.150.1dns0 : 192.168.150.2dns1 : 0.0.0.0 domain : sts.lab rootserver: 192.168.150.2 rootpath: filename : Wed Mar 30 19:34:27 UTC 2022 - From the dhcpserver perspective: Mar 30 19:34:10 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPREQUEST(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:34:10 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPACK(ens3) 192.168.150.105 52:54:00:f3:4e:0e ubuntu Mar 30 19:34:19 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPDISCOVER(ens3) 52:54:00:f3:4e:0e Mar 30 19:34:19 focal-dhcpsrv dnsmasq-dhcp[2773]: 1004609631 reply delay: 5 Mar 30 19:34:24 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPOFFER(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:34:24 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPDISCOVER(ens3) 52:54:00:f3:4e:0e Mar 30 19:34:24 focal-dhcpsrv dnsmasq-dhcp[2773]: 1004609631 reply delay: 5 Mar 30 19:34:26 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPOFFER(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:34:26 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPDISCOVER(ens3) 52:54:00:f3:4e:0e Mar 30 19:34:26 focal-dhcpsrv dnsmasq-dhcp[2773]: 1004609631 reply delay: 5 Mar 30 19:34:28 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPOFFER(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:34:28 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPREQUEST(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:34:28 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPACK(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:34:28 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPREQUEST(ens3) 192.168.150.105 52:54:00:f3:4e:0e Mar 30 19:34:28 focal-dhcpsrv dnsmasq-dhcp[2773]: DHCPACK(ens3) 192.168.150.105 52:54:00:f3:4e:0e It takes 18 seconds for the process to complete, as dnsmasq adds 3x 5 seconds delay in the process 3. Using the -t to specify a 15 seconds timeout (when the actual process takes 18 seconds): - From the ipconfig perspective: root@ubuntu:/etc/dhcp# date; /usr/lib/klibc/bin/ipconfig -t 15 ens9; date Wed Mar 30 19:37:11 UTC 2
[Bug 1947099] Re: ipconfig does not honour user-requested timeouts in some cases
Hi Robie, The user story here is about improving the time it takes to boot a Bionic instance on Oracle Cloud in a specific bare metal shape, called BM.GPU4.8. This is a pretty large instance, with 18x Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]: https://www.oracle.com/uk/cloud/compute/gpu.html Comparing the time it takes to boot Bionic with a CentOS instance we can see a big delta both in kernel and userspace: CentOS: Startup finished in 4min 3.548s (firmware) + 2min 23.525s (loader) + 3min 17.984s (kernel) + 1min 37.825s (initrd) + 38.723s (userspace) = 12min 1.608s multi-user.target reached after 38.714s in userspace Ubuntu: Startup finished in 4min 621ms (firmware) + 2min 23.174s (loader) + 7min 34.767s (kernel) + 5min 48.219s (userspace) = 19min 46.782s graphical.target reached after 5min 47.854s in userspace The userspace difference is a cloud-init problem that is being handled by other launchpad bugs. Here we are trying to focus on the kernel difference, which further was narrowed down to be related to klibc and it is what we are trying to address with this bug. Some details: Looking into details, here are the dmesg output from CentOS and Ubuntu, for kernel comparison: Ubuntu: https://pastebin.canonical.com/p/X7GFVjdV3b/ CentOS: https://pastebin.canonical.com/p/gNhFwJjyjw/ Looking there, one area that looks promising is around mlx5_core driver, where Ubuntu spends 218 seconds, while CentOS spends 44 seconds Khalid did a nice job investigating and found that the problem came from ipconfig from initramfs. Here are the details: The delay from the kernel side is due to the fact that every interface (18 in total) takes over 10 seconds each to initialize. The interfaces are brought up and used for DHCP as part of the iscsi initialization, during the initramfs stage. Specifically the script ```/usr/share/initramfs-tools/scripts/local-top/iscsi``` which is included in the initramfs and is installed as part of open-iscsi To initialize the network interfaces this script calls other helper functions that are part of the initramfs, specifically "configure_networking" in ```/usr/share/initramfs- tools/scripts/functions``` which is installed as part of initramfs- tools-core. The configure_networking function initializes networking by looping over all interfaces and issuing this call: ``` ipconfig -t 2 ``` where it is calling the ipconfig utility (part of klibc-utils) which also gets bundled in the initramfs. The -t 2 specifies a timeout of 2 seconds. This ipconfig tool attempts to perform DHCP on the given interface, within the specified timeout (always 2 seconds for each interface) Normally, ipconfig does not perform only one DHCP attempt - it performs multiple DHCP attempts, each attempt waiting for an exponentially increasing amount of time (1s, 2s, 4s, ..etc) up until the total amount of time has passed that is equal the user-specified timeout (2 in this case, so it should wait 1 second, followed by 1 second, for a total of 2 seconds). However, it appears ipconfig has a bug. In case it encounters an error sending a DHCP request on an interface (which is the case for 17 out of the 18 interfaces on this instance), it attempts to "try again later" and sets a timer of 10 seconds for the next attempt - ignoring completely the user-specified value (2 seconds). I believe this is a bug in ipconfig and have a working fix to limit the delay to the user- specified timeout so each interface actually takes a maximum of 2 seconds for initialization, even in error cases. The above is for bionic, which has the reported problem. In focal, even though the kernel and klibc-utils are identical to bionic, the version of initramfs-tools-core is different. In the focal version, configure_networking() has been reworked to avoid using ipconfig entirely. Instead, it uses ```dhclient```. And instead of doing that 18 times, dhclient is called only once with the full list of interfaces, as such: ``` dhclient -v -1 -cf /etc/dhcp/dhclient.conf.2 -pf /tmp/dhclient.pid.2 -4 enp12s0f0np0 enp12s0f1np1 enp138s0f0np0 enp138s0f1np1 enp148s0f0np0 enp148s0f1np1 enp195s0f0np0 enp195s0f1np1 enp209s0f0np0 enp209s0f1np1 enp22s0f0np0 enp22s0f1np1 enp45s0f0np0 enp45s0f1np1 enp72s0f0np0 enp72s0f1np1 enp76s0f0np0 enp76s0f1np1 ``` dhclient does DHCP on all those interfaces at the same time, asynchronously, and it daemonizes (goes in the background) as soon as it gets a DHCP response which happens relatively quickly (on interface enp45s0f0np0 usually) and so the boot-up continues *much* faster compared to bionic (180+ seconds are reduced to just 5-10 seconds). I'm sure there might be other use cases, but this is what we have been reported and are trying to work with. I hope this helps clarifying. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1947099 Title: ipconfig does not hono
[Bug 1946211] Re: [SRU] "radosgw-admin bucket limit check" has duplicate entries if bucket count exceeds 1000 (max_entries)
Hi, I've verified that the package from ussuri-proposed fixes the issue. Without the patch: https://pastebin.ubuntu.com/p/PFrkX7BkCT/ With the patch: https://pastebin.ubuntu.com/p/yytzD3sjV9/ Cheers and thank you for fixing it. - Fabio -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1946211 Title: [SRU] "radosgw-admin bucket limit check" has duplicate entries if bucket count exceeds 1000 (max_entries) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1946211/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1946211] Re: [SRU] "radosgw-admin bucket limit check" has duplicate entries if bucket count exceeds 1000 (max_entries)
Hi, I can confirm that the packages from -propose will resolve the issue. See evidences bellow: - While using 15.2.12, we can see the bug: https://pastebin.ubuntu.com/p/jYg8x4Q7vR/ - After installing 15.2.14 from -proposed, the bug is resolved: https://pastebin.ubuntu.com/p/mKFrTjddX7/ Thank you for your help addressing this issue. Regards, Fabio Martins ** Tags removed: verification-needed verification-needed-focal ** Tags added: verification-done verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1946211 Title: [SRU] "radosgw-admin bucket limit check" has duplicate entries if bucket count exceeds 1000 (max_entries) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1946211/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921104] Re: net/mlx5e: Add missing capability check for uplink follow
Another customer has provided positive feedback that it fixes the issue on Focal: 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921104 Title: net/mlx5e: Add missing capability check for uplink follow To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1921104/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1923115] Re: Networkd vs udev nic renaming race condition
Customer has provided a positive feedback that the package in -proposed fixed this bug -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1923115 Title: Networkd vs udev nic renaming race condition To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1923115/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919261] [NEW] Upgrading Ceph from 14.2.11-0ubuntu0.19.10.1~cloud4 to 15.2.8-0ubuntu0.20.04.1~cloud0 fails when ceph-mds is installed
Public bug reported: In a host where ceph-mds is installed, upgrading from 14.2.11-0ubuntu0.19.10.1~cloud4 to 15.2.8-0ubuntu0.20.04.1~cloud0 fails with: dpkg: error processing archive /tmp/apt-dpkg-install-Zen6uw/9-ceph-common_15.2.8-0ubuntu0.20.04.1~cloud0_amd64.deb (--unpack): trying to overwrite '/usr/bin/cephfs-data-scan', which is also in package ceph-mds 14.2.11-0ubuntu0.19.10.1~cloud4 dpkg-deb: error: paste subprocess was killed by signal (Broken pipe) Errors were encountered while processing: /tmp/apt-dpkg-install-Zen6uw/9-ceph-common_15.2.8-0ubuntu0.20.04.1~cloud0_amd64.deb E: Sub-process /usr/bin/dpkg returned an error code (1) The problem happens because /usr/bin/cephfs-data-scan is provided by ceph-mds in 14.2.11-0ubuntu0.19.10.1~cloud4: # dpkg -S /usr/bin/cephfs-data-scan ceph-mds: /usr/bin/cephfs-data-scan However in 15.2.8-0ubuntu0.20.04.1~cloud0 it is provided by ceph-common: # dpkg -S /usr/bin/cephfs-data-scan ceph-common: /usr/bin/cephfs-data-scan A quick workaround is to temporarily remove ceph-mds before the upgrade (dpkg -r ceph-mds) and then perform the upgrade process using apt install. It will upgrade all packages and reinstall ceph-mds. ** Affects: ceph (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919261 Title: Upgrading Ceph from 14.2.11-0ubuntu0.19.10.1~cloud4 to 15.2.8-0ubuntu0.20.04.1~cloud0 fails when ceph-mds is installed To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1919261/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1852077] Re: Backport: bonding: fix state transition issue in link monitoring
Hi Po, IIUC this bug is related to commit 627450ac21f7f4a44b949c5d5e2c35829ff1784f, which is in 4.15.0-74, which I see now in -updates / -security. Isn't it completed yet? $ git tag --contains 627450ac21f7f4a44b949c5d5e2c35829ff1784f Ubuntu-4.15.0-73.82 Ubuntu-4.15.0-74.84 Ubuntu-raspi2-4.15.0-1053.57 Ubuntu-snapdragon-4.15.0-1070.77 $ rmadison linux-image-4.15.0-74-generic linux-image-4.15.0-74-generic | 4.15.0-74.83~16.04.1 | xenial-security | amd64, arm64, armhf, i386, ppc64el, s390x linux-image-4.15.0-74-generic | 4.15.0-74.83~16.04.1 | xenial-updates | amd64, arm64, armhf, i386, ppc64el, s390x linux-image-4.15.0-74-generic | 4.15.0-74.84 | bionic-security | amd64, arm64, armhf, i386, ppc64el, s390x linux-image-4.15.0-74-generic | 4.15.0-74.84 | bionic-updates | amd64, arm64, armhf, i386, ppc64el, s390x -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1852077 Title: Backport: bonding: fix state transition issue in link monitoring To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1852077/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853510] Re: "*** buffer overflow detected" when running atop
** Package changed: tree (Ubuntu) => atop (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853510 Title: "*** buffer overflow detected" when running atop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/atop/+bug/1853510/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1829823] Re: libvirt-bin: during shutdown libvirt-bin is stopped before libvirt-guests causing hang
Hello Corey and Matthew, I just want to confirm that I've installed libvirt-bin 1.3.1-1ubuntu10.28~cloud0 from mitaka-proposed in a Ubuntu Trusty and it fixes the issue. I'm tagging verification-done-mitaka here. Thank you very much. Regards, Fabio Martins ** Tags removed: verification-mitaka-needed ** Tags added: verification-done-mitaka -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1829823 Title: libvirt-bin: during shutdown libvirt-bin is stopped before libvirt- guests causing hang To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1829823/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1813822] [NEW] Improve log messages during iSCSI login failures (specially related to iSCSI redirection)
Public bug reported: When an iSCSI Storage (eg.: SolidFire) uses iSCSI redirects, the initial login fails, but it just logs "Login authentication failed with target", while the tcpdump actually shows what exactly was happening. For example: 1. Ubuntu tries to login to the Target IP (10.5.0.26): 24 8.85875610.5.0.710.5.0.26 iSCSI 252 Login Command 2. The Storage (10.5.0.26) replies with: 27 8.86031810.5.0.26 10.5.0.7iSCSI 148 Login Response (Target moved temporarily) The iSCSI packet contains "TargetAddress=10.5.0.18:3260", which is the new iSCSI Portal that we should use: 3. Ubuntu then tries to login to the new portal: 50 9.37749110.5.0.710.5.0.18 iSCSI 252 Login Command 4. Which succeeds: 57 9.37908210.5.0.18 10.5.0.7iSCSI 156 Login Response (Success) But in /var/log/syslog, the only message that is displayed is: Jan 22 16:01:52 xenial-initiator iscsid: Login authentication failed with target iqn.1993-08.org.debian:01:4df860e415a6:redirect As an enhancement request, it would be good to see the login redirection being logged into syslog. The same behavior applies to Trusty, Xenial and Bionic. The same request has been made upstream, but it has been considered low priority so far: https://github.com/open-iscsi/open-iscsi/issues/117 ** Affects: open-iscsi (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1813822 Title: Improve log messages during iSCSI login failures (specially related to iSCSI redirection) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1813822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1663280] Re: Serious performance degradation of math functions
** Changed in: glibc (Ubuntu Xenial) Importance: Medium => Low -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1663280 Title: Serious performance degradation of math functions To manage notifications about this bug go to: https://bugs.launchpad.net/glibc/+bug/1663280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1793430] Re: Page leaking in cachefiles_read_backing_file while vmscan is active
** Tags removed: verification-needed-xenial ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1793430 Title: Page leaking in cachefiles_read_backing_file while vmscan is active To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1793430/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1734327] Re: Kernel panic on a nfsroot system
** Changed in: linux (Ubuntu Artful) Status: Fix Committed => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1734327 Title: Kernel panic on a nfsroot system To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1734327/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 683725] Re: When I format an usb key, after remount it opens the empty device twice.
Thank you for filing this bug. I tried to reproduce it in Xenial, but it seems this behavior is no longer happening and very likely is already fixed, so I'm closing this bug as invalid. This is also a duplicate of 686780. ** Changed in: nautilus (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/683725 Title: When I format an usb key, after remount it opens the empty device twice. To manage notifications about this bug go to: https://bugs.launchpad.net/hundredpapercuts/+bug/683725/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 686780] Re: USB format opens two nautilus windows
Thank you for filing this bug. I tried to reproduce it in Xenial, but it seems this behavior is no longer happening and very likely is already fixed, so I'm closing this bug as invalid. This is also a duplicate of 683725. ** Changed in: nautilus (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/686780 Title: USB format opens two nautilus windows To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/686780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 334806] Re: nautilus disappear desktop icons and freeze open folders on jaunty
We are closing this bug report as it is related to a no longer supported version of Ubuntu + Kernel + Nautilus and this issue has not been observed in any recent and supported releases. ** Changed in: nautilus (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/334806 Title: nautilus disappear desktop icons and freeze open folders on jaunty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/334806/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1207384] Re: apache2 failure to start on boot when binding to IPv6 address
Thank you for submitting this bug. We're sorry, but this problem needs to be fixed in ifupdown, not in apache2. Ubuntu 12.04 is no longer a supported version. This bug is being changed to invalid for this reason. ** Changed in: apache2 (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1207384 Title: apache2 failure to start on boot when binding to IPv6 address To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1207384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs