Re: Reducing initramfs size and speed up the generation
On Wed, Aug 09, 2023 at 05:30:04PM +, Benjamin Drung wrote: > > The benefits to this are that the firmware and base initrds only need be > > generated once regardless of number of kernels installed. And their > > generation is decoupled from kernel upgrades/installs and each other. > > So the firmware initrd only needs to be regenerated when the firmware > > package is upgraded, and that need not trigger the base initrd to be > > regenerated. Likewise, upgrading cryptsetup (or any other dependency of > > the base initrd) need not cause the firmware initrd to be regenerated. > > This approach could also be used with the early init microcode parts of > > the initrd. > This is an interesting idea. It raises some questions. > The list of firmware files to include in the initramfs is derived from > the kernel modules. Different kernel versions can require different > firmware files. How should that be handled with this approach? While it might be nice to further reduce the space requirements for /boot (especially in the face of ever-growing kernels+firmware), this question is precisely why I don't consider this viable. One of the properties of the current system is that installing new versions of packages that trigger regeneration of the initramfs for the most recent kernel should by default leave the initramfs for other kernels unmodified; this way, in the event of a regression, the last-known-good kernel can still be booted for recovery. If all of the kernels installed would be pointed at a common firmware initramfs that's pulled in by GRUB, then even in the case that the updated firmware package is *supposed* to be compatible with all the kernels on the system, it nevertheless loses this property that we haven't tampered with the last-known-good kernel and makes the system less resilient. We should prioritize resilience of boot recovery over reducing the size of /boot contents. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: +1 maintenance shift (07/AUG - 11/AUG)
Hi Andreas, On Fri, Aug 11, 2023 at 06:40:16PM -0300, Andreas Hasenack wrote: > canonistack isn't an option. Why? Do we need to open RTs? -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
+1 maintenance shift (07/AUG - 11/AUG)
# rust-phf & friends Had to rebuild rust-pallete, which was an FTBFS Filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1043222 and debian fixed it before I had to upload the package with a delta of our own. Then rust-markup5ever was failing to build. Traced to rust-tendril failing to build. Fixed upstream in 0.4.3 (https://github.com/servo/tendril/issues/67), which also needs a newer rust-futf. Asked debian to update (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1043295), which they did promptly before I uploaded to mantic (I was testing things in a ppa before to be sure), and that sorted the problem. # curl 8.2.1-1ubuntu1 DEP8 failure on ppc64el and s390x Added a retry loop right after slapd was restarted. Fixed in https://launchpad.net/ubuntu/+source/curl/8.2.1-1ubuntu2 and sent to debian via https://salsa.debian.org/debian/curl/-/merge_requests/17 # doxygen FTBFS, causing FTBFS in other packages too https://bugs.launchpad.net/ubuntu/+source/doxygen/+bug/2026834 I had seen this in a previous shift. Debian said they would update to the version that has the fix (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040864), but that didn't happen yet. I had two branches with options: a) 7 patches to make it work (https://git.launchpad.net/~ahasenack/ubuntu/+source/doxygen/tree/debian/patches?h=mantic-doxygen-cairo-compat) b) update to 1.9.7 (https://code.launchpad.net/~ahasenack/ubuntu/+source/doxygen/+git/doxygen/+ref/mantic-update-doxygen unfinished) I ended up with (c): export env var to make cairo produce pdfs without the compressed block, having checked that's the only thing that cairo uses that variable for, CURRENTLY: https://git.launchpad.net/ubuntu/+source/fenics-dolfinx/tree/debian/rules#n166 I tried to avoid this, bug given it's the second shift I encounter this, I decided to go ahead. I hope doxygen gets updated to 1.9.7+ soon. # dolfin-mpc uninstallable packages $ sudo apt install libdolfinx-mpc0.6 -t mantic-proposed libdolfinx-mpc0.6 : Depends: libbasix0.6 (>= 0.6.0) but it is not installable Depends: libdolfinx-real0.6 (>= 1:0.6.0) but it is not installable E: Unable to correct problems, you have held broken packages. Needed a rebuild with fenics-basix 0.6, done: https://launchpad.net/ubuntu/+source/dolfinx-mpc/0.6.1-3build1 # fenics saga Many issues around this ecosystem. One looked simple, and was just a src:mshr rebuild. But that's an FTBFS with the newer src:cgal 5.6 (https://bugs.launchpad.net/ubuntu/+source/mshr/+bug/2030809). Checked upstream, and project looks dead. Filed a bug with debian (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1043307), hoping for the best, and they fixed it in https://launchpad.net/ubuntu/+source/mshr/2019.2.0~git20200924.c27eb18+dfsg1-10 # rust-sequoia-ipc FTBFS Troubleshooting led to an ftbfs in src:rust-nettle-sys, which was caused by a src:rust-bindgen bug caused by LLVM-16. https://bugs.launchpad.net/ubuntu/+source/rust-sequoia-ipc/+bug/2030886 Applied the upstream patch to https://launchpad.net/ubuntu/+source/rust-bindgen/0.60.1-2ubuntu1 and uploaded, it's going through migration. Debian is unaffected because, at the time of this troubleshooting, they were still using LLVM-14. I did file https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1043374, though. I subscribed to rust-bindgen in debian to be notified whenever they update or apply the patch, so this package can become a sync again. If you see rust-* tests failing and using rust-bindgen without the fix, retrigger them with rust-bindgen from mantic-proposed. I did so for a few and they became green. # rust-bindgen (my upload) migration It's blocked on an arm64-only dep8 failure in rust-loopdev/0.4.0-3: 819s detach_a_backing_file_default stdout 819s thread 'detach_a_backing_file_default' panicked at 'assertion failed: `(left == right)` 819s left: `0`, 819s right: `1`: there should be no loopback devices mounted', tests/integration_test.rs:176:5 I spent the whole afternoon today trying to get an arm64 mantic VM to troubleshoot this (it has to be a VM), but that was very difficult. canonistack isn't an option. My raspberry pi4 fails to launch a lxd VM (similar to https://github.com/canonical/lxd/issues/7191, but disabling secureboot didn't help: ~LXD is investigating). An arm64 MAAS server I have access to is a lottery to find a node that deploys, but eventually I got one that deployed, only to find out the error doesn't happen there. The only place where I reproduced it is in a PPA with the Canonical DEP8 infrastructure, which means the troubleshoot cycle is upload to ppa, wait for build/publish, trigger dep8, wait, repeat, so I didn't get very far in the time that was left. My thought is that perhaps in our arm64 dep8 vms, /dev/loop3 (hardcoded in the failing test) is perhaps taken, but changing it to another one didn't help. Then perhaps there is the 100ms sleep that the test has and that we might have to tweak. But that's just a
Re: Amendment: Update on reducing initramfs size and speed up
On Fri, 2023-08-11 at 19:32 +, Benjamin Drung wrote: > On Mon, 2023-07-31 at 22:49 +, Benjamin Drung wrote: > > On Mon, 2023-07-31 at 20:45 +0100, Dimitri John Ledkov wrote: > > > On Mon, 31 Jul 2023 at 20:41, Benjamin Drung wrote: > > > > > > > > On Thu, 2023-07-27 at 11:51 +1200, Michael Hudson-Doyle wrote: > > > > > > > > > > > > > > > On Thu, 27 Jul 2023 at 09:21, Benjamin Drung > > > > > wrote: > > > > > > On Wed, 2023-07-26 at 17:53 +0200, Benjamin Drung wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > A few weeks ago, I posted an idea how to reduce the initramfs size > > > > > > > and > > > > > > > speed up the generation: > > > > > > > > > > > > > > https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html > > > > > > > > > > > > > > This post sparked a lively discussion. The initial idea was > > > > > > > ditched for > > > > > > > a better solution: mkinitramfs will put all compressed files > > > > > > > (kernel > > > > > > > modules and firmware files) into a cpio archive that is not > > > > > > > compressed > > > > > > > (because compressing compressed files makes no sense). All other > > > > > > > files > > > > > > > will be added to a cpio archive that gets compressed. As next > > > > > > > steps, the > > > > > > > kernel modules and firmware files need to be shipped compressed. > > > > > > > > > > > > > > After several iterations for the implementation and review by > > > > > > > Daves > > > > > > > Jones, I just uploaded initramfs-tools 0.142ubuntu8 to mantic that > > > > > > > puts > > > > > > > compressed kernel modules and firmware files in an uncompressed > > > > > > > cpio > > > > > > > (https://launchpad.net/bugs/2028567). > > > > > > > > > > > > > > I created/updated the follow-up tickets and added my patches to > > > > > > > them: > > > > > > > > > > > > > > Ship kernel modules Zstd compressed > > > > > > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568 > > > > > > > > > > > > > > compress firmware in /lib/firmware > > > > > > > https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1942260 > > > > > > > > > > > > > > And without further ado, here come the benchmark results: > > > > > > > > > > > > > > The benchmarks were done either on an AMD Ryzen 7 5700G with > > > > > > > schroot and > > > > > > > overlay on tmpfs or on the hardware mentioned. All tests were > > > > > > > running > > > > > > > the latest Ubuntu mantic development release. > > > > > > > > > > > > > > * minimal: schroot with linux-image-generic initramfs-tools zstd > > > > > > > * full: minimal + busybox-initramfs cryptsetup-initramfs > > > > > > >isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme- > > > > > > > spinner > > > > > > > * nvidia: full + linux-headers-generic nvidia-driver-525 > > > > > > > * nvidia fw: nvidia + compressed /lib/firmware/nvidia/525.125.06/ > > > > > > > * VisionFive 2: VisionFive 2 RISC-V board > > > > > > > * RPi Zero 2: Raspberry Pi Zero 2 ARM board (running armhf) > > > > > > > > > > > > > > "next" means using kernel/firmware/initramfs from ppa:bdrung/ppa > > > > > > > i.e. > > > > > > > * initramfs-tools 0.142ubuntu7bd4 > > > > > > > * linux 6.3.0-7.7bd2 > > > > > > > * linux-firmware 20230629.gitee91452d-0ubuntu1bd1 > > > > > > > > > > > > > > > >| build | size | uncompressed > > > > > > > > size | > > > > > > > > > test | time| in bytes | in MiB | in bytes | in > > > > > > > > MiB | > > > > > > > > > |-|---||--- > > > > > > > > -| > > > > > > > > > minimal| 4.30 s | 66701818 | 63.6 | 224087608 | > > > > > > > > 213.7 | > > > > > > > > > minimal next | 4.54 s | 59935186 | 57.2 | 67738810 | > > > > > > > > 64.6 | > > > > > > > > > full | 7.15 s | 118007038 | 112.5 | 387976843 | > > > > > > > > 370.0 | > > > > > > > > > full next | 7.29 s | 106937908 | 102.0 | 128610985 | > > > > > > > > 122.7 | > > > > > > > > > nvidia | 7.04 s | 209200523 | 199.5 | 513554279 | > > > > > > > > 489.8 | > > > > > > > > > nvidia next| 7.21 s | 195246287 | 186.2 | 235288174 | > > > > > > > > 224.4 | > > > > > > > > > nvidia fw next | 7.16 s | 191329102 | 182.5 | 213078234 | > > > > > > > > 203.2 | > > > > > > > > > VisionFive 2 | 142.9 s | 121895035 | 116.2 | 411160836 | > > > > > > > > 392.1 | > > > > > > > > > VF 2 next | 126.7 s | 111651453 | 106.5 | 134120804 | > > > > > > > > 127.9 | > > > > > > > > > RPi Zero 2 | 109.5 s | 39803044 | 40.0 | 69592789 | > > > > > > > > 66.4 | > > > > > > > > > RPi Zero 2 ² | 103.5 s | 39804499 | 40.0 | 69592789 | > > > > > > > > 66.4 | > > > > > > > > > RPi Zero 2 next| 101.2 s | 31463352 | 30.0 | 41145762 | > > > > > > > > 39.2 | > > > > > > > > > > > > > > ² Updated initramfs-tools (but no compressed modules or firmware) > > > > > > > > > > > > > > The build time was averaged over five
Re: Amendment: Update on reducing initramfs size and speed up
On Mon, 2023-07-31 at 22:49 +, Benjamin Drung wrote: > On Mon, 2023-07-31 at 20:45 +0100, Dimitri John Ledkov wrote: > > On Mon, 31 Jul 2023 at 20:41, Benjamin Drung wrote: > > > > > > On Thu, 2023-07-27 at 11:51 +1200, Michael Hudson-Doyle wrote: > > > > > > > > > > > > On Thu, 27 Jul 2023 at 09:21, Benjamin Drung > > > > wrote: > > > > > On Wed, 2023-07-26 at 17:53 +0200, Benjamin Drung wrote: > > > > > > Hi all, > > > > > > > > > > > > A few weeks ago, I posted an idea how to reduce the initramfs size > > > > > > and > > > > > > speed up the generation: > > > > > > > > > > > > https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html > > > > > > > > > > > > This post sparked a lively discussion. The initial idea was > > > > > > ditched for > > > > > > a better solution: mkinitramfs will put all compressed files > > > > > > (kernel > > > > > > modules and firmware files) into a cpio archive that is not > > > > > > compressed > > > > > > (because compressing compressed files makes no sense). All other > > > > > > files > > > > > > will be added to a cpio archive that gets compressed. As next > > > > > > steps, the > > > > > > kernel modules and firmware files need to be shipped compressed. > > > > > > > > > > > > After several iterations for the implementation and review by > > > > > > Daves > > > > > > Jones, I just uploaded initramfs-tools 0.142ubuntu8 to mantic that > > > > > > puts > > > > > > compressed kernel modules and firmware files in an uncompressed > > > > > > cpio > > > > > > (https://launchpad.net/bugs/2028567). > > > > > > > > > > > > I created/updated the follow-up tickets and added my patches to > > > > > > them: > > > > > > > > > > > > Ship kernel modules Zstd compressed > > > > > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568 > > > > > > > > > > > > compress firmware in /lib/firmware > > > > > > https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1942260 > > > > > > > > > > > > And without further ado, here come the benchmark results: > > > > > > > > > > > > The benchmarks were done either on an AMD Ryzen 7 5700G with > > > > > > schroot and > > > > > > overlay on tmpfs or on the hardware mentioned. All tests were > > > > > > running > > > > > > the latest Ubuntu mantic development release. > > > > > > > > > > > > * minimal: schroot with linux-image-generic initramfs-tools zstd > > > > > > * full: minimal + busybox-initramfs cryptsetup-initramfs > > > > > >isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme- > > > > > > spinner > > > > > > * nvidia: full + linux-headers-generic nvidia-driver-525 > > > > > > * nvidia fw: nvidia + compressed /lib/firmware/nvidia/525.125.06/ > > > > > > * VisionFive 2: VisionFive 2 RISC-V board > > > > > > * RPi Zero 2: Raspberry Pi Zero 2 ARM board (running armhf) > > > > > > > > > > > > "next" means using kernel/firmware/initramfs from ppa:bdrung/ppa > > > > > > i.e. > > > > > > * initramfs-tools 0.142ubuntu7bd4 > > > > > > * linux 6.3.0-7.7bd2 > > > > > > * linux-firmware 20230629.gitee91452d-0ubuntu1bd1 > > > > > > > > > > > > > >| build | size | uncompressed > > > > > > > size | > > > > > > > > test | time| in bytes | in MiB | in bytes | in > > > > > > > MiB | > > > > > > > > |-|---||--- > > > > > > > -| > > > > > > > > minimal| 4.30 s | 66701818 | 63.6 | 224087608 | > > > > > > > 213.7 | > > > > > > > > minimal next | 4.54 s | 59935186 | 57.2 | 67738810 | > > > > > > > 64.6 | > > > > > > > > full | 7.15 s | 118007038 | 112.5 | 387976843 | > > > > > > > 370.0 | > > > > > > > > full next | 7.29 s | 106937908 | 102.0 | 128610985 | > > > > > > > 122.7 | > > > > > > > > nvidia | 7.04 s | 209200523 | 199.5 | 513554279 | > > > > > > > 489.8 | > > > > > > > > nvidia next| 7.21 s | 195246287 | 186.2 | 235288174 | > > > > > > > 224.4 | > > > > > > > > nvidia fw next | 7.16 s | 191329102 | 182.5 | 213078234 | > > > > > > > 203.2 | > > > > > > > > VisionFive 2 | 142.9 s | 121895035 | 116.2 | 411160836 | > > > > > > > 392.1 | > > > > > > > > VF 2 next | 126.7 s | 111651453 | 106.5 | 134120804 | > > > > > > > 127.9 | > > > > > > > > RPi Zero 2 | 109.5 s | 39803044 | 40.0 | 69592789 | > > > > > > > 66.4 | > > > > > > > > RPi Zero 2 ² | 103.5 s | 39804499 | 40.0 | 69592789 | > > > > > > > 66.4 | > > > > > > > > RPi Zero 2 next| 101.2 s | 31463352 | 30.0 | 41145762 | > > > > > > > 39.2 | > > > > > > > > > > > > ² Updated initramfs-tools (but no compressed modules or firmware) > > > > > > > > > > > > The build time was averaged over five runs. > > > > > > > > > > > > > > improvement | build time | size | uncompressed size | > > > > > > > > --|||---| > > > > > > > > minimal | 105.6 % | 89.9 % | 30.2 % | > > > > > > > > full
