Re: Revisiting default initramfs compression
On Thu, 10 Mar 2022 at 02:24, Dave Jones wrote: > Hi Michael (and others), > > Julian's summarised this near perfectly, but I'll try and add a little > detail from the data I've gathered [1] (with others' generous help, in > particular Heinrich for the RISC-V bits): > > On Wed, Mar 09, 2022 at 09:46:19AM +0100, Julian Andres Klode wrote: > >On Wed, Mar 09, 2022 at 02:10:57PM +1300, Michael Hudson-Doyle wrote: > [snip] > >> > some time ago, the default compressor for initramfs was changed > >> > from lz4 -9 to zstd -19. This caused significant problems: > >> > > >> > >> Exactly three months later... we still haven't taken any action on this. > >> Time to do something! > > Agreed! > > >> I have a few questions below but tl;dr: unless there are immediate > >> objections, I'm going to make a change to initramfs-tools to allow the > >> compression level to be configured and set the default to 12 for zstd. > > > >So xypron had a patch to change the default level to 9 for sponsoring > >out for a couple of months now (no idea how that level came up). > > > >We pushed back on that as it does not account for low-memory systems > >which we need to take care of as well. > > Yes, zstd -12 is not safe on low memory systems. In particular it fails > to even run successfully on an otherwise entirely idle and unloaded Pi > Zero 2 (which, on arm64, has a little more than 200MB of RAM free at > runtime, whilst zstd -T0 -12 on a PC requested ~415MB resident at > runtime). > > In fact, I'm not convinced *any* of zstd's levels are actually useful on > a machine with as limited RAM as the Zero 2 (or 3A+) are. For example, > with ~200MB of RAM free, if the user is running a daemon that eats, say, > 100MB resident and we start up a compressor that eats 50MB (as zstd -T0 > does at level -1) we stand a fair chance of pushing the daemon into OOM. > Fair enough. > Now, in practice this doesn't actually matter right now as I've already > overridden initramfs' default to lz4 in ubuntu-raspi-settings, however I > think there are adjustments that should be made there too (and there is > the question of whether this is relevant for, say, minimal memory cloud > instances). > > >We then postponed any implementation to after a discussion in > >Frankfurt. > > > >I think the summary from the Frankfurt discussion was: > > > >- lz4 -1 is the right choice for low-memory systems > >- if you have more memory, zstd -1 becomes the best choice > >- pigz is outperforming both a bunch of times > > > >But that's really for waveform to share. > > Just to clarify a couple of things here: > > Firstly I actually think lz4 -2 is probably the ideal level for that > compressor. There's a large difference in compression performance > between lz4 -1 and lz4 -2 across all platforms tested, but no difference > in memory usage, and only a minimal increase in compression & > decompression time. However, lz4 is currently configured to use level -9 > which takes a considerable amount of extra time for little to no gain in > compression performance (at least with our initramfs inputs anyway). > > On machines with more generous RAM allowances, zstd -T0 -1 does appear > to be the ideal. The incremental gains in compression at higher levels > are outweighed by the extra time spent compressing (i.e. for our > initramfs inputs at least, the extra time spent on the compression is > not gained back on reading the compressed data at I/O speeds typical for > their respective platforms). > Moving all the way to -1 does feel like quite a change, but well. I'm not opposed to it. > [snipped some data] > > At this point, if you want some data to play with I'd highly recommend > cloning the following repo and following the instructions in the README: > > https://github.com/waveform80/compression > > >> Are people really going to run an arm64 userland on a Pi 0? > > I have no specific figures on this, but given the *vast* majority of > Ubuntu downloads for the Pi skew towards arm64 (about 10:1 last time I > looked) I think it's a safe assumption that at least some do. > > >> Any "real" solution for pi 0 has to involve doing at least the bulk > >> of the compression not on the pi, there's no real way around that. > >> Which is something we should do, but realistically it's not happening > >> for jammy. > > I'm not so sure about that. On a Pi Zero 2 under the arm64 arch, lz4 -2 > takes ~8MB of resident RAM, and ~1.9s of time to compress the initramfs > to approximately half its size. If we're willing to wait a little > longer, lz4 -4 takes 8.7s (and no extra RAM) to get it down to 40% of > its size. The current -9 level is pointless, however, taking 21s (and no > extra RAM) to get to 39%. > Interesting. > [snip] > >> I think this broadly makes sense. I'll notice that currently > >> initramfs-tools doesn't allow tuning the compression level at all :/ > >> Probably fairly routine to add support for that though. > > This would indeed be nice, as would the option to support "none"
Re: Proposal: revert recent debianutils changes for Jammy
On 2022-01-24 15:53, Gunnar Hjalmarsson wrote: On 2022-01-24 15:27, Dimitri John Ledkov wrote: Longer term it would be easier if we do a merge from debian now to preserve just the `export NO_PKG_MANGLE=1` delta (if that is still required), Setting that variable was my idea. The reason is that without it, the translation files are stripped at build time and imported to Launchpad. But we have no use for those particular translations; they are for man pages only, and the language packs don't deal with man pages. (Previously we have tricked the translators to spend time with those strings unnecessarily.) If that's the only remaining Debian/Ubuntu delta, I think we should try to have Debian set that variable. It makes no difference to Debian, but is useful for us. Then we can sync again. I just requested that Debian sets the NO_PKG_MANGLE variable. https://bugs.debian.org/1006970 If approved we can force sync when considered appropriate. -- Gunnar Hjalmarsson https://launchpad.net/~gunnarhj -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
+1 maintenance report
I was on +1 for Monday and Tuesday. Here are my notes. # mariadb # Debbug 1005950 was tracking openssl 3 and mariadb-10.6. The discussion there is that OpenSSL is officially part of ver 10.8 and 10.9+, but some backport work was done and uploaded to Experimental. Debian upstream is working on this (thanks Otto), but there is some additional test flakiness, so I have proposed skipping another test in that test suite. LP: #1964045 and MR on Salsa, under discussion. # qt6-base # Michael Hudson-Doyle outlined a plan for qt6-base last week, I just went and did it. LP: #1964273, MR on Salsa, debbug open. # other things I looked at quickly # * tpm2-tss-engine - left alone, already on the removal list - LP: #1959414 * opa-ff - left alone, LP: #1960841 - to quote Simon Chopin: "Having linux-tools-common Providing linux-cpupower would allow us to keep the aforementioned packages in sync with Debian without introducing a delta." * surf - had an autopkgtest failure on arm64. Here are the log differences between good and bad cases. A new version of webkit2gtk has been uploaded so unsure if this is still relevant. bad: Could not determine the accessibility bus address Cannot get default EGL display: EGL_BAD_PARAMETER Cannot create EGL context: invalid display (last error: EGL_SUCCESS) Too few characters detected (0) good: Estimating resolution as 123 Text has 1763 characters and 6 detected errors * softhsm2 - digging into the test failures shows many instances of the following - TestsNoPINInitBase.cpp 124 setUp() failed. There is some talk of openssl3 work upstream https://github.com/opendnssec/SoftHSMv2/pull/633 -Dan -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Call for nominations: Developer Membership Board
On Wed, Mar 9, 2022 at 9:19 AM Sebastien Bacher wrote: > > Hey Robie and DMB members, > > Le 01/03/2022 à 16:51, Robie Basak a écrit : > > Candidates must expect to be able to attend the majority of DMB > > meetings. Currently these take place on IRC, are scheduled on alternate > > Mondays with each meeting alternating between 1600 UTC and 1900 UTC, and > > last around an hour. > > Following the recent emails stating that we don't have enough candidate > I'm going to drop a note about ^ > > I was pondering sending my application, I'm busy but I think it's > important that we have a function DMB, but that 'must expect' statement > convinced me to not. I will comment here only to say that statement does not reflect my opinion at all, and was not discussed with the other DMB members before Robie sent it. I have absolutely no idea whether that statement is simply Robie's opinion, or actual policy of the DMB. Personally I do not think new members should be expected to attend the majority of DMB meetings; I think it's 100% fine for members to participate asynchronously using the mailing list (or proxy voting, but with ML voting, I don't really see the need for proxy voting). > I try to lock the 17h30-20h to be able to have some family time and > that's not something I'm wanting to compromise on at this point. > > I might not be the only one in that situation, since we are short on > candidate maybe it would help to try to be less rigid on that > requirement? (being open to different times? allow members to skip and > vote via email? ...) > > Cheers, > Sebastien Bacher > > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Call for nominations: Developer Membership Board
Hey Robie and DMB members, Le 01/03/2022 à 16:51, Robie Basak a écrit : Candidates must expect to be able to attend the majority of DMB meetings. Currently these take place on IRC, are scheduled on alternate Mondays with each meeting alternating between 1600 UTC and 1900 UTC, and last around an hour. Following the recent emails stating that we don't have enough candidate I'm going to drop a note about ^ I was pondering sending my application, I'm busy but I think it's important that we have a function DMB, but that 'must expect' statement convinced me to not. I try to lock the 17h30-20h to be able to have some family time and that's not something I'm wanting to compromise on at this point. I might not be the only one in that situation, since we are short on candidate maybe it would help to try to be less rigid on that requirement? (being open to different times? allow members to skip and vote via email? ...) Cheers, Sebastien Bacher -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Revisiting default initramfs compression
Hi Michael (and others), Julian's summarised this near perfectly, but I'll try and add a little detail from the data I've gathered [1] (with others' generous help, in particular Heinrich for the RISC-V bits): On Wed, Mar 09, 2022 at 09:46:19AM +0100, Julian Andres Klode wrote: On Wed, Mar 09, 2022 at 02:10:57PM +1300, Michael Hudson-Doyle wrote: [snip] > some time ago, the default compressor for initramfs was changed > from lz4 -9 to zstd -19. This caused significant problems: > Exactly three months later... we still haven't taken any action on this. Time to do something! Agreed! I have a few questions below but tl;dr: unless there are immediate objections, I'm going to make a change to initramfs-tools to allow the compression level to be configured and set the default to 12 for zstd. So xypron had a patch to change the default level to 9 for sponsoring out for a couple of months now (no idea how that level came up). We pushed back on that as it does not account for low-memory systems which we need to take care of as well. Yes, zstd -12 is not safe on low memory systems. In particular it fails to even run successfully on an otherwise entirely idle and unloaded Pi Zero 2 (which, on arm64, has a little more than 200MB of RAM free at runtime, whilst zstd -T0 -12 on a PC requested ~415MB resident at runtime). In fact, I'm not convinced *any* of zstd's levels are actually useful on a machine with as limited RAM as the Zero 2 (or 3A+) are. For example, with ~200MB of RAM free, if the user is running a daemon that eats, say, 100MB resident and we start up a compressor that eats 50MB (as zstd -T0 does at level -1) we stand a fair chance of pushing the daemon into OOM. Now, in practice this doesn't actually matter right now as I've already overridden initramfs' default to lz4 in ubuntu-raspi-settings, however I think there are adjustments that should be made there too (and there is the question of whether this is relevant for, say, minimal memory cloud instances). We then postponed any implementation to after a discussion in Frankfurt. I think the summary from the Frankfurt discussion was: - lz4 -1 is the right choice for low-memory systems - if you have more memory, zstd -1 becomes the best choice - pigz is outperforming both a bunch of times But that's really for waveform to share. Just to clarify a couple of things here: Firstly I actually think lz4 -2 is probably the ideal level for that compressor. There's a large difference in compression performance between lz4 -1 and lz4 -2 across all platforms tested, but no difference in memory usage, and only a minimal increase in compression & decompression time. However, lz4 is currently configured to use level -9 which takes a considerable amount of extra time for little to no gain in compression performance (at least with our initramfs inputs anyway). On machines with more generous RAM allowances, zstd -T0 -1 does appear to be the ideal. The incremental gains in compression at higher levels are outweighed by the extra time spent compressing (i.e. for our initramfs inputs at least, the extra time spent on the compression is not gained back on reading the compressed data at I/O speeds typical for their respective platforms). [snipped some data] At this point, if you want some data to play with I'd highly recommend cloning the following repo and following the instructions in the README: https://github.com/waveform80/compression Are people really going to run an arm64 userland on a Pi 0? I have no specific figures on this, but given the *vast* majority of Ubuntu downloads for the Pi skew towards arm64 (about 10:1 last time I looked) I think it's a safe assumption that at least some do. Any "real" solution for pi 0 has to involve doing at least the bulk of the compression not on the pi, there's no real way around that. Which is something we should do, but realistically it's not happening for jammy. I'm not so sure about that. On a Pi Zero 2 under the arm64 arch, lz4 -2 takes ~8MB of resident RAM, and ~1.9s of time to compress the initramfs to approximately half its size. If we're willing to wait a little longer, lz4 -4 takes 8.7s (and no extra RAM) to get it down to 40% of its size. The current -9 level is pointless, however, taking 21s (and no extra RAM) to get to 39%. [snip] I think this broadly makes sense. I'll notice that currently initramfs-tools doesn't allow tuning the compression level at all :/ Probably fairly routine to add support for that though. This would indeed be nice, as would the option to support "none" as a compression level (I opine a little more on this in the conclusion section of the analysis notebook in the repo linked above). Last time I looked at this, it didn't look difficult to add the ability to provide an entirely custom command for compression via the COMPRESS option. In fact, I have the feeling it was *intended* to permit this, but somewhere along
Re: Call for nominations: Developer Membership Board
On Tue, Mar 8, 2022 at 9:44 PM Seth Arnold wrote: > > On Tue, Mar 08, 2022 at 10:37:07AM -0500, Dan Streetman wrote: > > In any case, all this formality really is exhausting and I don't care > > to pursue it, since you seem to be saying that I can't call for a DMB > > I've been having this thought a lot the last few years. > > A lot of the Ubuntu structures feel like they may have had a place in the > early days, when there were thousands of enthusiasts all contributing at > once and trying to coordinate how they were going to do all that. > > Now it feels like we've got the ossified remains of a lot of committees > and teams and boards and I just don't know how much of it is still > relevant to the Ubuntu of today. Every piece serves a role so it's not > like I can just point to any one specific thing and say we ought to scrap > it, but I remember watching what felt like dozens of DMB meetings where > not enough members showed up, not enough members voted, and the poor > applicants could go weeks or months (or more?) without an answer. > > That kills enthusiasm. > > I'm sorry that I don't have a solution to recommend but I think I'd like > to suggest that we try stepping back from bureaucracy when opportunities > arise. Thanks Seth, I think that summarizes my frustrations extremely well. I don't know if I have the energy to step back into working on the DMB, but I am trying to pre-emptively address this with my proposed charter for the backports team (in case you haven't seen it): https://wiki.ubuntu.com/UbuntuBackports/Charter Essentially, what I see as absolutely critical to Ubuntu teams is for the team members to focus on the team *mission*, and appoint *one* person as administrator to handle all the administrivia/bureaucracy. The *mission* of the DMB has become completely and totally lost in the needless administration of rules, and that only serves to make the problem worse as the (volunteer!) team members lose interest. I think at this point, the DMB desperately needs a charter, so these team-wide discussions about administration can end, and an administrator to take responsibility (and have authority) to make sure the team makes progress on its mission. > > Thanks > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Revisiting default initramfs compression
On Wed, 9 Mar 2022, Julian Andres Klode wrote: > On Wed, Mar 09, 2022 at 02:10:57PM +1300, Michael Hudson-Doyle wrote: > > On Thu, 9 Dec 2021 at 06:13, Julian Andres Klode > > > > wrote: > > > changed from lz4 -9 to zstd -19. This caused significant problems: > > change to initramfs-tools to allow the compression level to be configured > xypron had a patch to change the default level to 9 ... > think the summary from the Frankfurt discussion was: ... I live in Frankfurt, but wasn't there...; otherwise some observations could have been made: Choosing the best (size) of hammer, to nail in a screw, is un-useful. Using four screwdrivers with four small screws seems quick; but only if four screws are an acceptable engineering solution. [Pigz chops the stream into independent (128 kB) blocks and round-robins these per CPU/thread + concats() afterwards]. When the output is made of independent concatentated chunks; each chunk *may* have its own compression level; ...chosen from reasonable heuristics [file size + content type (+ available local resources)]. With the same input stream, && the same heuristics, this will always generate the same (identical) outputs. Which means the (previous) outputs can be cached and (re-)used, or ...precalculated in the background. Which means initramfs generation *could* be turned into a ~1 second "cat *.cpio.gz > initramfs.cpio.gz" operation. -Paul -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Revisiting default initramfs compression
On Wed, Mar 09, 2022 at 02:10:57PM +1300, Michael Hudson-Doyle wrote: > On Thu, 9 Dec 2021 at 06:13, Julian Andres Klode > wrote: > > > Hi all, > > > > some time ago, the default compressor for initramfs was changed > > from lz4 -9 to zstd -19. This caused significant problems: > > > > Exactly three months later... we still haven't taken any action on this. > Time to do something! > > I have a few questions below but tl;dr: unless there are immediate > objections, I'm going to make a change to initramfs-tools to allow the > compression level to be configured and set the default to 12 for zstd. So xypron had a patch to change the default level to 9 for sponsoring out for a couple of months now (no idea how that level came up). We pushed back on that as it does not account for low-memory systems which we need to take care of as well. We then postponed any implementation to after a discussion in Frankfurt. I think the summary from the Frankfurt discussion was: - lz4 -1 is the right choice for low-memory systems - if you have more memory, zstd -1 becomes the best choice - pigz is outperforming both a bunch of times But that's really for waveform to share. As a rule we said that we want compression to not take more than 10% of available memory, so that we don't eat into whatever applications you run too much. We also need to compare sizes to what we have now and adjust the size of /boot accordingly. > > - it is very slow > > - it uses a lot of memory > > > > The former is a problem for everyone, the latter means that > > zstd just crashes on a Pi Zero. > > > > This is an analysis of what we have in terms of time spent, > > memory spent, and file size achieved, and where we can > > go from here. > > > > # Comparison of different compression levels > > > > ## Desktop (ThinkPad T480s, jammy) > > > > levelusertime elapsed memory fileSize > > lz4 9.65s11.09s12M 64M > > -1 5.69s 6.99s24M 57M > > -6 12.59s 8.58s99M 47M > > -1219.85s10.82s 249M 41M > > -1971.29s26.95s 519M 35M > > > > -> I believe that somewhere around -12 is a decent > >compromise between size and speed. > > > > I would agree that it's certainly better than -19. > > > > ## Pi 4 (arm64, focal) > > > > Times have been measured for mkinitramfs only. A full > > update-initramfs call spends much more time copying > > some firmware bits to boot partition with flash-kernel > > > > levelusertime elapsed memory fileSize > > lz421.10s64.85s21M 29M > > -1 13.73s44.55s21M 27M > > -6 26.07s49.09s91M 24M > > -1248.18s54.67s 203M 22M > > -19 130.07s92.80s 350M 20M > > > > -> 6 is essentially free if the Pi 4 is idle. Nice. > > -> -6 is still 20% of total RAM of a Pi 0 > > > > Are people really going to run an arm64 userland on a Pi 0? > > Any "real" solution for pi 0 has to involve doing at least the bulk of the > compression not on the pi, there's no real way around that. Which is > something we should do, but realistically it's not happening for jammy. You are right that armhf might be more useful there. > > > > -> There's no meaningful difference between -6 and -12 > >in terms of time elapsed. -6 uses 116% CPU, -12 uses > >145% CPU. > > > > ## Adaptive compression > > > > zstd also supports adaptive compression, compressing as hard as > > it can while not impacting I/O speed. So hardware with slow I/O > > like a Pi would compress harder to avoid idling. > > > > This is somewhat suboptimal with recent update-initramfs though, > > as it first writes the cpio archive to the disk and then compresses > > it rather than doing it in a pipe where that would be more > > meaningful. > > > > Question: Does zstd --adapt adapt to memory available? > > > > While attractive, this does feel a bit risky. We want to be able to make > reasonable predictions about the size of the initrd. > > > > # Way(s) forward > > > > To remedy the issue the proposal is to build with > > > > - zstd -1 on hardware with 512 MB or less memory > > - zstd between -1 and -19 on other hardware > > - zstd -19 during image building > > > > I think this broadly makes sense. I'll notice that currently > initramfs-tools doesn't allow tuning the compression level at all :/ > Probably fairly routine to add support for that though. > > > > Finding the right level between -1 and -19 is hard. The more > > cores you have, the less penalty you pay for higher level. > > > > You suggested -12 above. How about we try that to start with? > > Do you have an idea how to detect "512 MB or less memory"? Parse /proc/meminfo / free output. > > > Going for adaptive compression would remove the guess work, but > > will result in larger images on faster machines. Maybe that's > > fine, though - they probably have more space on /boot anyway? > > > > If we want to aim for 5%
