Re: Small rant: installer environment size
* Adam Williamson: > On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote: >> * Adam Williamson: >> >> > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is >> > 224M uncompressed. A quick test just compressing the file with xz on my >> > system shows it compresses to around 11M, though, so that's probably >> > all it adds up to after compression (the image is an xz-compressed >> > squashfs) >> >> Isn't the compression block-based? I think it would be interesting to >> measure the image size with the file removed. > > I'll try it tomorrow, it's not too hard. Have you posted the outcome of the experiment somewhere? Thanks, Florian ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Hi, On 12/8/22 06:58, Peter Robinson wrote: On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson wrote: Hi folks! Today I woke up and found https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me down a bit of an "installer environment size" rabbit hole. As of today, with that new dep in webkitgtk, Rawhide's network install images are 703M in size. Here's a potted history of network install image sizes: Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) Fedora 13: 208M Fedora 17: 162M (last "old UI") Fedora 18: 294M (first "new UI") Fedora 23: 415M Fedora 28: 583M Fedora 33: 686M Fedora 37: 665M Fedora Rawhide: 703M The installer does not really do much more in Rawhide than it did in FC8. Even after the UI rewrite in F18, we were only at 294M. Now the image is well over 2x as big and does...basically the same. Why does this matter? Well, the images being large is moderately annoying in itself just in terms of transfer times and so on. But more importantly, AIUI at least, the entire installer environment is loaded into RAM at startup - it kinda has to be, we don't have anywhere else to put it. The bigger it is, the more RAM you need to install Fedora. The size of the installer environment (for which the size of the network install image is more or less a perfect proxy) is one of the two key factors in this, the other being how much RAM DNF uses during package install. So, I did a bit of poking about into *what* is taking up all that space. There's a variety of answers, but there's two major culprits: 1. firmware 2. yelp (which pulls in webkitgtk and its deps) I've been using du and baobab (the GNOME visual disk usage analyzer, which is great) to examine the filesystems, but I ran a couple of test builds to confirm these suspects, especially after the impact of compression (it's hard to check the *compressed* size of things in the installer environment directly). I did a scratch build of lorax which does not pull in firmware packages, and had openQA build a netinst using that lorax. It came out at 489M - 214M smaller than current netinsts, a size we last managed in Fedora 26. I did a scratch build of anaconda with its requirement of yelp dropped (which would break help pages), and built a netinst with that; it came out at 662M - 41M smaller than current images. I haven't run a combined test yet, but it ought to come out around 448M, around the size of Fedora 24. Even then we'd still be about 50% larger than the Fedora 18 image, for not really any added functionality. I've moaned about the sheer amount and size of firmware blobs in other forums before, but 214M compressed is *really* obnoxious. We must be able to do something to clean this up (further than it's already cleaned up - this is *after* we dropped low-hanging fruit like enterprise switch 'firmwares' and garbage like that; most of the remaining size seems to be huge amounts of probably-very-similar firmware files for AMD graphics adapters and Intel wireless adapters). I know some folks were trying to work on this (there was talk that we could drop quite a lot of files that would only be loaded by older kernels no longer in Fedora); any news on how far along that effort is? I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI. The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network. Ideas on how to solve that problem welcome. Ok, I have a couple ideas, but they start with the question, why do we need fully accelerated graphics for an installer (live image excepted) that works nearly as well in text mode? That gets the GPU firmware off the install ramdisk. Just being a bit more fine grained with the firmware package and only installing the pieces needed by the running machine shrinks could shrink the footprint too. Something that looks for kernel firmware load errors, and installs a package solves the issue of HW that has been dynamically added after the fact (of course disk/network card firmware would still be needed by the installer). Although, just doing per arch firmware shrinks it too. Both the x86 and arm64 packages are both 177M, and it seems unlikely my arm machine needs amd microcode, or that my amd needs the dpaa firmware or firmware specific to some arm SBCs. So, ideas, but then someone needs to
Re: Small rant: installer environment size
On Friday, December 9, 2022, Adam Williamson wrote: > On Fri, 2022-12-09 at 12:04 +0100, Florian Weimer wrote: > > * Richard W. M. Jones: > > > > > You only need network / wifi firmware blobs (although I'm sure they > > > are in themselves large) and then you can fetch anything else needed > > > for the hardware including graphics, right? > > > > I think you need graphics to set up wifi. > > Yeah, this is an awkward chicken-and-egg problem. Even if we assume > you're on a wired network, kernel modules generally - AIUI - try to > load the firmware once, on initial module load, and if they can't find > it, just give up, right? So we still have an ordering problem: how can > we delay the loading of modules that need firmware until the network is > up for us to be able to access the firmware files? > > Maybe I'm missing something that would help there, but it seems > tricky... > > Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, > ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is > another 6.4M and I *think* that's all wifi. There's a few other minor > ones, but that's a little over 100M of just wifi, with Intel by a huge > margin the worst offender. > > Does anyone know anyone we can talk to at Intel about this? It's pretty > obnoxious. > > In terms of what the other big space takers are in general: > > * amdgpu/ (AMD video cards) is ~20M > * intel/ (mainly Intel bluetooth) is ~15M [0] > * qed/ (some very high-end QLogic network cards) is ~10M [0] > * i915/ (Intel video firmware) is 8.4M > * mediatek/ is 7.7M [1] > * qcom/ is 7.3M > > Then it trails off from there. Just the wifi plus those 6 things are > around 170M, so the large majority of all the space taken. > > [0] No, we can't lose this - people install with Bluetooth > mice/keyboards > [1] For a quick win right now possibly we could assume nobody's going > to use one of those as the interface for a Fedora install and drop > that, not sure if it's a safe assumption > It's not given that AMD wifi is rebranded mediatek, meaning it will drop wifi for lots of newer AMD laptops. ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 09, 2022 at 12:04:24PM +0100, Florian Weimer wrote: > * Richard W. M. Jones: > > > You only need network / wifi firmware blobs (although I'm sure they > > are in themselves large) and then you can fetch anything else needed > > for the hardware including graphics, right? > > I think you need graphics to set up wifi. I long for old school text mode installers ... At least you knew that the tab key would always work. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 4:56 PM Adam Williamson wrote: > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > I've done a few passes, dropping a bunch of older firmware upstream > > that are no longer supported in any stable kernel release, also a > > bunch of de-dupe and linking of files rather than shipping of multiple > > copies of the same firmware. It's improved things a bit, unfortunately > > a lot of the dead firmware was tiny compared to say average modern > > devices like GPUs or WiFI. > > > > The problem with a lot of the firmware, and with the new nvidia "open > > driver" which shoves a lot of stuff into firmware in order to have an > > upstreamable driver apparently the firmwares there are going to be > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > even just install so I don't know how we can get around this and still > > have a device work enough to be able to install the needed firmware > > across the network. > > > > Ideas on how to solve that problem welcome. > > Sorry if this is way off, but - do we need the GPU firmwares to run a > graphical install on the fallback path, just using the framebuffer set > up by the firmware? How crazy would it be to just do that - ship the > installer env with no GPU firmware? That has crossed my mind, and with simpledrm that may be more straight forward now, but TBH it's not something I am skilled enough to deal with, nor have the resources to test, or actually care enough about, but the big GPU firmwares are now all split out so that should be much more straightforward for someone with the resources to investigate. ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Richard W. M. Jones: > You only need network / wifi firmware blobs (although I'm sure they > are in themselves large) and then you can fetch anything else needed > for the hardware including graphics, right? I think you need graphics to set up wifi. Thanks, Florian ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 11:49:16AM -0800, Adam Williamson wrote: > On Thu, 2022-12-08 at 20:23 +0100, drago01 wrote: > > The problem I see here is not the presence of the firmware on the > > image, > > but the fact that it seems to be loaded into memory despite not being > > used. > > This is the direction Daniel was thinking down. I'm waiting for someone > with more expertise to reply, but I suspect the reply is going to be > along the lines of "yes, we *can* do that, but it's somewhat tricky > work that involves thinking about lots of paths that aren't obvious, > and somebody would need to dedicate their time to working on that". Split install.img into install.img + firmware.img? I think we already have support for multiple images (I see requests for updates.img when watching httpd logs while doing network installs), so the split should be easy. The somewhat more tricky part is probably to figure whenever we need the firmware or not. take care, Gerd ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote: > I've moaned about the sheer amount and size of firmware blobs in other > forums before, but 214M compressed is *really* obnoxious. We must be > able to do something to clean this up (further than it's already > cleaned up - this is *after* we dropped low-hanging fruit like > enterprise switch 'firmwares' and garbage like that; most of the > remaining size seems to be huge amounts of probably-very-similar > firmware files for AMD graphics adapters and Intel wireless adapters). > I know some folks were trying to work on this (there was talk that we > could drop quite a lot of files that would only be loaded by older > kernels no longer in Fedora); any news on how far along that effort is? You only need network / wifi firmware blobs (although I'm sure they are in themselves large) and then you can fetch anything else needed for the hardware including graphics, right? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thursday, December 8, 2022, Daniel P. Berrangé wrote: > On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > > On Thursday, December 8, 2022, Adam Williamson < > adamw...@fedoraproject.org> > > wrote: > > > > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > > > > > I've done a few passes, dropping a bunch of older firmware upstream > > > > that are no longer supported in any stable kernel release, also a > > > > bunch of de-dupe and linking of files rather than shipping of > multiple > > > > copies of the same firmware. It's improved things a bit, > unfortunately > > > > a lot of the dead firmware was tiny compared to say average modern > > > > devices like GPUs or WiFI. > > > > > > > > The problem with a lot of the firmware, and with the new nvidia "open > > > > driver" which shoves a lot of stuff into firmware in order to have an > > > > upstreamable driver apparently the firmwares there are going to be > > > > 30+Mb each, is that they're needed to bring up graphics/network etc > to > > > > even just install so I don't know how we can get around this and > still > > > > have a device work enough to be able to install the needed firmware > > > > across the network. > > > > > > > > Ideas on how to solve that problem welcome. > > > > > > Sorry if this is way off, but - do we need the GPU firmwares to run a > > > graphical install on the fallback path, just using the framebuffer set > > > up by the firmware? How crazy would it be to just do that - ship the > > > installer env with no GPU firmware? > > > > > > That would be very crazy, as you will have a degraded user experience > > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are > a > > non issue for today's hardware. > > Please bear in mind the difference between bare metal and virtual > machines. The bare metal machine may have 32 GB of RAM, making a > 800 MB install image a non-issue. For a public cloud virtual machine > though, this could bump your VM sizing up 1 level from 2 GB quota > to a 4 GB RAM quota, with correspondingly higher price point. Now > most people probably don't run the installer in a public cloud, > preferring pre-built disk images. Even in a local machine though, > you may be using most of your 32 GB of RAM for other things (well > firefox/chrome), so allowing extra for the VM is not without > resource cost. If we could figure out a way to knock a few 100 MB > off the installer RAM requirements that is valuable. > > The problem I see here is not the presence of the firmware on the image, but the fact that it seems to be loaded into memory despite not being used. > With regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/ > dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org-o-https://www.instagram.com/ > dberrange :| > ___ > kernel mailing list -- kernel@lists.fedoraproject.org > To unsubscribe send an email to kernel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject. > org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/kernel@ > lists.fedoraproject.org > Do not reply to spam, report it: https://pagure.io/fedora- > infrastructure/new_issue > ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > On Thursday, December 8, 2022, Adam Williamson > wrote: > > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > > > I've done a few passes, dropping a bunch of older firmware upstream > > > that are no longer supported in any stable kernel release, also a > > > bunch of de-dupe and linking of files rather than shipping of multiple > > > copies of the same firmware. It's improved things a bit, unfortunately > > > a lot of the dead firmware was tiny compared to say average modern > > > devices like GPUs or WiFI. > > > > > > The problem with a lot of the firmware, and with the new nvidia "open > > > driver" which shoves a lot of stuff into firmware in order to have an > > > upstreamable driver apparently the firmwares there are going to be > > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > > even just install so I don't know how we can get around this and still > > > have a device work enough to be able to install the needed firmware > > > across the network. > > > > > > Ideas on how to solve that problem welcome. > > > > Sorry if this is way off, but - do we need the GPU firmwares to run a > > graphical install on the fallback path, just using the framebuffer set > > up by the firmware? How crazy would it be to just do that - ship the > > installer env with no GPU firmware? > > > That would be very crazy, as you will have a degraded user experience > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a > non issue for today's hardware. Please bear in mind the difference between bare metal and virtual machines. The bare metal machine may have 32 GB of RAM, making a 800 MB install image a non-issue. For a public cloud virtual machine though, this could bump your VM sizing up 1 level from 2 GB quota to a 4 GB RAM quota, with correspondingly higher price point. Now most people probably don't run the installer in a public cloud, preferring pre-built disk images. Even in a local machine though, you may be using most of your 32 GB of RAM for other things (well firefox/chrome), so allowing extra for the VM is not without resource cost. If we could figure out a way to knock a few 100 MB off the installer RAM requirements that is valuable. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thursday, December 8, 2022, Adam Williamson wrote: > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > I've done a few passes, dropping a bunch of older firmware upstream > > that are no longer supported in any stable kernel release, also a > > bunch of de-dupe and linking of files rather than shipping of multiple > > copies of the same firmware. It's improved things a bit, unfortunately > > a lot of the dead firmware was tiny compared to say average modern > > devices like GPUs or WiFI. > > > > The problem with a lot of the firmware, and with the new nvidia "open > > driver" which shoves a lot of stuff into firmware in order to have an > > upstreamable driver apparently the firmwares there are going to be > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > even just install so I don't know how we can get around this and still > > have a device work enough to be able to install the needed firmware > > across the network. > > > > Ideas on how to solve that problem welcome. > > Sorry if this is way off, but - do we need the GPU firmwares to run a > graphical install on the fallback path, just using the framebuffer set > up by the firmware? How crazy would it be to just do that - ship the > installer env with no GPU firmware? That would be very crazy, as you will have a degraded user experience (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a non issue for today's hardware. > Adam Williamson > Fedora QA > IRC: adamw | Twitter: adamw_ha > https://www.happyassassin.net > > ___ > desktop mailing list -- desk...@lists.fedoraproject.org > To unsubscribe send an email to desktop-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject. > org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/desktop@ > lists.fedoraproject.org > Do not reply to spam, report it: https://pagure.io/fedora- > infrastructure/new_issue > ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote: > Hi folks! Today I woke up and found > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me > down a bit of an "installer environment size" rabbit hole. snip > Why does this matter? Well, the images being large is moderately > annoying in itself just in terms of transfer times and so on. But more > importantly, AIUI at least, the entire installer environment is loaded > into RAM at startup - it kinda has to be, we don't have anywhere else > to put it. The bigger it is, the more RAM you need to install Fedora. > The size of the installer environment (for which the size of the > network install image is more or less a perfect proxy) is one of the > two key factors in this, the other being how much RAM DNF uses during > package install. Is there something that can be done to optimize the RAM usage, in spite of the large installer env size ? If we're installing off DVD media, it shouldn't be required to pull all of the content into RAM, since it can be fetched on demand from the media. IOW, 99% of the firmware never need leave the ISO, so shouldn't matter if firmware is GBs in size [1] if we never load it off the media. Same for languages, only the one we actually want to use should ever get into RAM. If we're installing off a network source, we need to pull content into RAM, but that doesn't mean we should pull everything in at once upfront. Is it possible to delay pulling in non-NIC firmware until we have a NIC configured, and just rely on the basic generic framebuffer setup by UEFI/BIOS until we get far enugh to pull in video card firmware ? For localization, is it possible to split the localization into per-language bundles, and delay loading off the network until we know what language we want to load, instead of pre-loading all languages ? With regards, Daniel [1] Yes, I know it matters for user media download size in reality -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson wrote: > > Hi folks! Today I woke up and found > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me > down a bit of an "installer environment size" rabbit hole. > > As of today, with that new dep in webkitgtk, Rawhide's network install > images are 703M in size. Here's a potted history of network install > image sizes: > > Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) > Fedora 13: 208M > Fedora 17: 162M (last "old UI") > Fedora 18: 294M (first "new UI") > Fedora 23: 415M > Fedora 28: 583M > Fedora 33: 686M > Fedora 37: 665M > Fedora Rawhide: 703M > > The installer does not really do much more in Rawhide than it did in > FC8. Even after the UI rewrite in F18, we were only at 294M. Now the > image is well over 2x as big and does...basically the same. > > Why does this matter? Well, the images being large is moderately > annoying in itself just in terms of transfer times and so on. But more > importantly, AIUI at least, the entire installer environment is loaded > into RAM at startup - it kinda has to be, we don't have anywhere else > to put it. The bigger it is, the more RAM you need to install Fedora. > The size of the installer environment (for which the size of the > network install image is more or less a perfect proxy) is one of the > two key factors in this, the other being how much RAM DNF uses during > package install. > > So, I did a bit of poking about into *what* is taking up all that > space. There's a variety of answers, but there's two major culprits: > > 1. firmware > 2. yelp (which pulls in webkitgtk and its deps) > > I've been using du and baobab (the GNOME visual disk usage analyzer, > which is great) to examine the filesystems, but I ran a couple of test > builds to confirm these suspects, especially after the impact of > compression (it's hard to check the *compressed* size of things in the > installer environment directly). > > I did a scratch build of lorax which does not pull in firmware > packages, and had openQA build a netinst using that lorax. It came out > at 489M - 214M smaller than current netinsts, a size we last managed in > Fedora 26. I did a scratch build of anaconda with its requirement of > yelp dropped (which would break help pages), and built a netinst with > that; it came out at 662M - 41M smaller than current images. I haven't > run a combined test yet, but it ought to come out around 448M, around > the size of Fedora 24. > > Even then we'd still be about 50% larger than the Fedora 18 image, for > not really any added functionality. > > I've moaned about the sheer amount and size of firmware blobs in other > forums before, but 214M compressed is *really* obnoxious. We must be > able to do something to clean this up (further than it's already > cleaned up - this is *after* we dropped low-hanging fruit like > enterprise switch 'firmwares' and garbage like that; most of the > remaining size seems to be huge amounts of probably-very-similar > firmware files for AMD graphics adapters and Intel wireless adapters). > I know some folks were trying to work on this (there was talk that we > could drop quite a lot of files that would only be loaded by older > kernels no longer in Fedora); any news on how far along that effort is? I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI. The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network. Ideas on how to solve that problem welcome. Peter ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Adam Williamson: > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is > 224M uncompressed. A quick test just compressing the file with xz on my > system shows it compresses to around 11M, though, so that's probably > all it adds up to after compression (the image is an xz-compressed > squashfs) Isn't the compression block-based? I think it would be interesting to measure the image size with the file removed. For the non-live installer, we can *significantly* cut down its size, without degrading localization of the installer itself. > 2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to > 23M. We are, I think, basically stuck with this for mesa-dri-drivers , > but does it have to be so *big*? It has all the targets in it. As it's for JIT, we'd only need one target. Thanks, Florian ___ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
