Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
Hi Chris, On 2015-09-22 04:26, Chris Murphy wrote: ... For the cloud image, extlinux actually works. The problem pops up with any image intended for baremetal whre UEFI Secure Boot support is needed, and right now GRUB2 does and extlinux doesn't, so any "atomic" image would need GRUB2. While you are on the topic again (*) :-), could you please explain the technical details of the above syslinux.efi/ldlinux.e64 limitation and the possible approaches to overcome it. Thanks, Alek (*) In the current thread "Re: RFC: switching from grubby to grub2-mkconfig" -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, Sep 25, 2015 at 10:09:50AM -0400, Paul Wouters wrote: > It's nice to be able to edit the .py for testing without going through > hoops or building/installing rpms. Right, but you know that if you're doing that in /usr/lib, that's *gross*, right? :) -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On 09/25/2015 08:36 AM, Matthew Miller wrote: On Fri, Sep 25, 2015 at 10:09:50AM -0400, Paul Wouters wrote: It's nice to be able to edit the .py for testing without going through hoops or building/installing rpms. Right, but you know that if you're doing that in /usr/lib, that's *gross*, right? :) To heck with gross, it's incredibly efficient. I've done it numerous times to track down problems. But that said, I'd be happy to install -pysource sub-package in order to do it. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA DivisionFAX: 303-415-9702 3380 Mitchell Lane or...@cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On 9/25/15, Matthew Millerwrote: > On Fri, Sep 25, 2015 at 11:30:38AM -0400, Bohuslav Kabrda wrote: >> Maybe we could utilize weak dependencies here. The python-foo package >> would have only bytecompiled files and would Recommend >> python-foo-sourcefiles. That way python-foo-sourcefiles would be >> installed in standard setting, but it would be possible to opt out of >> it. Just my 2 cents here, I'd seek another solution for this but implemented as a plugin or an option of dnf, whereas introducing new subpkg of large amount of packages sounds tedious to me. For example define a new kind of file as %cache in %files, and process them like find-lang, then let users decide whether files marked as %cache will be installed in default or not. -- Yours sincerely, Christopher Meng http://awk.io -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, Sep 25, 2015 at 11:30:38AM -0400, Bohuslav Kabrda wrote: > Maybe we could utilize weak dependencies here. The python-foo package > would have only bytecompiled files and would Recommend > python-foo-sourcefiles. That way python-foo-sourcefiles would be > installed in standard setting, but it would be possible to opt out of > it. Great idea! -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On 09/25/2015 10:04 AM, Matthew Miller wrote: On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote: Also, you might consider to ship the precompiled bytecode just optionally, using recommends. On contrary, if you insist on shipping the bytecode, why you don't drop the .py files? I see a lot of duplication all around python packages Wait, we can do that? Why don't we? Everything I see in online discussion is centered on, basically, transparency. But we wouldn't be doing it for obfuscation. The srpms would still be there, and for that matter we could ship the .py files in a subpackage. I wondered how much of a saving it'd be: I have around 250 python packages installed (plus python code from assorted other packages, of course). The total size of .py files on my system is 320MB! There are even 24 .py files that are larger than 500kB (mostly from third party packages, like e.g. gnuradio, freecad). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
- Original Message - > On 09/25/2015 08:36 AM, Matthew Miller wrote: > > On Fri, Sep 25, 2015 at 10:09:50AM -0400, Paul Wouters wrote: > >> It's nice to be able to edit the .py for testing without going through > >> hoops or building/installing rpms. > > > > Right, but you know that if you're doing that in /usr/lib, that's > > *gross*, right? :) > > To heck with gross, it's incredibly efficient. I've done it numerous > times to track down problems. But that said, I'd be happy to install > -pysource sub-package in order to do it. Maybe we could utilize weak dependencies here. The python-foo package would have only bytecompiled files and would Recommend python-foo-sourcefiles. That way python-foo-sourcefiles would be installed in standard setting, but it would be possible to opt out of it. > -- > Orion Poplawski > Technical Manager 303-415-9701 x222 > NWRA/CoRA DivisionFAX: 303-415-9702 > 3380 Mitchell Lane or...@cora.nwra.com > Boulder, CO 80301 http://www.cora.nwra.com > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- Regards, Slavek Kabrda -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On 09/25/2015 10:01 AM, Christopher Meng wrote: > On 9/25/15, Matthew Millerwrote: >> On Fri, Sep 25, 2015 at 11:30:38AM -0400, Bohuslav Kabrda wrote: >>> Maybe we could utilize weak dependencies here. The python-foo package >>> would have only bytecompiled files and would Recommend >>> python-foo-sourcefiles. That way python-foo-sourcefiles would be >>> installed in standard setting, but it would be possible to opt out of >>> it. > > Just my 2 cents here, I'd seek another solution for this but > implemented as a plugin or an option of dnf, whereas introducing new > subpkg of large amount of packages sounds tedious to me. Yes, that would be tedious. I'd only be for it if it could be made automatic - which I think could be possible, similar to how debuginfo is produced. It would be a lot trickier though as the files live side-by-side. > > For example define a new kind of file as %cache in %files, and process > them like find-lang, then let users decide whether files marked as > %cache will be installed in default or not. > This is an interesting idea too though. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane or...@nwra.com Boulder, CO 80301 http://www.nwra.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
* Matthew Miller: > On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote: >> Also, you might consider to ship the precompiled bytecode just >> optionally, using recommends. >> >> On contrary, if you insist on shipping the bytecode, why you don't drop >> the .py files? I see a lot of duplication all around python packages > > Wait, we can do that? Why don't we? It alters backtraces: Here's a silly example: Traceback (most recent call last): File "u.py", line 2, in t.t() File "/tmp/t.py", line 2, in t a() File "/tmp/t.py", line 5, in a b() File "/tmp/t.py", line 8, in b c() NameError: global name 'c' is not defined This turns into: Traceback (most recent call last): File "u.py", line 2, in t.t() File "/tmp/t.py", line 2, in t File "/tmp/t.py", line 5, in a File "/tmp/t.py", line 8, in b NameError: global name 'c' is not defined Such a change will not be universally well-received. In any case, source code (with comments stripped and whitespace normalized) is often smaller than bytecode (or most other forms of serializing syntax), and compresses better. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
Dne 25.9.2015 v 16:04 Matthew Miller napsal(a): > On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote: >> Also, you might consider to ship the precompiled bytecode just >> optionally, using recommends. Just to be clear, not/optional shipping the bytecode is the preferred method IMO >> >> On contrary, if you insist on shipping the bytecode, why you don't drop >> the .py files? I see a lot of duplication all around python packages I proposed this because some might complain that not shipping the bytecode might result in some performance penalty, but I'd say that this applies just for first run, so it is negligible Vít > Wait, we can do that? Why don't we? > > Everything I see in online discussion is centered on, basically, > transparency. But we wouldn't be doing it for obfuscation. The srpms > would still be there, and for that matter we could ship the .py files > in a subpackage. > -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
Dne 25.9.2015 v 16:15 Mathieu Bridon napsal(a): > On Fri, 2015-09-25 at 10:04 -0400, Matthew Miller wrote: >> On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote: >>> Also, you might consider to ship the precompiled bytecode just >>> optionally, using recommends. >>> >>> On contrary, if you insist on shipping the bytecode, why you don't >>> drop >>> the .py files? I see a lot of duplication all around python >>> packages >> Wait, we can do that? Why don't we? >> >> Everything I see in online discussion is centered on, basically, >> transparency. But we wouldn't be doing it for obfuscation. The srpms >> would still be there, and for that matter we could ship the .py files >> in a subpackage. > Or maybe rpm could have a macro a bit like %{_install_langs} (which > controls what language files are installed, even though they are all in > the packages) to control what gets installed for Python stuff, even > though everything is in the packages. > > On Workstation, that macro would be set so that both the byte-code and > the code would be installed > (it is invaluable for learning and > debugging purposes to be able to read/edit the code). Come on, this is not an argument. We don't install source code for any other language which produces some binary libraries except Python while you cannot deny that it would be invaluable for the same purposes. I understand you sentiment, but I rather appreciate Orion's position, let me quote: "But that said, I'd be happy to install -pysource sub-package in order to do it." Vít > > And on Cloud, that macro would be set to have only the bytecode > installed, so that you'd save the space. > > -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, Sep 25, 2015 at 3:25 PM, Vít Ondruchwrote: > Dne 25.9.2015 v 16:15 Mathieu Bridon napsal(a): > > (it is invaluable for learning and > > debugging purposes to be able to read/edit the code). > > Come on, this is not an argument. We don't install source code for any > other language which produces some binary libraries except Python while > you cannot deny that it would be invaluable for the same purposes. I > understand you sentiment, but I rather appreciate Orion's position, let > me quote: "But that said, I'd be happy to install -pysource sub-package > in order to do it." > > > Vít > > If it's possible to auto-generate -pysource subpackages like we do with debuginfo packages, then I would certainly welcome it. If my understanding is correct, Python module importing works fine with only *.pyc files, right? -- 真実はいつも一つ!/ Always, there's only one truth! -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Thu, Sep 24, 2015 at 12:59:09AM +, Zbigniew Jędrzejewski-Szmek wrote: > > cares?" adds up. If it *got* us anything in terms of functionality, > > okay, I can sell that to people, but for the base, this is just bloat. > Bummer. The reason for libxkbcommon dependency is to be able to make > sure that the new config is valid. Before that was added we had a set > of rules and heuristics implemented in localed and regular bug reports > when typos and other mistakes were not caught by localed but Xorg > would not accept the new config. This is more important than might > seem, 'cause people tend to get grumpy when a misconfigured keyboard > mapping prevents them from typing in their password. So this dependency > does bring useful functionality. Well, it brings in useful functionality in some cases, but not in others. It's unfortunate that we don't have a good way for each Fedora edition or spin to decide which functionality is wanted. This particular functionality seems very important on Workstation, and moderately valuable on Server (and the tradeoff there isn't disk space, but potential increased surface for bugs and security vulnerabilities), but just deadweight for Cloud guests. Even with Atomic running on bare metal, logging in at the console isn't an important use case. (If need be, the console environment could be provided in a container.) -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Fri, Sep 25, 2015 at 4:00 PM, Matthew Millerwrote: > On Thu, Sep 24, 2015 at 12:59:09AM +, Zbigniew Jędrzejewski-Szmek wrote: >> > cares?" adds up. If it *got* us anything in terms of functionality, >> > okay, I can sell that to people, but for the base, this is just bloat. >> Bummer. The reason for libxkbcommon dependency is to be able to make >> sure that the new config is valid. Before that was added we had a set >> of rules and heuristics implemented in localed and regular bug reports >> when typos and other mistakes were not caught by localed but Xorg >> would not accept the new config. This is more important than might >> seem, 'cause people tend to get grumpy when a misconfigured keyboard >> mapping prevents them from typing in their password. So this dependency >> does bring useful functionality. > > Well, it brings in useful functionality in some cases, but not in > others. It's unfortunate that we don't have a good way for each Fedora > edition or spin to decide which functionality is wanted. This > particular functionality seems very important on Workstation, and > moderately valuable on Server (and the tradeoff there isn't disk space, > but potential increased surface for bugs and security vulnerabilities), > but just deadweight for Cloud guests. Even with Atomic running on bare > metal, logging in at the console isn't an important use case. (If need > be, the console environment could be provided in a container.) Couldn't additional dracut subpackages similar to dracut-config-generic (which is essentially just wrapping a dracut config snippet) be cerated to include some modules? A spin could then create a meta package to pull in the required dracut packages. Just my 2ct. - fabian > > -- > Matthew Miller > > Fedora Project Leader > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote: > Also, you might consider to ship the precompiled bytecode just > optionally, using recommends. > > On contrary, if you insist on shipping the bytecode, why you don't drop > the .py files? I see a lot of duplication all around python packages Wait, we can do that? Why don't we? Everything I see in online discussion is centered on, basically, transparency. But we wouldn't be doing it for obfuscation. The srpms would still be there, and for that matter we could ship the .py files in a subpackage. -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, 25 Sep 2015, Matthew Miller wrote: On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote: Also, you might consider to ship the precompiled bytecode just optionally, using recommends. On contrary, if you insist on shipping the bytecode, why you don't drop the .py files? I see a lot of duplication all around python packages Wait, we can do that? Why don't we? Everything I see in online discussion is centered on, basically, transparency. But we wouldn't be doing it for obfuscation. The srpms would still be there, and for that matter we could ship the .py files in a subpackage. It's nice to be able to edit the .py for testing without going through hoops or building/installing rpms. It's also nice to be able to read the .py code. That is one reason people use script languages :P Paul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, 2015-09-25 at 10:04 -0400, Matthew Miller wrote: > On Thu, Sep 24, 2015 at 10:10:40AM +0200, Vít Ondruch wrote: > > Also, you might consider to ship the precompiled bytecode just > > optionally, using recommends. > > > > On contrary, if you insist on shipping the bytecode, why you don't > > drop > > the .py files? I see a lot of duplication all around python > > packages > > Wait, we can do that? Why don't we? > > Everything I see in online discussion is centered on, basically, > transparency. But we wouldn't be doing it for obfuscation. The srpms > would still be there, and for that matter we could ship the .py files > in a subpackage. Or maybe rpm could have a macro a bit like %{_install_langs} (which controls what language files are installed, even though they are all in the packages) to control what gets installed for Python stuff, even though everything is in the packages. On Workstation, that macro would be set so that both the byte-code and the code would be installed (it is invaluable for learning and debugging purposes to be able to read/edit the code). And on Cloud, that macro would be set to have only the bytecode installed, so that you'd save the space. -- Mathieu -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, Sep 25, 2015 at 4:04 PM, Matthew Millerwrote: >> On contrary, if you insist on shipping the bytecode, why you don't drop >> the .py files? I see a lot of duplication all around python packages Don't do that, it has impact on functionality. We did this for years [0], but you wonder how many components actually glob for *.py files i.e. to load plugins. Blacklisting files is not worth the space. - fabian [0] https://bugzilla.redhat.com/show_bug.cgi?id=1233106 -- Fabian Deutsch RHEV Hypervisor Red Hat -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, 2015-09-25 at 16:17 +0200, Fabian Deutsch wrote: > On Fri, Sep 25, 2015 at 4:04 PM, Matthew Miller >wrote: > > > On contrary, if you insist on shipping the bytecode, why you > > > don't drop > > > the .py files? I see a lot of duplication all around python > > > packages > > Don't do that, it has impact on functionality. > We did this for years [0], but you wonder how many components > actually > glob for *.py files i.e. to load plugins. > Blacklisting files is not worth the space. > Just as an FYI, rolekit does exactly this. signature.asc Description: This is a digitally signed message part -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: python: dropping the .py files [was Re: Fedora 23 cloud image (and, for that matter, minimal anything)] bloat
On Fri, Sep 25, 2015 at 4:26 PM, Stephen Gallagherwrote: > On Fri, 2015-09-25 at 16:17 +0200, Fabian Deutsch wrote: >> On Fri, Sep 25, 2015 at 4:04 PM, Matthew Miller >> wrote: >> > > On contrary, if you insist on shipping the bytecode, why you >> > > don't drop >> > > the .py files? I see a lot of duplication all around python >> > > packages >> >> Don't do that, it has impact on functionality. >> We did this for years [0], but you wonder how many components >> actually >> glob for *.py files i.e. to load plugins. >> Blacklisting files is not worth the space. >> > > > Just as an FYI, rolekit does exactly this. virt-who, sosreport, and python-urllib3 are just other examples of packages doing this. - fabian -- Fabian Deutsch RHEV Hypervisor Red Hat -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
Dne 22.9.2015 v 11:07 Matej Stuchlik napsal(a): > > When it comes to python3, one way to shave off ~9MiB from python3-libs, and > possibly quite a bit more overall, would be to not install both optimized and > unoptimized bytecode, as we do now, but just the unoptimized one (the > performance > hit should be very small). Isn't it time to use %filetriggers to generate the bytecode instead of shipping it in packages? You would save a lot of space, you could drop the most of the python{2,3} packages etc. Also, you might consider to ship the precompiled bytecode just optionally, using recommends. On contrary, if you insist on shipping the bytecode, why you don't drop the .py files? I see a lot of duplication all around python packages Vít -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Thu, 2015-09-24 at 00:59 +, Zbigniew Jędrzejewski-Szmek wrote: > Bummer. The reason for libxkbcommon dependency is to be able to make > sure that the new config is valid. Before that was added we had a set > of rules and heuristics implemented in localed and regular bug reports > when typos and other mistakes were not caught by localed but Xorg > would not accept the new config. This is more important than might > seem, 'cause people tend to get grumpy when a misconfigured keyboard > mapping prevents them from typing in their password. So this dependency > does bring useful functionality. We can certainly make xkeyboard-config smaller. About half its on-disk footprint is localized strings, %langpack would help. The geometry subdir is completely useless, and I'm reasonably sure the xml it installs is really just source data and not anything apps or libs use. All told that'd get xkeyboard-config down to a hair over 2M on disk. - ajax -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Thu, 2015-09-24 at 11:00 -0400, Adam Jackson wrote: > On Thu, 2015-09-24 at 00:59 +, Zbigniew Jędrzejewski-Szmek wrote: > > > Bummer. The reason for libxkbcommon dependency is to be able to > > make > > sure that the new config is valid. Before that was added we had a > > set > > of rules and heuristics implemented in localed and regular bug > > reports > > when typos and other mistakes were not caught by localed but Xorg > > would not accept the new config. This is more important than might > > seem, 'cause people tend to get grumpy when a misconfigured > > keyboard > > mapping prevents them from typing in their password. So this > > dependency > > does bring useful functionality. > > We can certainly make xkeyboard-config smaller. About half its on- > disk > footprint is localized strings, %langpack would help. The geometry > subdir is completely useless, and I'm reasonably sure the xml it > installs is really just source data and not anything apps or libs > use. > All told that'd get xkeyboard-config down to a hair over 2M on disk. > Not saying that it is very useful, but I believe gkbd-keyboard-display does use geometry information. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
Am 21.09.2015 um 11:07 schrieb Matthew Miller: > Fedora-Cloud-Base-20141203-21.x86_64.qcow2: 151M > Fedora-Cloud-Base-23_Beta-20150915.x86_64.qcow2: 275M > > In just one year — 82% more awesome? > > I'd really like this to stay below 200MB as a competitive threshold. > Or, if we're going to be bigger than that, be bigger for REASONS, not > just accretion. > > tl;dr: grub2 is a lot to blame, but there seem to be some new > questionable dep chains from systemd, and general dep growth across the > board. > > > Disk use at first boot: > > [f21]$ df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/vda120G 359M 19G 2% / > > [f23b]$ df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/vda120G 578M 19G 4% / > > RPMs installed: > > [f21]$ rpm -qa | wc -l > 226 > > [f23b]$ rpm -qa | wc -l > 264 > > Top 20 rpms by reported size: > > $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 > 120417342 glibc-common > 42307839 kernel-core > 25000497 python-libs > 22438155 systemd > 14623272 coreutils > 14000291 glibc > 11282056 ruby-libs # hey, at least we lost this > 10845519 glib2 > 10593004 selinux-policy-targeted > 9389116 cracklib-dicts # https://bugzilla.redhat.com/show_bug.cgi?id=865521 > 9078043 python-boto > 8792531 util-linux > 7084188 bash > 6669884 gnupg2 > 5844544 yum > 4893790 policycoreutils > 3786564 file-libs > 3540004 shadow-utils > 3458312 groff-base # who doesn't love groff? > 2997717 tar > > > $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 > 125195206 glibc-common > 86298752 linux-firmware # sadface, but hard > 53291365 kernel-core > 36004297 grub2-tools # this is ridiculous > 28453336 python3-libs # 13% growth > 27233273 systemd # 21% growth > 16648994 grub2# *sigh* > 14486819 glibc > 14287847 coreutils# this package got _smaller!_ > 11143743 glib2 > 11129880 selinux-policy-targeted > 9389116 cracklib-dicts > 9261499 python3-boto > 9237998 util-linux > 9224255 fedora-logos # this is also grub's fault. > 7517574 gnupg2 > 7143418 bash > 6574678 python3-pip # :( > 583 hwdata# this is ALSO grub's fault > 5423400 xkeyboard-config # really looks like a systemd dep chain > involving plymouth > > Okay, let's look on disk: > > > [f21]$ sudo du -sh * 2>/dev/null|sort -h > [...] > 36K home > 40K root > 228Krun > 21M boot > 22M etc > 34M var > 276Musr > > > [f23b]$ sudo du -sh * 2>/dev/null|sort -h > [...] > 40K root > 264Krun > 16M etc > 45M boot # ugh > 171Mvar # oww > 463Musr # oww oww oww > > Breakdown: > >- boot is mostly grub, but initramfs is also doubled you probably don't need lvm, raid and all that stuff in the initramfs.. Also maybe not network and a rescue initramfs? Is dracut-network and dracut-config-generic in the install set? If yes, remove it. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Wed, Sep 23, 2015 at 06:35:04PM +, Zbigniew Jędrzejewski-Szmek wrote: > > > >5423400 xkeyboard-config # really looks like a systemd dep chain > > > >involving plymouth > > > plymouth luckily is no hard dependency for anything > > "Involving plymouth" might have been wrong. libxkbcommon.so.0 looks > > like a direct dependency, from /usr/lib/systemd/systemd-localed. > Yes, libxkbcommon is used to validate the configuration set > with 'localectl set-keymap' or 'localectl set-x11-keymap' > (or directly through dbus api). This is not relevant in most cloud images and in very, very few containers. Can we make the dependency optional in some way? -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Mon, Sep 21, 2015 at 11:35:54AM -0400, Matthew Miller wrote: > On Mon, Sep 21, 2015 at 05:11:24PM +0200, Reindl Harald wrote: > > >5423400 xkeyboard-config # really looks like a systemd dep chain > > >involving plymouth > > plymouth luckily is no hard dependency for anything > > "Involving plymouth" might have been wrong. libxkbcommon.so.0 looks > like a direct dependency, from /usr/lib/systemd/systemd-localed. Yes, libxkbcommon is used to validate the configuration set with 'localectl set-keymap' or 'localectl set-x11-keymap' (or directly through dbus api). > I also don't know how plymouth crept in, though. Zbyszek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Wed, Sep 23, 2015 at 02:46:52PM -0400, Matthew Miller wrote: > On Wed, Sep 23, 2015 at 06:35:04PM +, Zbigniew Jędrzejewski-Szmek wrote: > > > > >5423400 xkeyboard-config # really looks like a systemd dep chain > > > > >involving plymouth > > > > plymouth luckily is no hard dependency for anything > > > "Involving plymouth" might have been wrong. libxkbcommon.so.0 looks > > > like a direct dependency, from /usr/lib/systemd/systemd-localed. > > Yes, libxkbcommon is used to validate the configuration set > > with 'localectl set-keymap' or 'localectl set-x11-keymap' > > (or directly through dbus api). > > This is not relevant in most cloud images and in very, very few > containers. Can we make the dependency optional in some way? Not really. localed is simply linked to libxkbcommon.so, so it will not run without. Anyway, libxkbcommon.rpm is 260kb, so some potential (complicated) scheme to avoid a dependency simply does not seem worth the trouble. Zbyszek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Wed, Sep 23, 2015 at 07:07:09PM +, Zbigniew Jędrzejewski-Szmek wrote: > Not really. localed is simply linked to libxkbcommon.so, so it will > not run without. Anyway, libxkbcommon.rpm is 260kb, so some potential > (complicated) scheme to avoid a dependency simply does not seem worth > the trouble. No; bring brings in xkeyboard-config, which is over 5mb, putting it in the top 20 biggest packages. 5mb-on-disk isn't _huge_ but overall we grew 220mb on disk over two releases, and a whole bunch of "eh, who cares?" adds up. If it *got* us anything in terms of functionality, okay, I can sell that to people, but for the base, this is just bloat. -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
- Original Message - > From: "Matthew Miller" <mat...@fedoraproject.org> > To: "Development discussions related to Fedora" > <devel@lists.fedoraproject.org> > Sent: Tuesday, September 22, 2015 3:49:19 PM > Subject: Re: Fedora 23 cloud image (and, for that matter, minimal anything) > bloat > > On Tue, Sep 22, 2015 at 05:07:58AM -0400, Matej Stuchlik wrote: > > When it comes to python3, one way to shave off ~9MiB from > > python3-libs, and possibly quite a bit more overall, would be to not > > install both optimized and unoptimized bytecode, as we do now, but > > just the unoptimized one (the performance hit should be very small). > > > > I'll look into if that could be done. > > > > We could also move few things from -libs to -devel, possibly. > > Cool. Let's put you on the future list for > https://fedorahosted.org/fedora-badges/ticket/378 :) I've tried to move idlelib and lib2to3 to -tools subpackage but unfortunately I had to revert this change as distutils uses lib2to3 module and idlelib is used by -tkinter meaning that -tkinter would have to depend on -tools subpackage but -tools already requires -tkinter so it would lead to cyclic dep. > > > > -- > Matthew Miller > <mat...@fedoraproject.org> > Fedora Project Leader > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- Robert Kuska {rkuska} -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Wed, Sep 23, 2015 at 03:28:57PM -0400, Matthew Miller wrote: > On Wed, Sep 23, 2015 at 07:07:09PM +, Zbigniew Jędrzejewski-Szmek wrote: > > Not really. localed is simply linked to libxkbcommon.so, so it will > > not run without. Anyway, libxkbcommon.rpm is 260kb, so some potential > > (complicated) scheme to avoid a dependency simply does not seem worth > > the trouble. > > No; bring brings in xkeyboard-config, which is over 5mb, putting it in > the top 20 biggest packages. 5mb-on-disk isn't _huge_ but overall we > grew 220mb on disk over two releases, and a whole bunch of "eh, who > cares?" adds up. If it *got* us anything in terms of functionality, > okay, I can sell that to people, but for the base, this is just bloat. Bummer. The reason for libxkbcommon dependency is to be able to make sure that the new config is valid. Before that was added we had a set of rules and heuristics implemented in localed and regular bug reports when typos and other mistakes were not caught by localed but Xorg would not accept the new config. This is more important than might seem, 'cause people tend to get grumpy when a misconfigured keyboard mapping prevents them from typing in their password. So this dependency does bring useful functionality. Maybe it would be possible to change libxkbcommon to be able to provide verification of the mapping configuration without having the mappings installed, i.e. remove the dependency on xkeyboard-config. Then the dependnecy on xkeyboard-config would have to be added to xserver or to other consumers. I don't know if this is even feasible. Zbyszek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
- Original Message - > From: "Matthew Miller" <mat...@fedoraproject.org> > To: "Fedora Development List" <devel@lists.fedoraproject.org> > Sent: Monday, September 21, 2015 5:07:40 PM > Subject: Fedora 23 cloud image (and, for that matter, minimal anything) bloat > > Fedora-Cloud-Base-20141203-21.x86_64.qcow2: 151M > Fedora-Cloud-Base-23_Beta-20150915.x86_64.qcow2: 275M > > In just one year — 82% more awesome? > > I'd really like this to stay below 200MB as a competitive threshold. > Or, if we're going to be bigger than that, be bigger for REASONS, not > just accretion. > > tl;dr: grub2 is a lot to blame, but there seem to be some new > questionable dep chains from systemd, and general dep growth across the > board. > > > Disk use at first boot: > > [f21]$ df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/vda120G 359M 19G 2% / > > [f23b]$ df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/vda120G 578M 19G 4% / > > RPMs installed: > > [f21]$ rpm -qa | wc -l > 226 > > [f23b]$ rpm -qa | wc -l > 264 > > Top 20 rpms by reported size: > > $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 > 120417342 glibc-common > 42307839 kernel-core > 25000497 python-libs > 22438155 systemd > 14623272 coreutils > 14000291 glibc > 11282056 ruby-libs # hey, at least we lost this > 10845519 glib2 > 10593004 selinux-policy-targeted > 9389116 cracklib-dicts # > https://bugzilla.redhat.com/show_bug.cgi?id=865521 > 9078043 python-boto > 8792531 util-linux > 7084188 bash > 6669884 gnupg2 > 5844544 yum > 4893790 policycoreutils > 3786564 file-libs > 3540004 shadow-utils > 3458312 groff-base # who doesn't love groff? > 2997717 tar > > > $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 > 125195206 glibc-common > 86298752 linux-firmware # sadface, but hard > 53291365 kernel-core > 36004297 grub2-tools # this is ridiculous > 28453336 python3-libs # 13% growth > 27233273 systemd # 21% growth > 16648994 grub2# *sigh* > 14486819 glibc > 14287847 coreutils# this package got _smaller!_ > 11143743 glib2 > 11129880 selinux-policy-targeted > 9389116 cracklib-dicts > 9261499 python3-boto > 9237998 util-linux > 9224255 fedora-logos # this is also grub's fault. > 7517574 gnupg2 > 7143418 bash > 6574678 python3-pip # :( > 583 hwdata# this is ALSO grub's fault > 5423400 xkeyboard-config # really looks like a systemd dep chain > involving plymouth When it comes to python3, one way to shave off ~9MiB from python3-libs, and possibly quite a bit more overall, would be to not install both optimized and unoptimized bytecode, as we do now, but just the unoptimized one (the performance hit should be very small). I'll look into if that could be done. We could also move few things from -libs to -devel, possibly. Matt > Okay, let's look on disk: > > > [f21]$ sudo du -sh * 2>/dev/null|sort -h > [...] > 36K home > 40K root > 228Krun > 21M boot > 22M etc > 34M var > 276Musr > > > [f23b]$ sudo du -sh * 2>/dev/null|sort -h > [...] > 40K root > 264Krun > 16M etc > 45M boot # ugh > 171Mvar # oww > 463Musr # oww oww oww > > Breakdown: > >- boot is mostly grub, but initramfs is also doubled >- var: DNF cache is full of stuff! 123M... oh, hey, that wasn't > there when we booted. `df -h /` is now up to 702M. Maybe multiple > repos would help here, or some other more clever design. Does > every cloud image in the world _really_ need to replicate > dependency data so I can `dnf install > /usr/share/minetest/builtin/mainmenu/init_simple.lua`? > >- usr: > > * linux-firmware is the biggest thing here > * python3 is another good chunk > * also kernel modules, but, eh, whatchagonnado > * oh, and of course, grub2. > > -- > Matthew Miller > <mat...@fedoraproject.org> > Fedora Project Leader > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tue, Sep 22, 2015 at 11:07 AM, Matej Stuchlikwrote: > [...] > When it comes to python3, one way to shave off ~9MiB from python3-libs, and > possibly quite a bit more overall, would be to not install both optimized and > unoptimized bytecode, as we do now, but just the unoptimized one (the > performance > hit should be very small). How small is "very small" ? Have you measured it? I don't think 9MB of disk space is worth taking a performance hit for ... -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Mon, Sep 21, 2015 at 07:26:52PM -0600, Chris Murphy wrote: > For the cloud image, extlinux actually works. The problem pops up with > any image intended for baremetal whre UEFI Secure Boot support is > needed, and right now GRUB2 does and extlinux doesn't, so any "atomic" > image would need GRUB2. In theory, yes. In practice: https://bugzilla.redhat.com/show_bug.cgi?id=1250874 [...] > So, I'm not sure where 70MB is coming from. All four are needed for On disk, and counting deps not previously required. -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tue, Sep 22, 2015 at 05:07:58AM -0400, Matej Stuchlik wrote: > When it comes to python3, one way to shave off ~9MiB from > python3-libs, and possibly quite a bit more overall, would be to not > install both optimized and unoptimized bytecode, as we do now, but > just the unoptimized one (the performance hit should be very small). > > I'll look into if that could be done. > > We could also move few things from -libs to -devel, possibly. Cool. Let's put you on the future list for https://fedorahosted.org/fedora-badges/ticket/378 :) -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tue, Sep 22, 2015 at 2:25 PM, Neal Gompawrote: > On Tue, Sep 22, 2015 at 5:36 AM, Matej Stuchlik wrote: >> >> Also note that it's possibly not just 9MB. For instance python3-boto, also >> on this list, would >> save 4.7MB, python3-pip 2.9MB. In general most python packages could go >> down in size by ~20-30%. > > However, this approach would break with Python 3.5 (where pyo data is merged > into *.pyc data) To be more precise, AFAIU there's no "merging", *.pyo goes away but in exchange there are actually two new optimized bytecode files, *.opt-1.pyc and *.opt-2.pyc: https://www.python.org/dev/peps/pep-0488/ If you want to exclude them from packages, they should be there listed as %ghost so they're removed in case they get generated by something run with -O or -OO. Ditto *.pyo if you intend to exclude them from python < 3.5 packages. Also, be careful with measuring space savings when working with *.pyo. It is a common case that *.pyc and *.pyo are identical, and when they are rpmbuild already hardlinks them. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tue, Sep 22, 2015 at 5:36 AM, Matej Stuchlik <mstuc...@redhat.com> wrote: > - Original Message - > > From: "drago01" <drag...@gmail.com> > > To: "Development discussions related to Fedora" < > devel@lists.fedoraproject.org> > > Sent: Tuesday, September 22, 2015 11:20:27 AM > > Subject: Re: Fedora 23 cloud image (and, for that matter, minimal > anything) bloat > > > > On Tue, Sep 22, 2015 at 11:07 AM, Matej Stuchlik <mstuc...@redhat.com> > wrote: > > > [...] > > > When it comes to python3, one way to shave off ~9MiB from > python3-libs, and > > > possibly quite a bit more overall, would be to not install both > optimized > > > and > > > unoptimized bytecode, as we do now, but just the unoptimized one (the > > > performance > > > hit should be very small). > > > > How small is "very small" ? Have you measured it? > > I don't think 9MB of disk space is worth taking a performance hit for ... > > The only difference between unoptimized and "optimized" bytecode should be > that > the latter is missing all docstrings, has disable asserts and sets > __debug__ to False, > I can't imagine this being significant, performance wise. > > That said I do not plan on doing this before I measure the performance > difference and > discuss it on python-sig and python-linux. > > Also note that it's possibly not just 9MB. For instance python3-boto, also > on this list, would > save 4.7MB, python3-pip 2.9MB. In general most python packages could go > down in size by ~20-30%. > > Matt > > However, this approach would break with Python 3.5 (where pyo data is merged into *.pyc data), so I would consider it an ill-advised approach anyway. It may work for F23, but it won't work for F24, and then we'd be back to square one again. -- 真実はいつも一つ!/ Always, there's only one truth! -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
- Original Message - > From: "drago01" <drag...@gmail.com> > To: "Development discussions related to Fedora" > <devel@lists.fedoraproject.org> > Sent: Tuesday, September 22, 2015 11:20:27 AM > Subject: Re: Fedora 23 cloud image (and, for that matter, minimal > anything) bloat > > On Tue, Sep 22, 2015 at 11:07 AM, Matej Stuchlik <mstuc...@redhat.com> wrote: > > [...] > > When it comes to python3, one way to shave off ~9MiB from python3-libs, and > > possibly quite a bit more overall, would be to not install both optimized > > and > > unoptimized bytecode, as we do now, but just the unoptimized one (the > > performance > > hit should be very small). > > How small is "very small" ? Have you measured it? > I don't think 9MB of disk space is worth taking a performance hit for ... The only difference between unoptimized and "optimized" bytecode should be that the latter is missing all docstrings, has disable asserts and sets __debug__ to False, I can't imagine this being significant, performance wise. That said I do not plan on doing this before I measure the performance difference and discuss it on python-sig and python-linux. Also note that it's possibly not just 9MB. For instance python3-boto, also on this list, would save 4.7MB, python3-pip 2.9MB. In general most python packages could go down in size by ~20-30%. Matt > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tue, Sep 22, 2015 at 4:13 PM, Matej Stuchlikwrote: > - Original Message - >> From: "Ville Skyttä" >> >> Managed to fiddle around some more and looks like the above is a false >> concern, many *.pyc, *.opt-1.pyc and *.opt-2.pyc are identical. >> So, https://github.com/rpm-software-management/rpm/pull/16 > > Is it? See my next email, Python 3.5.0 RPM seems ~50% bigger right now, unless > I'm making some silly mistake, I don't think I've seen the email you refer to yet, nor do I have access to a Python 3.5.0 rpm -- where can I find one? Anyway I wouldn't be surprised if the majority of that difference is because rpmbuild's brp-hardlink-python only hardlinks pyc and pyo, not the new optimized files. The pull request I have open should address that. > and from what I see *.opt-2.pyc is nearly always different from the other > ones. The only quick and silly test I did was to run rpmlint without any arguments using python3.5 binaries grabbed from an Arch Linux package, with -O and -OO. It failed early because not all required modules were available, but it generated pyc files for five modules: in three of them all pyc, opt-1.pyc and opt-2.pyc were identical, and in two pyc and opt-1.pyc were identical but opt-2.pyc differed. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tue, Sep 22, 2015 at 3:32 PM, Matej Stuchlikwrote: > - Original Message - >> From: "Ville Skyttä" >> >> Also, be careful with measuring space savings when working with *.pyo. >> It is a common case that *.pyc and *.pyo are identical, and when they >> are rpmbuild already hardlinks them. > > That's really interesting, I've had no idea: BTW I just had a peek into some Arch Linux Python 3.5 packages, and it seems they contain *no* identical *.pyc and corresponding *.opt-1.pyc files. This is bad news wrt the hardlinking. I haven't found any *.opt-2.pyc files to play with, so I don't know what's the situation with them. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tuesday, 22 September 2015 at 03:26, Chris Murphy wrote: > On Mon, Sep 21, 2015 at 2:37 PM, Matthew Miller >wrote: [...] > > But *that* said, the current packaging means that grub2 adds 70MB on > > disk — about 12% of the entire cloud image. I'm not saying grub2 is > > evil, just that this is a big portion of the gain and is worth > > attacking in order to reverse it. I talked briefly to Peter Jones and > > he says there's quite a bit which can be done there, if going back to > > extlinux doesn't seem like a viable option. > > These are needed on BIOS systems > grub2-tools is 7MiB Installed size is actually 36MB. /usr/sbin/grub2-sparc64-setup is 2MB and probably doesn't work on anything else than ppc64(le). /usr/sbin/grub2-macbless is also 2MB and probably doesn't work on non-Macs > grub2 is 3.8MiB > > These are needed on UEFI systems > shim-signed is 633KiB > grub2-efi is 837KiB > > So, I'm not sure where 70MB is coming from. [...] Matt was talking about 70MB on disk (installed size), not rpm package size. Regards, Dominik -- Fedora http://fedoraproject.org/wiki/User:Rathann RPMFusion http://rpmfusion.org "Faith manages." -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations" -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
- Original Message - > From: "Ville Skyttä" <ville.sky...@iki.fi> > To: "Development discussions related to Fedora" > <devel@lists.fedoraproject.org> > Sent: Tuesday, September 22, 2015 1:36:09 PM > Subject: Re: Fedora 23 cloud image (and, for that matter, minimal > anything) bloat > > On Tue, Sep 22, 2015 at 2:25 PM, Neal Gompa <ngomp...@gmail.com> wrote: > > On Tue, Sep 22, 2015 at 5:36 AM, Matej Stuchlik <mstuc...@redhat.com> > > wrote: > >> > >> Also note that it's possibly not just 9MB. For instance python3-boto, also > >> on this list, would > >> save 4.7MB, python3-pip 2.9MB. In general most python packages could go > >> down in size by ~20-30%. > > > > However, this approach would break with Python 3.5 (where pyo data is > > merged > > into *.pyc data) > > To be more precise, AFAIU there's no "merging", *.pyo goes away but in > exchange there are actually two new optimized bytecode files, > *.opt-1.pyc and *.opt-2.pyc: https://www.python.org/dev/peps/pep-0488/ > If you want to exclude them from packages, they should be there listed > as %ghost so they're removed in case they get generated by something > run with -O or -OO. Ditto *.pyo if you intend to exclude them from > python < 3.5 packages. > > Also, be careful with measuring space savings when working with *.pyo. > It is a common case that *.pyc and *.pyo are identical, and when they > are rpmbuild already hardlinks them. That's really interesting, I've had no idea: % find /usr/ -name "*.pyc" | wc -l # All .pyc files 18376 % find /usr/ -name "*.pyc" -links +1 | wc -l # .pyc files that are hard linked 15761 find /usr/ -name '*.pyc' -links 1 -print0 | du --files0-from=- -ch | tail -n1 # Total size of non-hardlinked .pyc files 65M total If I'm correct here it looks like it wouldn't really be worth it. > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Tue, Sep 22, 2015 at 3:43 PM, Ville Skyttäwrote: > > BTW I just had a peek into some Arch Linux Python 3.5 packages, and it > seems they contain *no* identical *.pyc and corresponding *.opt-1.pyc > files. This is bad news wrt the hardlinking. I haven't found any > *.opt-2.pyc files to play with, so I don't know what's the situation > with them. Managed to fiddle around some more and looks like the above is a false concern, many *.pyc, *.opt-1.pyc and *.opt-2.pyc are identical. So, https://github.com/rpm-software-management/rpm/pull/16 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
- Original Message - > From: "Ville Skyttä" <ville.sky...@iki.fi> > To: "Development discussions related to Fedora" > <devel@lists.fedoraproject.org> > Sent: Tuesday, September 22, 2015 2:43:33 PM > Subject: Re: Fedora 23 cloud image (and, for that matter, minimal > anything) bloat > > On Tue, Sep 22, 2015 at 3:32 PM, Matej Stuchlik <mstuc...@redhat.com> wrote: > > - Original Message - > >> From: "Ville Skyttä" <ville.sky...@iki.fi> > >> > >> Also, be careful with measuring space savings when working with *.pyo. > >> It is a common case that *.pyc and *.pyo are identical, and when they > >> are rpmbuild already hardlinks them. > > > > That's really interesting, I've had no idea: > > BTW I just had a peek into some Arch Linux Python 3.5 packages, and it > seems they contain *no* identical *.pyc and corresponding *.opt-1.pyc > files. This is bad news wrt the hardlinking. I haven't found any > *.opt-2.pyc files to play with, so I don't know what's the situation > with them. And sure enough: % rpm2cpio python3-libs-3.4.3-5.fc24.x86_64.rpm | cpio -idmv 2> /dev/null && du -ch usr/ | tail -n1 31M total % rpm2cpio python3-libs-3.5.0-1.fc24.x86_64.rpm | cpio -idmv 2> /dev/null && du -ch usr/ | tail -n1 43M total That's definitely not great. > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
- Original Message - > From: "Ville Skyttä" <ville.sky...@iki.fi> > To: "Development discussions related to Fedora" > <devel@lists.fedoraproject.org> > Sent: Tuesday, September 22, 2015 3:03:24 PM > Subject: Re: Fedora 23 cloud image (and, for that matter, minimal > anything) bloat > > On Tue, Sep 22, 2015 at 3:43 PM, Ville Skyttä <ville.sky...@iki.fi> wrote: > > > > BTW I just had a peek into some Arch Linux Python 3.5 packages, and it > > seems they contain *no* identical *.pyc and corresponding *.opt-1.pyc > > files. This is bad news wrt the hardlinking. I haven't found any > > *.opt-2.pyc files to play with, so I don't know what's the situation > > with them. > > Managed to fiddle around some more and looks like the above is a false > concern, many *.pyc, *.opt-1.pyc and *.opt-2.pyc are identical. > So, https://github.com/rpm-software-management/rpm/pull/16 Is it? See my next email, Python 3.5.0 RPM seems ~50% bigger right now, unless I'm making some silly mistake, and from what I see *.opt-2.pyc is nearly always different from the other ones. > -- > devel mailing list > devel@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Mon, Sep 21, 2015 at 2:37 PM, Matthew Millerwrote: > On Mon, Sep 21, 2015 at 03:39:46PM -0400, Josh Boyer wrote: >> It's worth pointing out that your griping about grub2 "growth" seems >> misleading. The sizes of the grub2 packages did not change between >> F22 and F23. What seems to have happened is that the cloud images >> added the grub2 packages, which weren't there in F22. > > Sorry, yeah -- that griping is not out of context in the Cloud WG where > I also posted this, but is here. Grub2 has the huge virtue of *It > Actually Works*, a claim which currently none of the competition can > make. For the cloud image, extlinux actually works. The problem pops up with any image intended for baremetal whre UEFI Secure Boot support is needed, and right now GRUB2 does and extlinux doesn't, so any "atomic" image would need GRUB2. > But *that* said, the current packaging means that grub2 adds 70MB on > disk — about 12% of the entire cloud image. I'm not saying grub2 is > evil, just that this is a big portion of the gain and is worth > attacking in order to reverse it. I talked briefly to Peter Jones and > he says there's quite a bit which can be done there, if going back to > extlinux doesn't seem like a viable option. These are needed on BIOS systems grub2-tools is 7MiB grub2 is 3.8MiB These are needed on UEFI systems shim-signed is 633KiB grub2-efi is 837KiB So, I'm not sure where 70MB is coming from. All four are needed for the hybrid BIOS+UEFI supported media. So yeah I'd say it can shrink quite a bit. -- Chris Murphy -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Mon, Sep 21, 2015 at 05:11:24PM +0200, Reindl Harald wrote: > >5423400 xkeyboard-config # really looks like a systemd dep chain > >involving plymouth > plymouth luckily is no hard dependency for anything "Involving plymouth" might have been wrong. libxkbcommon.so.0 looks like a direct dependency, from /usr/lib/systemd/systemd-localed. I also don't know how plymouth crept in, though. -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Mon, Sep 21, 2015 at 11:07:40AM -0400, Matthew Miller wrote: > Okay, let's look on disk: > > > [f21]$ sudo du -sh * 2>/dev/null|sort -h > [...] > 36K home > 40K root > 228Krun > 21M boot > 22M etc > 34M var > 276Musr I find that KDE filelight (also GNOME baobab) are great for visualizing exactly where space is going. Some pictures: https://rwmj.wordpress.com/2015/07/23/why-has-the-libguestfs-appliance-grown-by-118-mb/#content Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
Am 21.09.2015 um 17:07 schrieb Matthew Miller: 5423400 xkeyboard-config # really looks like a systemd dep chain involving plymouth plymouth luckily is no hard dependency for anything [root@rawhide ~]# rpm -qa | grep keyboard xkeyboard-config-2.15-2.fc23.noarch [root@rawhide ~]# rpm -qa | grep ply python-ply-3.6-2.fc23.noarch signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On 21/09/15 16:07, Matthew Miller wrote: > Fedora-Cloud-Base-20141203-21.x86_64.qcow2: 151M > Fedora-Cloud-Base-23_Beta-20150915.x86_64.qcow2: 275M > > In just one year — 82% more awesome? > > I'd really like this to stay below 200MB as a competitive threshold. > Or, if we're going to be bigger than that, be bigger for REASONS, not > just accretion. > > tl;dr: grub2 is a lot to blame, but there seem to be some new > questionable dep chains from systemd, and general dep growth across the > board. > > > Disk use at first boot: > > [f21]$ df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/vda120G 359M 19G 2% / > > [f23b]$ df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/vda120G 578M 19G 4% / > > RPMs installed: > > [f21]$ rpm -qa | wc -l > 226 > > [f23b]$ rpm -qa | wc -l > 264 > > Top 20 rpms by reported size: > > $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 > 120417342 glibc-common > 42307839 kernel-core > 25000497 python-libs > 22438155 systemd > 14623272 coreutils > 14000291 glibc > 11282056 ruby-libs # hey, at least we lost this > 10845519 glib2 > 10593004 selinux-policy-targeted > 9389116 cracklib-dicts # https://bugzilla.redhat.com/show_bug.cgi?id=865521 > 9078043 python-boto > 8792531 util-linux > 7084188 bash > 6669884 gnupg2 > 5844544 yum > 4893790 policycoreutils > 3786564 file-libs > 3540004 shadow-utils > 3458312 groff-base # who doesn't love groff? > 2997717 tar > > > $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 > 125195206 glibc-common > 86298752 linux-firmware # sadface, but hard > 53291365 kernel-core > 36004297 grub2-tools # this is ridiculous > 28453336 python3-libs # 13% growth > 27233273 systemd # 21% growth > 16648994 grub2# *sigh* > 14486819 glibc > 14287847 coreutils# this package got _smaller!_ ... and more secure, due to lots of hard work minimizing dependencies, like using glibc's printf implementation rather than our own. Thanks for noticing :) Note coreutils also got a new ./configure --enable-single-binary option recently to build all tools to a single binary like busybox. The tradeoff is somewhat higher startup cost, and RAM cost, due to all shared libs being linked. Chromium is using this build configuration I understand. cheers, Pádraig. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Fedora 23 cloud image (and, for that matter, minimal anything) bloat
Fedora-Cloud-Base-20141203-21.x86_64.qcow2: 151M Fedora-Cloud-Base-23_Beta-20150915.x86_64.qcow2: 275M In just one year — 82% more awesome? I'd really like this to stay below 200MB as a competitive threshold. Or, if we're going to be bigger than that, be bigger for REASONS, not just accretion. tl;dr: grub2 is a lot to blame, but there seem to be some new questionable dep chains from systemd, and general dep growth across the board. Disk use at first boot: [f21]$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/vda120G 359M 19G 2% / [f23b]$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/vda120G 578M 19G 4% / RPMs installed: [f21]$ rpm -qa | wc -l 226 [f23b]$ rpm -qa | wc -l 264 Top 20 rpms by reported size: $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 120417342 glibc-common 42307839 kernel-core 25000497 python-libs 22438155 systemd 14623272 coreutils 14000291 glibc 11282056 ruby-libs # hey, at least we lost this 10845519 glib2 10593004 selinux-policy-targeted 9389116 cracklib-dicts # https://bugzilla.redhat.com/show_bug.cgi?id=865521 9078043 python-boto 8792531 util-linux 7084188 bash 6669884 gnupg2 5844544 yum 4893790 policycoreutils 3786564 file-libs 3540004 shadow-utils 3458312 groff-base # who doesn't love groff? 2997717 tar $ rpm -qa --qf '%{size} %{name}\n'|sort -nr|head -20 125195206 glibc-common 86298752 linux-firmware # sadface, but hard 53291365 kernel-core 36004297 grub2-tools # this is ridiculous 28453336 python3-libs # 13% growth 27233273 systemd # 21% growth 16648994 grub2# *sigh* 14486819 glibc 14287847 coreutils# this package got _smaller!_ 11143743 glib2 11129880 selinux-policy-targeted 9389116 cracklib-dicts 9261499 python3-boto 9237998 util-linux 9224255 fedora-logos # this is also grub's fault. 7517574 gnupg2 7143418 bash 6574678 python3-pip # :( 583 hwdata# this is ALSO grub's fault 5423400 xkeyboard-config # really looks like a systemd dep chain involving plymouth Okay, let's look on disk: [f21]$ sudo du -sh * 2>/dev/null|sort -h [...] 36K home 40K root 228K run 21M boot 22M etc 34M var 276M usr [f23b]$ sudo du -sh * 2>/dev/null|sort -h [...] 40K root 264K run 16M etc 45M boot # ugh 171M var # oww 463M usr # oww oww oww Breakdown: - boot is mostly grub, but initramfs is also doubled - var: DNF cache is full of stuff! 123M... oh, hey, that wasn't there when we booted. `df -h /` is now up to 702M. Maybe multiple repos would help here, or some other more clever design. Does every cloud image in the world _really_ need to replicate dependency data so I can `dnf install /usr/share/minetest/builtin/mainmenu/init_simple.lua`? - usr: * linux-firmware is the biggest thing here * python3 is another good chunk * also kernel modules, but, eh, whatchagonnado * oh, and of course, grub2. -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Mon, Sep 21, 2015 at 03:39:46PM -0400, Josh Boyer wrote: > It's worth pointing out that your griping about grub2 "growth" seems > misleading. The sizes of the grub2 packages did not change between > F22 and F23. What seems to have happened is that the cloud images > added the grub2 packages, which weren't there in F22. Sorry, yeah -- that griping is not out of context in the Cloud WG where I also posted this, but is here. Grub2 has the huge virtue of *It Actually Works*, a claim which currently none of the competition can make. But *that* said, the current packaging means that grub2 adds 70MB on disk — about 12% of the entire cloud image. I'm not saying grub2 is evil, just that this is a big portion of the gain and is worth attacking in order to reverse it. I talked briefly to Peter Jones and he says there's quite a bit which can be done there, if going back to extlinux doesn't seem like a viable option. But grub2 isn't the only growth — only about a third of it. -- Matthew MillerFedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Fedora 23 cloud image (and, for that matter, minimal anything) bloat
On Mon, Sep 21, 2015 at 11:07 AM, Matthew Millerwrote: > Fedora-Cloud-Base-20141203-21.x86_64.qcow2: 151M > Fedora-Cloud-Base-23_Beta-20150915.x86_64.qcow2: 275M > > In just one year — 82% more awesome? > > I'd really like this to stay below 200MB as a competitive threshold. > Or, if we're going to be bigger than that, be bigger for REASONS, not > just accretion. > > tl;dr: grub2 is a lot to blame, but there seem to be some new > questionable dep chains from systemd, and general dep growth across the > board. It's worth pointing out that your griping about grub2 "growth" seems misleading. The sizes of the grub2 packages did not change between F22 and F23. What seems to have happened is that the cloud images added the grub2 packages, which weren't there in F22. josh -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct