Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, 8 May 2019 11:08:47 -0700 Matt Turner wrote: > On Wed, May 8, 2019 at 3:19 AM Alexis Ballier > wrote: > > > > On Wed, 08 May 2019 12:01:21 +0200 > > Michał Górny wrote: > > > > > On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote: > > > > On Wed, 08 May 2019 11:41:41 +0200 > > > > Michał Górny wrote: > > > > > > > > > > There's multilib that adds a lot of flags with a single > > > > > > eclass change, but I'd guess the number of packages and > > > > > > flags is constantly growing, so sooner or later you'll be > > > > > > hit by this again and no multilib killing will help you > > > > > > then. > > > > > > > > > > > > I think it is more future proof to use the addition of > > > > > > multilib flags to fix pkgcheck rather than actively > > > > > > reducing the number of multilib flags to cope with its > > > > > > limitations. > > > > > > > > > > Then please do it, by all means. The reality is simple. If > > > > > the tool is broken, you either fix it or stop doing what you > > > > > know that breaks it. Being unable to do the former, and > > > > > having no good replacement, I'd go for the latter. > > > > > > > > Well, why is it slow ? IO ? CPU ? Did you collect profiling > > > > data ? > > > > > > CPU definitely. More detail than that, I don't and I don't have > > > time to investigate. > > > > So you don't have time to change 3 lines to add cProfile but do have > > time to send various emails and rework the entire multilib system ? > > weird. > > This isn't productive. > > If you'd like to do the work you're suggesting, I'm sure Michał will > support that, but as is you're just passive-aggressively questioning > his choices in the regards to the multilib system he created and the > CI system he created. > You're right about the passive-aggressive part, sorry about it. But maybe there's a single not so important check that's killing the performance and could be, instead, disabled meanwhile. No data has been shown except that adding useflags to packages make CI explode. I am indeed questioning the choices of making tree wide changes because a CI system is under performing and see nothing wrong about this.
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, May 8, 2019 at 3:19 AM Alexis Ballier wrote: > > On Wed, 08 May 2019 12:01:21 +0200 > Michał Górny wrote: > > > On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote: > > > On Wed, 08 May 2019 11:41:41 +0200 > > > Michał Górny wrote: > > > > > > > > There's multilib that adds a lot of flags with a single eclass > > > > > change, but I'd guess the number of packages and flags is > > > > > constantly growing, so sooner or later you'll be hit by this > > > > > again and no multilib killing will help you then. > > > > > > > > > > I think it is more future proof to use the addition of multilib > > > > > flags to fix pkgcheck rather than actively reducing the number > > > > > of multilib flags to cope with its limitations. > > > > > > > > Then please do it, by all means. The reality is simple. If the > > > > tool is broken, you either fix it or stop doing what you know > > > > that breaks it. Being unable to do the former, and having no good > > > > replacement, I'd go for the latter. > > > > > > Well, why is it slow ? IO ? CPU ? Did you collect profiling data ? > > > > CPU definitely. More detail than that, I don't and I don't have time > > to investigate. > > So you don't have time to change 3 lines to add cProfile but do have > time to send various emails and rework the entire multilib system ? > weird. This isn't productive. If you'd like to do the work you're suggesting, I'm sure Michał will support that, but as is you're just passive-aggressively questioning his choices in the regards to the multilib system he created and the CI system he created.
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, 08 May 2019 12:31:47 +0200 Michał Górny wrote: > On Wed, 2019-05-08 at 12:19 +0200, Alexis Ballier wrote: > > On Wed, 08 May 2019 12:01:21 +0200 > > Michał Górny wrote: > > > > > On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote: > > > > On Wed, 08 May 2019 11:41:41 +0200 > > > > Michał Górny wrote: > > > > > > > > > > There's multilib that adds a lot of flags with a single > > > > > > eclass change, but I'd guess the number of packages and > > > > > > flags is constantly growing, so sooner or later you'll be > > > > > > hit by this again and no multilib killing will help you > > > > > > then. > > > > > > > > > > > > I think it is more future proof to use the addition of > > > > > > multilib flags to fix pkgcheck rather than actively > > > > > > reducing the number of multilib flags to cope with its > > > > > > limitations. > > > > > > > > > > Then please do it, by all means. The reality is simple. If > > > > > the tool is broken, you either fix it or stop doing what you > > > > > know that breaks it. Being unable to do the former, and > > > > > having no good replacement, I'd go for the latter. > > > > > > > > Well, why is it slow ? IO ? CPU ? Did you collect profiling > > > > data ? > > > > > > CPU definitely. More detail than that, I don't and I don't have > > > time to investigate. > > > > So you don't have time to change 3 lines to add cProfile but do have > > time to send various emails and rework the entire multilib system ? > > weird. > > Instead of being so smug, you could have provided helpful instructions > on how to do it. Or did it yourself. ;) I wasn't being smug, rather the contrary by assuming you would know better than me. So here we go: https://docs.python.org/2/library/profile.html $ python -m cProfile -o /full/path/to/file.log /usr/bin/command args This will give you a profile log. Then you need something to visualize it, I like runsnakerun. http://www.vrplumber.com/programming/runsnakerun/ > Removing unused flags from multilib-build.eclass is certainly less > work than that. As a temporarily solution yes.
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, 2019-05-08 at 12:19 +0200, Alexis Ballier wrote: > On Wed, 08 May 2019 12:01:21 +0200 > Michał Górny wrote: > > > On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote: > > > On Wed, 08 May 2019 11:41:41 +0200 > > > Michał Górny wrote: > > > > > > > > There's multilib that adds a lot of flags with a single eclass > > > > > change, but I'd guess the number of packages and flags is > > > > > constantly growing, so sooner or later you'll be hit by this > > > > > again and no multilib killing will help you then. > > > > > > > > > > I think it is more future proof to use the addition of multilib > > > > > flags to fix pkgcheck rather than actively reducing the number > > > > > of multilib flags to cope with its limitations. > > > > > > > > Then please do it, by all means. The reality is simple. If the > > > > tool is broken, you either fix it or stop doing what you know > > > > that breaks it. Being unable to do the former, and having no good > > > > replacement, I'd go for the latter. > > > > > > Well, why is it slow ? IO ? CPU ? Did you collect profiling data ? > > > > CPU definitely. More detail than that, I don't and I don't have time > > to investigate. > > So you don't have time to change 3 lines to add cProfile but do have > time to send various emails and rework the entire multilib system ? > weird. Instead of being so smug, you could have provided helpful instructions on how to do it. Or did it yourself. Removing unused flags from multilib-build.eclass is certainly less work than that. -- Best regards, Michał Górny
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, 08 May 2019 12:01:21 +0200 Michał Górny wrote: > On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote: > > On Wed, 08 May 2019 11:41:41 +0200 > > Michał Górny wrote: > > > > > > There's multilib that adds a lot of flags with a single eclass > > > > change, but I'd guess the number of packages and flags is > > > > constantly growing, so sooner or later you'll be hit by this > > > > again and no multilib killing will help you then. > > > > > > > > I think it is more future proof to use the addition of multilib > > > > flags to fix pkgcheck rather than actively reducing the number > > > > of multilib flags to cope with its limitations. > > > > > > Then please do it, by all means. The reality is simple. If the > > > tool is broken, you either fix it or stop doing what you know > > > that breaks it. Being unable to do the former, and having no good > > > replacement, I'd go for the latter. > > > > Well, why is it slow ? IO ? CPU ? Did you collect profiling data ? > > CPU definitely. More detail than that, I don't and I don't have time > to investigate. So you don't have time to change 3 lines to add cProfile but do have time to send various emails and rework the entire multilib system ? weird. > > > > Also, remember that multilib is not entirely about skype or > > > > slack, this was made with multibin in mind too: for example an > > > > ABI may perform better than another one on specific workflows > > > > (x32) and it may make sense to use this abi for a specific > > > > binary (which would be manually built for now). > > > > > > No, it weren't. Just because someone found it convenient to use > > > the design beyond its original purpose doesn't mean it had a > > > different purpose. Besides, how many 'multibin' packages do we > > > have right now? > > > > It was, because portage multilib was doing this and claimed to have > > a use for this. > > I am not talking of 'portage multilib'. I am talking of > multilib-build implementation. If you are only concerned about the > former, we can surely drop those abi_* flags that are irrelevant to > it, can't we? I am also talking about multilib-build and its way of doing cleanly and replacing portage multilib. > > Its original purpose was generic enough to allow multibin > > to be added easily. The number of multibin packages is only > > relevant to show that this is not important enough for someone > > to step up and do it: Everybody can easily build anything themselves > > with the current implementation. > > Exactly. And that's why I believe having fast CI is more important > than DIY multibin. If someone can 'easily build anything > themselves', they can also add needed flags locally. It's a "little" more complex than just adding one flag to an eclass and cannot really be done with an overlay; in the end, this requires to fork gentoo.git more or less.
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote: > On Wed, 08 May 2019 11:41:41 +0200 > Michał Górny wrote: > > > > There's multilib that adds a lot of flags with a single eclass > > > change, but I'd guess the number of packages and flags is > > > constantly growing, so sooner or later you'll be hit by this again > > > and no multilib killing will help you then. > > > > > > I think it is more future proof to use the addition of multilib > > > flags to fix pkgcheck rather than actively reducing the number of > > > multilib flags to cope with its limitations. > > > > Then please do it, by all means. The reality is simple. If the tool > > is broken, you either fix it or stop doing what you know that breaks > > it. Being unable to do the former, and having no good replacement, > > I'd go for the latter. > > Well, why is it slow ? IO ? CPU ? Did you collect profiling data ? CPU definitely. More detail than that, I don't and I don't have time to investigate. > Where are the scripts to repro the issue ? https://github.com/mgorny/pkgcore/tree/repo-mirror-ci https://github.com/mgorny/pkgcheck/tree/repo-mirror-ci (repo-mirror-ci branch in both cases) I suppose master branch suffers from the same problem but I didn't find time to check (and it is not very helpful until I find time to rebase all my fixes that weren't merged before refactoring everything). > > > Also, remember that multilib is not entirely about skype or slack, > > > this was made with multibin in mind too: for example an ABI may > > > perform better than another one on specific workflows (x32) and it > > > may make sense to use this abi for a specific binary (which would > > > be manually built for now). > > > > No, it weren't. Just because someone found it convenient to use the > > design beyond its original purpose doesn't mean it had a different > > purpose. Besides, how many 'multibin' packages do we have right now? > > It was, because portage multilib was doing this and claimed to have a > use for this. I am not talking of 'portage multilib'. I am talking of multilib-build implementation. If you are only concerned about the former, we can surely drop those abi_* flags that are irrelevant to it, can't we? > Its original purpose was generic enough to allow multibin > to be added easily. The number of multibin packages is only > relevant to show that this is not important enough for someone > to step up and do it: Everybody can easily build anything themselves > with the current implementation. Exactly. And that's why I believe having fast CI is more important than DIY multibin. If someone can 'easily build anything themselves', they can also add needed flags locally. -- Best regards, Michał Górny
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, 08 May 2019 11:41:41 +0200 Michał Górny wrote: > > There's multilib that adds a lot of flags with a single eclass > > change, but I'd guess the number of packages and flags is > > constantly growing, so sooner or later you'll be hit by this again > > and no multilib killing will help you then. > > > > I think it is more future proof to use the addition of multilib > > flags to fix pkgcheck rather than actively reducing the number of > > multilib flags to cope with its limitations. > > Then please do it, by all means. The reality is simple. If the tool > is broken, you either fix it or stop doing what you know that breaks > it. Being unable to do the former, and having no good replacement, > I'd go for the latter. Well, why is it slow ? IO ? CPU ? Did you collect profiling data ? Where are the scripts to repro the issue ? > > Also, remember that multilib is not entirely about skype or slack, > > this was made with multibin in mind too: for example an ABI may > > perform better than another one on specific workflows (x32) and it > > may make sense to use this abi for a specific binary (which would > > be manually built for now). > > No, it weren't. Just because someone found it convenient to use the > design beyond its original purpose doesn't mean it had a different > purpose. Besides, how many 'multibin' packages do we have right now? It was, because portage multilib was doing this and claimed to have a use for this. Its original purpose was generic enough to allow multibin to be added easily. The number of multibin packages is only relevant to show that this is not important enough for someone to step up and do it: Everybody can easily build anything themselves with the current implementation. > That said, yes, I am also concerned about people proactively adding > multilib to all random libraries which probably will never have any > real multilib use case and only waste time of people who have > abi_x86_32 enabled globally. That last part is easily fixable: Have a useflag config option that says "default off unless some package wants it on". This can be either in the ebuild themselves (alike +/- iuse defaults) or, more simply, in the package manager. None ever saw the light.
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Wed, 2019-05-08 at 11:29 +0200, Alexis Ballier wrote: > On Tue, 07 May 2019 23:47:30 +0200 > Michał Górny wrote: > > > While the large number of flags is practically invisible to user with > > all the USE_EXPAND hiding, it negatively impacts pkgcheck. When > > the number reached 10, CI became unusable. We're currently back down > > to 8, thanks to powerpc team, but the problem is going to happen again > > sooner or later. Ideally we'd improve pkgcheck but I'm not aware of > > anyone having a good idea how to do it. > > While I don't disagree with your rationale below, I think this > motivation is the wrong one: What sort of algorithm does it use > to explode when going from 8 to 10 flags ?!? A poor one. > > There's multilib that adds a lot of flags with a single eclass change, > but I'd guess the number of packages and flags is constantly growing, > so sooner or later you'll be hit by this again and no multilib killing > will help you then. > > I think it is more future proof to use the addition of multilib flags > to fix pkgcheck rather than actively reducing the number of multilib > flags to cope with its limitations. Then please do it, by all means. The reality is simple. If the tool is broken, you either fix it or stop doing what you know that breaks it. Being unable to do the former, and having no good replacement, I'd go for the latter. > Also, remember that multilib is not entirely about skype or slack, > this was made with multibin in mind too: for example an ABI may perform > better than another one on specific workflows (x32) and it may make > sense to use this abi for a specific binary (which would be manually > built for now). No, it weren't. Just because someone found it convenient to use the design beyond its original purpose doesn't mean it had a different purpose. Besides, how many 'multibin' packages do we have right now? That said, yes, I am also concerned about people proactively adding multilib to all random libraries which probably will never have any real multilib use case and only waste time of people who have abi_x86_32 enabled globally. -- Best regards, Michał Górny
Re: [gentoo-dev] Rethinking multilib flags in Gentoo
On Tue, 07 May 2019 23:47:30 +0200 Michał Górny wrote: > While the large number of flags is practically invisible to user with > all the USE_EXPAND hiding, it negatively impacts pkgcheck. When > the number reached 10, CI became unusable. We're currently back down > to 8, thanks to powerpc team, but the problem is going to happen again > sooner or later. Ideally we'd improve pkgcheck but I'm not aware of > anyone having a good idea how to do it. While I don't disagree with your rationale below, I think this motivation is the wrong one: What sort of algorithm does it use to explode when going from 8 to 10 flags ?!? There's multilib that adds a lot of flags with a single eclass change, but I'd guess the number of packages and flags is constantly growing, so sooner or later you'll be hit by this again and no multilib killing will help you then. I think it is more future proof to use the addition of multilib flags to fix pkgcheck rather than actively reducing the number of multilib flags to cope with its limitations. Also, remember that multilib is not entirely about skype or slack, this was made with multibin in mind too: for example an ABI may perform better than another one on specific workflows (x32) and it may make sense to use this abi for a specific binary (which would be manually built for now). Alexis.
