Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
> On 30/09/12 06:15 PM, Brian Harring wrote:
>
> yngwin has a point that I've not seen addressed.
>
> What /is/ wrong with the whole CDEPEND intermediate var idea?
>
The problem appears as we introduce more DEPEND variables
(which is what prompted the proposal, IIRC). If we have
ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some
(i.e. not total) sharing going on then the COMMON_DEPEND
pattern starts to fall apart. You potentially need,
AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
(COMMON_DEPEND)
This obviously gets worse as more DEPEND vars are introduced.
>>>
>>> Well not really, no -- the additional *DEPENDs that are being
>>> proposed (or at least mentioned) for new EAPI will either remove
>>> atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so
>>> tersely that a COMMON_DEPEND or other intermediate variable won't
>>> really be necessary for them.
Another thing I wanted to point out is that those "potential" extra
variables are not needed in practice. We already have 98% of the tree
(if I got the previously mentioned stats right) that does fine with
just one or two ({R,}DEPEND). The majority of that other 2% needs just
one more variable. There may be corner cases where more vars would be
needed, but those will never be more than a few ebuilds.
It's just not worth it to completely change the way we do things (or
use two systems in parallel) just for a few ebuilds that would
significantly benefit.
If we were a new distro and designing our ebuild format from scratch,
then yes, I would say your proposal has merit. But we aren't. We have
hundreds of people and tens of thousands of ebuilds using *DEPEND just
fine. There are no big problems, only corner-cases. We're not talking
about incremental improvements either (such as was the case e.g. with
use deps).
Let's just keep things simple, and refrain from "fixing" what isn't broken.
--
Cheers,
Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin
Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 30/09/12 06:15 PM, Brian Harring wrote: > > Pardon the belated response; responding to emails that are quick > where possible, but lagging on -dev. Missed this one however... > No worries, there's a lot going on.. :D yngwin has a point that I've not seen addressed. What /is/ wrong with the whole CDEPEND intermediate var idea? >>> >>> The problem appears as we introduce more DEPEND variables >>> (which is what prompted the proposal, IIRC). If we have >>> ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some >>> (i.e. not total) sharing going on then the COMMON_DEPEND >>> pattern starts to fall apart. You potentially need, >>> >>> AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND >>> ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND >>> (COMMON_DEPEND) >>> >>> This obviously gets worse as more DEPEND vars are introduced. >>> >> >> Well not really, no -- the additional *DEPENDs that are being >> proposed (or at least mentioned) for new EAPI will either remove >> atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so >> tersely that a COMMON_DEPEND or other intermediate variable won't >> really be necessary for them. > > It depends on the dep type actually, and how we're viewing those > deps- if they must be complete or not. [ .. Snip! .. ] > > The point I'm trying to make here is that each dep phase should be > authorative; in doing so, you start getting a lot of potential > subsets (DEPEND is a subset of TDEPEND, TDEPEND isn't completely, > but mostly a subset of RDEPEND as RDEPEND is a likely a superset of > DEPEND; PDEPEND is a superset of RDEPEND). > > So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in > the ebuild. Or you could just use a syntax form that allows you > to directly inline that up front, rather than having to muck around > w/ intermediate vars. > ... I think what you've just described here might be where the primary difference in thinking is for most of us, between moving to DEPENDENCIES and keeping the current *DEPENDs -- to me, from an ebuild writer's perspective, the *DEPENDS -shouldn't- be authoritative. IE, instead of thinking of PDEPEND as a superset of RDEPEND I consider they are two separate sets, which should not intersect, and are unioned together to form the full set of runtime dependencies. IE: FULL_RUNTIME_DEPEND="$RDEPEND $PDEPEND" somewhere inside portage, if portage actually needed it this way. And I see this as how many of the other proposed new *DEPENDs would work too, ie, they are a refined subset of the bigger sets and should not intersect with the 'parent' *DEPEND that was used instead on older EAPIs. So if this were to change, it might make sense (as Duncan i think pointed out in his response to this message), to a debate on whether or not ebuilds must specify an authoritative list for each dep phase. (I haven't read through PMS but I'm going to assume that it doesn't specify this anywhere yet--and if it does, i'm sure Ciaran or someone will quote it in response :) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBrKKsACgkQ2ugaI38ACPBA9wD/a+XVf2g9s6dcpW1513aXQpYV tK94QdLkax0Hs6tKFqcA/0+v6oFON2Bd3hedv9DHg7N42NNvtTKK6sOw18OpL0aO =frmC -END PGP SIGNATURE-
[gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
Brian Harring posted on Sun, 30 Sep 2012 15:15:18 -0700 as excerpted: > The point I'm trying to make here is that each dep phase should be > authorative; in doing so, you start getting a lot of potential subsets > (DEPEND is a subset of TDEPEND, TDEPEND isn't completely, but mostly a > subset of RDEPEND as RDEPEND is a likely a superset of DEPEND; PDEPEND > is a superset of RDEPEND). > > So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in the > ebuild. Or you could just use a syntax form that allows you to directly > inline that up front, rather than having to muck around w/ intermediate > vars. Thanks /very/ much! You said you weren't sure you were being clear, but this is the first time I've /really/ understood what must surely be the root, at any reasonable level at all. Let me see if I've got it right: Yes, in some ways all we're dealing with here is "optics", but the _problem_ is that with the proposed proliferation in detailed depend- types, what is now a simple CDEPEND and thus conceptually easy to handle, breaks into 10/20/50/whatever-large-number different shards, and what's conceptually easy to handle /now/ becomes many many times more difficult to handle, both conceptually for package maintainers and practically for iterative resolution in the PMs, due to the interplay of all the resulting *CDEPENDs. The proposed solution to that explosion in conceptual complexity not only changes the "optics", but by making most of those detail-depend-types absolute/authoritative, allows both package managers (the programs, machine) and package maintainers (the humans) to consider each depend- type separately, thus decreasing both conceptual complexity to a once again manageable level for package maintainers (humans), and practical complexity for package managers (machine), increasing efficiency, reducing resolution time and probably eventually memory/installed-db/ cache size as well. Of course now I better understand Ciaran's argument for labels as well, since it would extend the absolute/authoritative principle even further, into the actual deps specification method in ebuilds/eclasses, thereby reducing conceptual context load even further via more explicitly absolute deps at the local level. But like you, in practice I don't see that going anywhere in gentoo, in the near/short-intermediate future, primarily due to political realities, but practically, also due to the conceptual leap it'd require from devs (as Ciaran himself points out in response to your statistical analysis of exherbo's repo, former gentoo devs simply don't tend to take advantage of this aspect of labels in exherbo either; the conceptual leap is in practice simply too much). Thus, while academically, his label approach is arguably better in terms of efficiency of absolutes, in practice, there's little or no difference between how it's used, and how your filtering approach will be used. Further, given the conceptual distance between labels and gentoo's current approach, with filters falling in between and political reality, the pragmatic filters approach at least has /some/ chance of passing the dev-debate stage and being approved, implemented and actually available for use in something like a reasonable timeframe (say EAPI-6, in a year's time, a bit more for actual in-tree use, given the historic EAPI-a-year processing). But exherbo style labels support altho academically better, given political reality, in all likelihood would take at least 2-3 years to pass and be usable in-tree. And even then, its practical use as demonstrated in exherbo wouldn't take advantage of the differences for another couple years after that, at least. Given that, having use of the useful pragmatic approach in a year's time or so seems best, as opposed to an arguably (as you've pointed out, no practical demonstration of it in exherbo yet, at least that you've been able to find) more academically ideal approach in three, without any real benefit over the pragmatic for five or more. >> Besides, this isn't actually a -problem- as there's nothing which >> really requires one to use such helpers; ebuild writers just, well, >> can. :) > > Getting to this point; yes, people could hack around it manually. Pretty > sure that hasn't been in doubt. That's clearer now. Yes, people could continue to hack around it via CDEPENDS, etc. But the number of common vars (or alternative RDEPEND="$DEPEND ..." assignments) and the resulting conceptual load on the human maintainer is set to increase exponentially as the number of depend-types increases linearly. At some point it's just no longer practically maintainable, and the whole process breaks down. What the single dependencies variable aims to do in both the filtering and labels forms is prevent that ultimate conceptual overload and resulting process breakdown by allowing direct compound assignments, thus eliminating the intermediate assignmen
Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
Pardon the belated response; responding to emails that are quick where possible, but lagging on -dev. Missed this one however... On Wed, Sep 19, 2012 at 09:16:02AM -0400, Ian Stakenvicius wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > On 19/09/12 09:09 AM, Michael Orlitzky wrote: > > On 09/19/2012 06:59 AM, Duncan wrote: > >> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as > >> excerpted: > >> > >>> On 16 September 2012 21:15, Brian Harring > >>> wrote: > >> > So... basically, people are already doing this manually with > their own intermediate vars. > >>> > >>> And this works fine, so it doesn't warrant a cosmetic change. > >> > >> @ferringb: > >> > >> yngwin has a point that I've not seen addressed. > >> > >> What /is/ wrong with the whole CDEPEND intermediate var idea? It > >> seems to work and /I/ don't know of any problems with it (and it > >> would appear, neither does yngwin), yet you talk about it as if > >> there's something wrong with it. > >> > >> And while we're at it, do DEPEND="$RDEPEND ..." style solutions > >> have the same problems (or lack thereof)? > > > > The problem appears as we introduce more DEPEND variables (which is > > what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND, > > CDEPEND, and DDEPEND, and there's only some (i.e. not total) > > sharing going on then the COMMON_DEPEND pattern starts to fall > > apart. You potentially need, > > > > AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND > > ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND > > (COMMON_DEPEND) > > > > This obviously gets worse as more DEPEND vars are introduced. > > > > Well not really, no -- the additional *DEPENDs that are being proposed > (or at least mentioned) for new EAPI will either remove atoms from > COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a > COMMON_DEPEND or other intermediate variable won't really be necessary > for them. It depends on the dep type actually, and how we're viewing those deps- if they must be complete or not. Consider test depends for example. Either we can specify it as "in addition to depend"- which sucks, because it's entirely possible for a DEPEND target to not be required for TDEPEND. While that's a corner case, it's actually easy to address; just require TDEPEND to be comprehensive for the test phase. Thus DEPEND and TDEPEND suddenly share a lot, and TDEPEND shares part of RDEPEND. Moving on to a real world example... PDEPEND. Currently a full graph resolution requires pulling RDEPEND and PDEPEND, collapsing them, and ensuring they're fullfilled for anything that is already installed (ie, PDEPEND is required after the transaction is completed). In our current separated var setup, this as said, this requires the PM to track/handle it separately. However, were PDEPEND to be complete/full- that would mean the PM could just render for dep:post and know "this is what is necessary outside of the transactional block of building/installing it". So RDEPEND and PDEPEND actually share a *shitton*- they're the same contents in 97% of the tree. Meaning dep:run,post? for 97% of the tree, w/ 3% needing to have an addition dep:post? section. Offhand, comprehensive deps make things easier for devs- it gives them the ability to be crystal-freaking-clear as to what's needed at each stage; via them doing so, it means the resolver has a greater space to dig itself out of fucked up situations if necessary. Additionally, it actually makes life easier for PM authors. If we had TDEPEND (test), this is how things would go; note this is written for just an install, not a replacement (replace is similar, just noisier deflecting from my point): 1) ensure DEPEND is satisfied 2) run phases setup -> compile 3) ensure TDEPEND is satisfied; Implicitly, DEPEND in the process (if there is a cycle induced by TDEPEND + DEPEND being required, we have no way out of it). 4) run install phase 5) ensure RDEPEND is satisfied. No longer care about TDEPEND/DEPEND. 6) run pkg_preinst, merge, pkg_postinst. 7) a transaction is opened up, that must be completed before the PM exits; before that exit, RDEPEND must still be satisfied, as must PDEPEND. With what I'm suggesting. 1) ensure dep:build is satisfied 2) run phases setup -> compile 3) Ensure dep:test is satisfied. If cycle breaking is necessary, anything in dep:build but not dep:test can be pulled. 4) install phase 5) ensure dep:run is satisfied. Again, if cycle breaking is necessary, punt whatever isn't in dep:run as necessary to break that cycle. 6) pkg_preinst, merge, pkg_postinst. 7) transactional block; ensure dep:post is satisfied before exiting said transaction. That's for install; for the PM considering a pre-installed pkg, it converts from; 1) ensure RDEPEND and PDEPEND are satisfied to 1) ensure dep:post is satisfied. Tbh, I'm not sure I'm communicating these gains as well as I could be; at first glance, I'm
Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 19/09/12 09:09 AM, Michael Orlitzky wrote: > On 09/19/2012 06:59 AM, Duncan wrote: >> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as >> excerpted: >> >>> On 16 September 2012 21:15, Brian Harring >>> wrote: >> So... basically, people are already doing this manually with their own intermediate vars. >>> >>> And this works fine, so it doesn't warrant a cosmetic change. >> >> @ferringb: >> >> yngwin has a point that I've not seen addressed. >> >> What /is/ wrong with the whole CDEPEND intermediate var idea? It >> seems to work and /I/ don't know of any problems with it (and it >> would appear, neither does yngwin), yet you talk about it as if >> there's something wrong with it. >> >> And while we're at it, do DEPEND="$RDEPEND ..." style solutions >> have the same problems (or lack thereof)? > > The problem appears as we introduce more DEPEND variables (which is > what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND, > CDEPEND, and DDEPEND, and there's only some (i.e. not total) > sharing going on then the COMMON_DEPEND pattern starts to fall > apart. You potentially need, > > AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND > ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND > (COMMON_DEPEND) > > This obviously gets worse as more DEPEND vars are introduced. > Well not really, no -- the additional *DEPENDs that are being proposed (or at least mentioned) for new EAPI will either remove atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a COMMON_DEPEND or other intermediate variable won't really be necessary for them. Besides, this isn't actually a -problem- as there's nothing which really requires one to use such helpers; ebuild writers just, well, can. :) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBZxZIACgkQ2ugaI38ACPDp4wD/atjvaOsi/ntDMB1Dj7lSAVmW 45qKz6+OO+H/+6eFeVIA/Rz0s7FiG6d2frboHXpYrDBzM1FZcU85AqZti34tR8+h =E78Z -END PGP SIGNATURE-
Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
On 09/19/2012 06:59 AM, Duncan wrote: > Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as excerpted: > >> On 16 September 2012 21:15, Brian Harring wrote: > >>> So... basically, people are already doing this manually with their own >>> intermediate vars. >> >> And this works fine, so it doesn't warrant a cosmetic change. > > @ferringb: > > yngwin has a point that I've not seen addressed. > > What /is/ wrong with the whole CDEPEND intermediate var idea? It seems > to work and /I/ don't know of any problems with it (and it would appear, > neither does yngwin), yet you talk about it as if there's something wrong > with it. > > And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the > same problems (or lack thereof)? The problem appears as we introduce more DEPEND variables (which is what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some (i.e. not total) sharing going on then the COMMON_DEPEND pattern starts to fall apart. You potentially need, AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND (COMMON_DEPEND) This obviously gets worse as more DEPEND vars are introduced.
[gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as excerpted: > On 16 September 2012 21:15, Brian Harring wrote: >> So... basically, people are already doing this manually with their own >> intermediate vars. > > And this works fine, so it doesn't warrant a cosmetic change. @ferringb: yngwin has a point that I've not seen addressed. What /is/ wrong with the whole CDEPEND intermediate var idea? It seems to work and /I/ don't know of any problems with it (and it would appear, neither does yngwin), yet you talk about it as if there's something wrong with it. And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the same problems (or lack thereof)? FWIW I personally like the whole single-var idea, and CERTAINLY appreciate the various statistical cache savings, etc. If we were starting from scratch now, I'd definitely favor the single var approach. But the combined developer mental cost of having to learn the new method and then maintain a working understanding of both over some longer period is nothing to sneeze at, and I'm not entirely convinced that it's worth that cost, even assuming a doubling of the number of dependency types with a lot of commonality between them, and the added benefit a single deps var would have in that case. And the case for a single deps var isn't being helped by the implication that there's something wrong with both the intermediate var and copying var methods, without ever saying what that "wrong" might be, in the face of the experience of many that those existing methods "just work". So if there's something wrong with them, let's get it out there where people can see it. And if there isn't, please eliminate the noise of that implication from the argument. Thanks. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
