Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Wed, 31 Jan 2007 17:47:26 + Stephen Bennett <[EMAIL PROTECTED]> wrote: > On Tue, 30 Jan 2007 17:06:51 +0100 > Marius Mauch <[EMAIL PROTECTED]> wrote: > > > The idea is to add a special category (let's call it "active" for > > now) that has the following properties: > > - this category doesn't exist in portdir or vdb (= no ebuilds) > > - when portage ($pkgmanager) encounters a "active/foo" atom in a > > dependency string it executes some special code (e.g. > > "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if > > that atom is satisfied > > Given that in the general case the package manager can't change the > active provider and will have to bail with an appropriate message that > the user needs to change it themselves, the obvious solution to this is > the previously-discussed-somewhere-I-can't-remember ebuild function, > called on depgraph creation, to check that it will be able to compile > in the current system environment. It's considerably simpler and more > generally useful than subverting DEPEND to add weird special-case hacks > to it. Yeah, thinking about it I have to agree tat generic dep_check() support (to give the thing a name) would be a better solution here. Marius -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
Kevin F. Quinn wrote: > On Wed, 31 Jan 2007 12:27:10 -0500 > Alec Warner <[EMAIL PROTECTED]> wrote: > >>> Hmm; one could get the same benefit by introducing a new interface >>> (e.g. pkg_env_check()) which is defined to return true if the >>> environment is ok, false otherwise (with some text to stdout, >>> perhaps). The package manager would then run this function, after >>> building the depgraph and finding the candidate packages to merge, >>> for each candidate package - if any package fails is env_check, >>> none of the packages get emerged. Note this is then completely >>> independent of depgraph creation. >>> >>> In the 'tr1' example, I'd imagine something like this: >>> >>> use.local.desc: boost: Use boost library for tr1 rather than gcc's. >>> >>> ebuild: >>> >>> ... >>> inherit ... toolchain-funcs versionator ... >>> ... >>> DEPEND=... boost? ( dev-libs/boost ) ... >>> ... >>> pkg_env_check() { >>> use boost && return 0 >>> version_is_at_least "4.1" $(gcc-version) && return 0 >>> echo "Either USE boost, or switch to gcc later than 4.1" >>> } >>> >>> >>> (with a default definition, "pkg_env_check() { return 0; }" ) >> In an ideal system you'd want this stuff in the metadata cache so that >> the resolver can deal with it up front. > > You're talking about the metadata on the host, rather than the stuff on > the rsync servers? I'm not sure you could cache the results even on > the host - you would need to know what could affect the results so as > to know when the cached information is out of date and has to be > recalculated. That would either have to checked on every emerge, or > made a separate switch (i.e. rely on the user to tell emerge when the > environment has changed). I am talking about the host yeah; cache was a bad term on my part; obviously you cannot cache stuff like this. > My concern about dynamic dependencies runs to use deps, as well :) > One could consider use-deps to be a special case of Marius' active > checks. "how pkg P was built" isn't so different from "slot S of P is > active" in terms of dep-graph creation; both are asking about the > state of host & target systems, rather than the tree. > > In the case of USE deps, things are saner because the data doesn't > change without the package manager knowing about it. Effectively the > depgraph becomes static w.r.t. the tree + installation record (rather > than just static w.r.t. the tree). With active checks implemented in > the depgraph, however, that is no longer the case - the depgraph can > change independently of the tree and the installation record. > I am unsure how fast these types of checks would or could be. I mean we can add arbitrary key caching and arbitrary key matching, but then that grows the cache substantially and probably slows down dependency resolution. As you stated, some things just can't be cached properly and still have value. >> With that said, I'm not sure how easy it would be to rewrite that >> code; and this is simpler in that it's just a few bash functions as >> opposed to more resolver foo. > > There's a lot to be said for keeping things simple, of course :) It's > easy enough to mess up things like dep-graphs in any case - introducing > these sorts of dynamic dependencies can render it substantially more > complex. I think the complexity exists already though; currently we are just hiding it, requiring people to find workarounds in an otherwise complex playground of building packages. > > Another way to look at it, is to consider how often this sort of thing > comes up. My understanding is that it is relatively rare; across the > 10,000+ packages in the tree only a handful use 'built_with_use' fex. > That makes a strong case for having a simple solution in the near term, > and re-visit if it becomes commonplace. > I think your understanding is off then. A quick grep (qrep -H built_with_use | wc -l) gives me 814 calls to 'built_with_use' (qgrep is in portage-utils). If you grep through eclasses you will also see 53 separate calls; so you can imagine what the real usage could be (definately at least 1/20th the tree, not something I'd call minor). -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Wed, 31 Jan 2007 12:27:10 -0500 Alec Warner <[EMAIL PROTECTED]> wrote: > > Hmm; one could get the same benefit by introducing a new interface > > (e.g. pkg_env_check()) which is defined to return true if the > > environment is ok, false otherwise (with some text to stdout, > > perhaps). The package manager would then run this function, after > > building the depgraph and finding the candidate packages to merge, > > for each candidate package - if any package fails is env_check, > > none of the packages get emerged. Note this is then completely > > independent of depgraph creation. > > > > In the 'tr1' example, I'd imagine something like this: > > > > use.local.desc: boost: Use boost library for tr1 rather than gcc's. > > > > ebuild: > > > > ... > > inherit ... toolchain-funcs versionator ... > > ... > > DEPEND=... boost? ( dev-libs/boost ) ... > > ... > > pkg_env_check() { > > use boost && return 0 > > version_is_at_least "4.1" $(gcc-version) && return 0 > > echo "Either USE boost, or switch to gcc later than 4.1" > > } > > > > > > (with a default definition, "pkg_env_check() { return 0; }" ) > > In an ideal system you'd want this stuff in the metadata cache so that > the resolver can deal with it up front. You're talking about the metadata on the host, rather than the stuff on the rsync servers? I'm not sure you could cache the results even on the host - you would need to know what could affect the results so as to know when the cached information is out of date and has to be recalculated. That would either have to checked on every emerge, or made a separate switch (i.e. rely on the user to tell emerge when the environment has changed). > All resolution is a brute > force metadata search; and assuming we had all the necessary data up > front, we can optimize the search there (see pkgcore and restriction > subsystem) versus IMHO doing a 'dumb' search and then running through > a list of criteria for inclusion. > > This is the same reason why built_with_use in pkg_setup is really just > use_deps; these metadata should be included during resolution, not > after. My concern about dynamic dependencies runs to use deps, as well :) One could consider use-deps to be a special case of Marius' active checks. "how pkg P was built" isn't so different from "slot S of P is active" in terms of dep-graph creation; both are asking about the state of host & target systems, rather than the tree. In the case of USE deps, things are saner because the data doesn't change without the package manager knowing about it. Effectively the depgraph becomes static w.r.t. the tree + installation record (rather than just static w.r.t. the tree). With active checks implemented in the depgraph, however, that is no longer the case - the depgraph can change independently of the tree and the installation record. > With that said, I'm not sure how easy it would be to rewrite that > code; and this is simpler in that it's just a few bash functions as > opposed to more resolver foo. There's a lot to be said for keeping things simple, of course :) It's easy enough to mess up things like dep-graphs in any case - introducing these sorts of dynamic dependencies can render it substantially more complex. Another way to look at it, is to consider how often this sort of thing comes up. My understanding is that it is relatively rare; across the 10,000+ packages in the tree only a handful use 'built_with_use' fex. That makes a strong case for having a simple solution in the near term, and re-visit if it becomes commonplace. -- Kevin F. Quinn signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
Stephen Bennett wrote: > On Tue, 30 Jan 2007 17:06:51 +0100 > Marius Mauch <[EMAIL PROTECTED]> wrote: > >> The idea is to add a special category (let's call it "active" for >> now) that has the following properties: >> - this category doesn't exist in portdir or vdb (= no ebuilds) >> - when portage ($pkgmanager) encounters a "active/foo" atom in a >> dependency string it executes some special code (e.g. >> "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if >> that atom is satisfied > > Given that in the general case the package manager can't change the > active provider and will have to bail with an appropriate message that > the user needs to change it themselves, the obvious solution to this is > the previously-discussed-somewhere-I-can't-remember ebuild function, > called on depgraph creation, to check that it will be able to compile > in the current system environment. It's considerably simpler and more > generally useful than subverting DEPEND to add weird special-case hacks > to it. No one said subverting DEPEND was necessarily required. This stuff is essentially another visibility filter. Think for example along the ACCEPT_RESTRICT lines, but less fugly. User has FEATURES="sandbox" ebuild has RESTRICT="sandbox" Ebuild is not visible because it is incompatable with current environment. The only problem here being some visibility filters are 'soft' (like sandbox and USE vars) and some filters are hard (dependencies). USE flags can often be flippy flopped to get a solution (as with sandbox, which can be selectively turned off). -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Tue, 30 Jan 2007 17:06:51 +0100 Marius Mauch <[EMAIL PROTECTED]> wrote: > The idea is to add a special category (let's call it "active" for > now) that has the following properties: > - this category doesn't exist in portdir or vdb (= no ebuilds) > - when portage ($pkgmanager) encounters a "active/foo" atom in a > dependency string it executes some special code (e.g. > "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if > that atom is satisfied Given that in the general case the package manager can't change the active provider and will have to bail with an appropriate message that the user needs to change it themselves, the obvious solution to this is the previously-discussed-somewhere-I-can't-remember ebuild function, called on depgraph creation, to check that it will be able to compile in the current system environment. It's considerably simpler and more generally useful than subverting DEPEND to add weird special-case hacks to it. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
> > > Hmm; one could get the same benefit by introducing a new interface > (e.g. pkg_env_check()) which is defined to return true if the > environment is ok, false otherwise (with some text to stdout, perhaps). > The package manager would then run this function, after building the > depgraph and finding the candidate packages to merge, for each > candidate package - if any package fails is env_check, none of the > packages get emerged. Note this is then completely independent of > depgraph creation. > > In the 'tr1' example, I'd imagine something like this: > > use.local.desc: boost: Use boost library for tr1 rather than gcc's. > > ebuild: > > ... > inherit ... toolchain-funcs versionator ... > ... > DEPEND=... boost? ( dev-libs/boost ) ... > ... > pkg_env_check() { > use boost && return 0 > version_is_at_least "4.1" $(gcc-version) && return 0 > echo "Either USE boost, or switch to gcc later than 4.1" > } > > > (with a default definition, "pkg_env_check() { return 0; }" ) In an ideal system you'd want this stuff in the metadata cache so that the resolver can deal with it up front. All resolution is a brute force metadata search; and assuming we had all the necessary data up front, we can optimize the search there (see pkgcore and restriction subsystem) versus IMHO doing a 'dumb' search and then running through a list of criteria for inclusion. This is the same reason why built_with_use in pkg_setup is really just use_deps; these metadata should be included during resolution, not after. With that said, I'm not sure how easy it would be to rewrite that code; and this is simpler in that it's just a few bash functions as opposed to more resolver foo. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kevin F. Quinn wrote: > On a general note, introducing dynamic dependencies into the depgraph > worries me, although I'm not sure I can articulate why. Can you articulate "metadata cache"? :) - -- Vlastimil Babka (Caster) Gentoo/Java -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFwJf8tbrAj05h3oQRAgmCAJ4gYozwZqYL8f2Qi3mLx+cE9G4shwCeNnhM +n+IOpU9vuEw7g7xkSLHEi0= =d+dS -END PGP SIGNATURE- -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Tue, 30 Jan 2007 17:06:51 +0100 Marius Mauch <[EMAIL PROTECTED]> wrote: > Sometimes a package has to depend on a specific version of a slotted > package being the "active" one to build correctly, like in the > current "tr1" discussion on -dev [1] or with packages that depend on > the running kernel. Just to note, since you mentioned both "build" and "running" in the same sentence, that there can be various types of dependency here. Some dependencies will be pure build-time dependencies (for example, app-emulation/qemu needs gcc-3 to build) whereas others will be run-time dependencies (availability of >gcc-4.1 tr1 library when executing the emerged package, inclusion of a particular kernel configuration switch on the target). That gives you active BDEPENDs and active RDEPENDs to consider. Often there will be elements of both - the gcc-4 tr1 dependency may require the compiler to be gcc-4.1, and the target to also contain the gcc-4.1 tr1 library for example. > Currently this isn't really possible, however I while ago I got an > idea how to solve this. Keep in mind this is just a rough idea and > I'm pretty sure some people can/will point out why it is a stupid > idea, but anyway: > > The idea is to add a special category (let's call it "active" for > now) that has the following properties: > - this category doesn't exist in portdir or vdb (= no ebuilds) > - when portage ($pkgmanager) encounters a "active/foo" atom in a > dependency string it executes some special code (e.g. > "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if > that atom is satisfied This could also be done as an entry point in the ebuild (e.g. pkg_active()) - assuming you can source an ebuild when calculating dependencies. Not sure if that would be better or worse. Either way, these scripts need to be able to deal with the build-time/run-time difference I mentioned above. This comes down to defining the syntax of calling pkg_active() (or the script, for that matter) - e.g. "pkg_active RDEPEND >4.1" Also it's unclear what you intend 'foo' to be - it could be the name of the package that is being checked for active-ness (e.g. 'gcc' - returning true if the required version is active) or it could be related to the package that needs various things active (e.g. 'qemu', returning true if gcc-3 is active), or it could be generic (e.g. 'gcc-provides-tr1', returning true if the currently active gcc provides the tr1 library). Next, what do you do if the atom is not satisfied? For normal depends, it would add the required package to the list of stuff to be emerged. That can't happen here, so I guess you'd handle it somewhat like a blocker or mask error. So the difference between this idea, and the current way to handle it (script a build-time check in pkg_setup() fex) is that you find out about the build problem earlier. This is nice, but I don't think it's a significant benefit. Worth bearing in mind that you will be able to get conflicting active dependencies - e.g. 'emerge world' trying to build both app-emulation/qemu (which requires gcc-3 active) and one of these tr1 packages (requiring >gcc-4.1 active). On a general note, introducing dynamic dependencies into the depgraph worries me, although I'm not sure I can articulate why. Currently, everything is static (you can parse the tree, and you get the exact same depgraph on all systems for a given set of USE flags, no matter what), and dynamic requirements must be dealt with at build time (pkg_setup()). Implementing dynamic dependencies means that the depgraph now depends on the host and target systems, as well as the tree. This is a big conceptual change, so I would urge caution. Hmm; one could get the same benefit by introducing a new interface (e.g. pkg_env_check()) which is defined to return true if the environment is ok, false otherwise (with some text to stdout, perhaps). The package manager would then run this function, after building the depgraph and finding the candidate packages to merge, for each candidate package - if any package fails is env_check, none of the packages get emerged. Note this is then completely independent of depgraph creation. In the 'tr1' example, I'd imagine something like this: use.local.desc: boost: Use boost library for tr1 rather than gcc's. ebuild: ... inherit ... toolchain-funcs versionator ... ... DEPEND=... boost? ( dev-libs/boost ) ... ... pkg_env_check() { use boost && return 0 version_is_at_least "4.1" $(gcc-version) && return 0 echo "Either USE boost, or switch to gcc later than 4.1" } (with a default definition, "pkg_env_check() { return 0; }" ) -- Kevin F. Quinn signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Tue, 30 Jan 2007 18:04:41 +0200 Petteri Räty <[EMAIL PROTECTED]> wrote: > Marius Mauch wrote: > > Sometimes a package has to depend on a specific version of a slotted > > package being the "active" one to build correctly, like in the current > > "tr1" discussion on -dev [1] or with packages that depend on the running > > kernel. > > > > Currently this isn't really possible, however I while ago I got an idea how > > to solve this. Keep in mind this is just a rough idea and I'm pretty sure > > some people can/will point out why it is a stupid idea, but anyway: > > > > The idea is to add a special category (let's call it "active" for now) that > > has the following properties: > > - this category doesn't exist in portdir or vdb (= no ebuilds) > > - when portage ($pkgmanager) encounters a "active/foo" atom in a dependency > > string it executes some special code (e.g. > > "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if that > > atom is satisfied > > > > (and yes, this kinda goes with multi-repo/multi-format support) > > > > Marius > > To work this would need code in the tools like gcc-config. Otherwise > people could easily break their systems without anything giving them a > notice. Not sure I can follow. If we had such a check script for gcc why would it need additional code in gcc-config? Marius -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Tue, 30 Jan 2007 13:01:47 -0500 Alec Warner <[EMAIL PROTECTED]> wrote: > Marius Mauch wrote: > > Sometimes a package has to depend on a specific version of a slotted > > package being the "active" one to build correctly, like in the current > > "tr1" discussion on -dev [1] or with packages that depend on the running > > kernel. > > > > Currently this isn't really possible, however I while ago I got an idea how > > to solve this. Keep in mind this is just a rough idea and I'm pretty sure > > some people can/will point out why it is a stupid idea, but anyway: > > > > The idea is to add a special category (let's call it "active" for now) that > > has the following properties: > > - this category doesn't exist in portdir or vdb (= no ebuilds) > > - when portage ($pkgmanager) encounters a "active/foo" atom in a dependency > > string it executes some special code (e.g. > > "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if that > > atom is satisfied > > > > (and yes, this kinda goes with multi-repo/multi-format support) > > > > Marius > > I don't see why how this is any less complicated than just adding more > functionality to || deps (runtime versus compile time) Please explain. If a package can only be built if the currently running kernel is higher than 2.6.19 for example, how would you possibly express that in *DEPEND syntax? Marius -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Tue, 30 Jan 2007 08:25:31 -0800 Brian Harring <[EMAIL PROTECTED]> wrote: > On Tue, Jan 30, 2007 at 05:06:51PM +0100, Marius Mauch wrote: > > Sometimes a package has to depend on a specific version of a > > slotted package being the "active" one to build correctly, like in > > the current "tr1" discussion on -dev [1] or with packages that > > depend on the running kernel. > > tr1 is partially addressed via addition of a 'binding' modifier for > rdeps, to state that ||() deps are locked down after compilation. And how would that solve the actual issue of expressing "I need /usr/bin/gcc to run gcc-4.1 and not gcc-3.4"? The lockdown of || deps is a completely separate issue, unless I'm missing something. > > The idea is to add a special category (let's call it "active" for > > now) that has the following properties: > > - this category doesn't exist in portdir or vdb (= no ebuilds) > > - when portage ($pkgmanager) encounters a "active/foo" atom in a > > dependency string it executes some special code (e.g. > > "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if > > that atom is satisfied > > Non deterministic resolution; previous steps in the graph can cause > that value to flip to a different setting by the time the 'dep' is > encountered. Ok, that's a problem, though for the use cases at hand (gcc and kernel) it would be mostly irrelevant. > That's ignoring the kick in the nads usage of this will due to > resolution... Neglectable IMO, it's not such a common use case anyway, and I don't think I have to compare it to the current "solution" (die in setup or compile). > > (and yes, this kinda goes with multi-repo/multi-format support) > > Don't really see how this enables multiple standalone repos in any > sane way, so that one requires justification... Where did I say anything about enabling? It would need more or less a separate repository (dbapi) instance, so it would require such support. Marius -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
Marius Mauch wrote: > Sometimes a package has to depend on a specific version of a slotted package > being the "active" one to build correctly, like in the current "tr1" > discussion on -dev [1] or with packages that depend on the running kernel. > > Currently this isn't really possible, however I while ago I got an idea how > to solve this. Keep in mind this is just a rough idea and I'm pretty sure > some people can/will point out why it is a stupid idea, but anyway: > > The idea is to add a special category (let's call it "active" for now) that > has the following properties: > - this category doesn't exist in portdir or vdb (= no ebuilds) > - when portage ($pkgmanager) encounters a "active/foo" atom in a dependency > string it executes some special code (e.g. "$PORTDIR/scripts/active-check/foo > =active/foo-1") to determine if that atom is satisfied > > (and yes, this kinda goes with multi-repo/multi-format support) > > Marius I don't see why how this is any less complicated than just adding more functionality to || deps (runtime versus compile time) -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
On Tue, Jan 30, 2007 at 05:06:51PM +0100, Marius Mauch wrote: > Sometimes a package has to depend on a specific version of a > slotted package being the "active" one to build correctly, like in > the current "tr1" discussion on -dev [1] or with packages that > depend on the running kernel. tr1 is partially addressed via addition of a 'binding' modifier for rdeps, to state that ||() deps are locked down after compilation. Doesn't gurantee the user doesn't pop back to 3.4 after compilation, but that's their own mess. > The idea is to add a special category (let's call it "active" for > now) that has the following properties: > - this category doesn't exist in portdir or vdb (= no ebuilds) > - when portage ($pkgmanager) encounters a "active/foo" atom in a > dependency string it executes some special code (e.g. > "$PORTDIR/scripts/active-check/foo =active/foo-1") to determine if > that atom is satisfied Non deterministic resolution; previous steps in the graph can cause that value to flip to a different setting by the time the 'dep' is encountered. That's ignoring the kick in the nads usage of this will due to resolution... > (and yes, this kinda goes with multi-repo/multi-format support) Don't really see how this enables multiple standalone repos in any sane way, so that one requires justification... ~harring pgpetWeCOF0tF.pgp Description: PGP signature
Re: [gentoo-portage-dev] [RFC] Depending on "active" version
Marius Mauch wrote: > Sometimes a package has to depend on a specific version of a slotted package > being the "active" one to build correctly, like in the current "tr1" > discussion on -dev [1] or with packages that depend on the running kernel. > > Currently this isn't really possible, however I while ago I got an idea how > to solve this. Keep in mind this is just a rough idea and I'm pretty sure > some people can/will point out why it is a stupid idea, but anyway: > > The idea is to add a special category (let's call it "active" for now) that > has the following properties: > - this category doesn't exist in portdir or vdb (= no ebuilds) > - when portage ($pkgmanager) encounters a "active/foo" atom in a dependency > string it executes some special code (e.g. "$PORTDIR/scripts/active-check/foo > =active/foo-1") to determine if that atom is satisfied > > (and yes, this kinda goes with multi-repo/multi-format support) > > Marius To work this would need code in the tools like gcc-config. Otherwise people could easily break their systems without anything giving them a notice. Regards, Petteri signature.asc Description: OpenPGP digital signature