[gentoo-dev] Re: sub-slots (for EAPI 5)
On 07-09-2012 13:51:24 -0400, Ian Stakenvicius wrote:
> On 07/09/12 01:13 PM, Fabian Groffen wrote:
> > On 06-09-2012 09:25:53 -0400, Ian Stakenvicius wrote:
> >> #1 - there is both a specification, and an initial
> >> implementation, AND a fork of the tree that is kept
> >> semi-up-to-date on my dev overlay.
> >
> > I was interested in a (formal) specification, not a proof of
> > concept.
> >
>
> Ahh.. sorry, i figured the modified slot-operator spec that Ciaran
> and Zac did was considered formal. Are you looking for a GLEP, then?
> or...
No, not a GLEP, per se. I'm trying to understand what sub-slot does and
is. I think I'm starting to understand now. However, for this feature
to be added to an EAPI, IMO it would be nice if there are resources that
make it for most developers very clear how this feature should be used
(and how not), and what kind of problems it can solve.
I guess real-life examples, more extensively described than you did
before, with exactly where it goes wrong, and how the situation is
improved would help.
> if you s/slot-deps/slot-operators/ , then yes i'm aware. to me,
> "slot-deps" is something we got in (iirc) EAPI=2.
The wiki calls it "Slot operator dependencies", which I abbreviate dto
slot-deps. Sorry for the confusion.
> >> [1] No advantage as sub-slots wouldn't relate to this, UNLESS
> >> the masking would then remove -all- SLOT="0/5" versions from the
> >> tree. In that case, the old SLOT="0/3" provider would be the
> >> 'upgrade' path and so 'foo' and 'bar' would be triggered for
> >> rebuild (since the lib they were linked to would be disappearing
> >> during emerge -uDN)
> >
> > So your example SLOTs are wrong, and should have included the
> > minor (always!?!) since downgrading a lib that goes back to an
> > older minor means functions no longer exist, thus a consumer can
> > potentially break.
>
> If those (missing) functions are necessary then the either the full
> slot, or the particular minimum package version, of libfnord would
> need to be specified in the consumer. This isn't any different from
> how things work now.
Eh, no. Now it just always breaks when you perform a downgrade, and
revdev-rebuild or @preserved-libs won't help you. I prefer that you
give best practices how to use sub-slots to make Portage also able to do
a recompile of bar when libfnord in the same SLOT gets downgraded.
(Because minors are used for compatible changes -- additions -- to the
ABI.)
(And the recompile is preferably done against the headers of the
downgraded version, but with the newer version's lib still around, such
that if this is a vital binary such as Python, it will not break down --
however, this is most likely too much to ask.)
> This is why, for the rebuild of bar-2.4 to be triggered on upgrade to
> libfnord-3.5 the SLOT= var within libfnord-3.5.ebuild would need to
> have something other than SLOT="0/5", ie, SLOT="0/5.12"
Yeah, but can I also avoid bar-2.4 being recompiled when I install
libfnord-3.5? It's not necessary, because the 5-ABI of libfnord is
supposed to be backwards compatible. (At least that's the idea.)
Like mentioned before, I DO want bar-2.4 to be recompiled if I have to
downgrade libfnord to a version before the one I had installed when I
compiled bar-2.4, though.
> > Do you allow sub-slot to depend on e.g. USE-flags in use? Suppose
> > libfnord has a USE-flag cow that adds special cow interfaces to
> > the ABI/API. Would SLOT="X/${PV}$(use cow && echo -- -cow)" work?
> > (I think SLOT is supposed to be metadata-static, but does the
> > sub-slot part count to that?)
>
> No, afaik slots (including sub-slots) can't be dynamic. Plus we
> already have comprehensive use deps solutions so why would we need that?
Because the ABI changes. But I guess you're right that we can safely
ignore packages that screw it up badly enough here.
Thanks
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: Digital signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On Fri, 7 Sep 2012 20:17:17 +0200 Fabian Groffen wrote: > Eh, no. Now it just always breaks when you perform a downgrade, and > revdev-rebuild or @preserved-libs won't help you. I prefer that you > give best practices how to use sub-slots to make Portage also able to > do a recompile of bar when libfnord in the same SLOT gets downgraded. > (Because minors are used for compatible changes -- additions -- to the > ABI.) Downgrades aren't covered by sub-slots, slots, regular dependencies, libtool, or anything else. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 07/09/12 02:17 PM, Fabian Groffen wrote:
>
> No, not a GLEP, per se. I'm trying to understand what sub-slot
> does and is. I think I'm starting to understand now. However, for
> this feature to be added to an EAPI, IMO it would be nice if there
> are resources that make it for most developers very clear how this
> feature should be used (and how not), and what kind of problems it
> can solve.
>
> I guess real-life examples, more extensively described than you
> did before, with exactly where it goes wrong, and how the situation
> is improved would help.
>
I agree; I expect devmanual.gentoo.org would need a nice big page (or
section in the SLOTs page) describing it, if not at the very least a
nice entry on wiki.g.o
[1] No advantage as sub-slots wouldn't relate to this,
UNLESS the masking would then remove -all- SLOT="0/5"
versions from the tree. In that case, the old SLOT="0/3"
provider would be the 'upgrade' path and so 'foo' and 'bar'
would be triggered for rebuild (since the lib they were
linked to would be disappearing during emerge -uDN)
>>>
>>> So your example SLOTs are wrong, and should have included the
>>> minor (always!?!) since downgrading a lib that goes back to an
>>> older minor means functions no longer exist, thus a consumer
>>> can potentially break.
>>
>> If those (missing) functions are necessary then the either the
>> full slot, or the particular minimum package version, of libfnord
>> would need to be specified in the consumer. This isn't any
>> different from how things work now.
>
> Eh, no. Now it just always breaks when you perform a downgrade,
> and revdev-rebuild or @preserved-libs won't help you. I prefer
> that you give best practices how to use sub-slots to make Portage
> also able to do a recompile of bar when libfnord in the same SLOT
> gets downgraded. (Because minors are used for compatible changes --
> additions -- to the ABI.) (And the recompile is preferably done
> against the headers of the downgraded version, but with the newer
> version's lib still around, such that if this is a vital binary
> such as Python, it will not break down -- however, this is most
> likely too much to ask.)
>
I guess maybe i'm not following your example. To spell it out better,
here's what I'm understanding:
bar-1.0 has (prior to slot-operators) an RDEPEND="app-cat/libfnord".
No version specified. As such, it'll build successfully against
either libfnord-1 or libfnord-2. Right now (as i understand it) a
downgrade from libfnord-2 to libfnord-1 would "break" bar-1.0 bot
revdep-rebuild would detect and fix it.
sub-slots + slot-operators would alleviate what I just described,
auto-rebuilding bar (at least that's the theory; i've had some issues
getting portage to downgrade things even with regular slots so some
work on the implementation might be needed with this, dunno)
*IF* on the other hand, you're referring to the case of libfnord-2.1
downgrading to libfnord-2 , then yes I can see how bar-1.0 would be
broken and revdep-rebuild wouldn't fix it. However, as far as I
understand it, proper LTVERSIONing should mean that bar wouldn't break
as anything which would break bar on a libfnord-2 downgrade shouldn't
be used by bar in libfnord-2.1 -- ie, the differences would be
internal to libfnord and not directly used by bar.
If a package is -not- properly LTVERSIONed though, sub-slot bumps
could help alleviate this issue at the ebuild level.
>> This is why, for the rebuild of bar-2.4 to be triggered on
>> upgrade to libfnord-3.5 the SLOT= var within libfnord-3.5.ebuild
>> would need to have something other than SLOT="0/5", ie,
>> SLOT="0/5.12"
>
> Yeah, but can I also avoid bar-2.4 being recompiled when I install
> libfnord-3.5? It's not necessary, because the 5-ABI of libfnord
> is supposed to be backwards compatible. (At least that's the
> idea.) Like mentioned before, I DO want bar-2.4 to be recompiled if
> I have to downgrade libfnord to a version before the one I had
> installed when I compiled bar-2.4, though.
>
Sure -- since bar-2.4 does support a codepath for >> Do you allow sub-slot to depend on e.g. USE-flags in use?
>>> Suppose libfnord has a USE-flag cow that adds special cow
>>> interfaces to the ABI/API. Would SLOT="X/${PV}$(use cow &&
>>> echo -- -cow)" work? (I think SLOT is supposed to be
>>> metadata-static, but does the sub-slot part count to that?)
>>
>> No, afaik slots (including sub-slots) can't be dynamic. Plus we
>> already have comprehensive use deps solutions so why would we
>> need that?
>
> Because the ABI changes. But I guess you're right that we can
> safely ignore packages that screw it up badly enough here.
I think that the best way to resolve something like this would be more
comprehensive use dep usage -- ie, if bar uses the bits of libfnord
available through USE="cow" then bar is effectively dependant on those
bits and so a libfnord:=[cow?] dep (a
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On 07-09-2012 19:21:57 +0100, Ciaran McCreesh wrote: > On Fri, 7 Sep 2012 20:17:17 +0200 > Fabian Groffen wrote: > > Eh, no. Now it just always breaks when you perform a downgrade, and > > revdev-rebuild or @preserved-libs won't help you. I prefer that you > > give best practices how to use sub-slots to make Portage also able to > > do a recompile of bar when libfnord in the same SLOT gets downgraded. > > (Because minors are used for compatible changes -- additions -- to the > > ABI.) > > Downgrades aren't covered by sub-slots, slots, regular dependencies, > libtool, or anything else. It seems I mistakenly took slot-operator-deps and sub-slots as something that can be mapped onto ABIs. Doing so, however has proven to be wrong. It appears slot-operator-deps do have some resemblance with ABI here (especially if :* would be written in PMS such that it only allows upgrades, no downgrades), but sub-slots are completely unrelated. I don't like the mixing of the two in a single var, at all. I think I'd much more prefer Portage to understand ABIs and potentially versions, to make it explicit why it is doing what. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On Fri, 7 Sep 2012 20:49:35 +0200 Fabian Groffen wrote: > On 07-09-2012 19:21:57 +0100, Ciaran McCreesh wrote: > > On Fri, 7 Sep 2012 20:17:17 +0200 > > Fabian Groffen wrote: > > > Eh, no. Now it just always breaks when you perform a downgrade, > > > and revdev-rebuild or @preserved-libs won't help you. I prefer > > > that you give best practices how to use sub-slots to make Portage > > > also able to do a recompile of bar when libfnord in the same SLOT > > > gets downgraded. (Because minors are used for compatible changes > > > -- additions -- to the ABI.) > > > > Downgrades aren't covered by sub-slots, slots, regular dependencies, > > libtool, or anything else. > > It seems I mistakenly took slot-operator-deps and sub-slots as > something that can be mapped onto ABIs. Doing so, however has proven > to be wrong. It's not entirely wrong. There's a reason we stopped using the word "ABI", though: it's a meaningless term with a lot of misleading connotations. > It appears slot-operator-deps do have some resemblance with ABI here > (especially if :* would be written in PMS such that it only allows > upgrades, no downgrades), but sub-slots are completely unrelated. Downgrades are a different, unrelated problem. If you're trying to solve that, you'll need a different, orthogonal solution. Note, though, that downgrade breakages are typically not covered by whatever you think an ABI is. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On 07-09-2012 14:39:38 -0400, Ian Stakenvicius wrote: > I guess maybe i'm not following your example. To spell it out better, > here's what I'm understanding: > > bar-1.0 has (prior to slot-operators) an RDEPEND="app-cat/libfnord". > No version specified. As such, it'll build successfully against > either libfnord-1 or libfnord-2. Right now (as i understand it) a > downgrade from libfnord-2 to libfnord-1 would "break" bar-1.0 bot > revdep-rebuild would detect and fix it. right or portage would preserve libfnord-2's libs > sub-slots + slot-operators would alleviate what I just described, > auto-rebuilding bar (at least that's the theory; i've had some issues > getting portage to downgrade things even with regular slots so some > work on the implementation might be needed with this, dunno) > > *IF* on the other hand, you're referring to the case of libfnord-2.1 > downgrading to libfnord-2 , then yes I can see how bar-1.0 would be > broken and revdep-rebuild wouldn't fix it. However, as far as I > understand it, proper LTVERSIONing should mean that bar wouldn't break > as anything which would break bar on a libfnord-2 downgrade shouldn't > be used by bar in libfnord-2.1 -- ie, the differences would be > internal to libfnord and not directly used by bar. Well, yes and no. The no applies to the case where bar works around a missing function by e.g. implementing it itself, conditionally. A package maintainer might not have been aware of this, hence not expressed this in the dependencies (e.g. requiring the version that contains the missing function). Anyway, we should avoid downgrades of libraries, so no need to discuss this any further. It doesn't break more than it does already. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On 09/07/2012 11:17 AM, Fabian Groffen wrote: > No, not a GLEP, per se. I'm trying to understand what sub-slot does and > is. I think I'm starting to understand now. However, for this feature > to be added to an EAPI, IMO it would be nice if there are resources that > make it for most developers very clear how this feature should be used > (and how not), and what kind of problems it can solve. > > I guess real-life examples, more extensively described than you did > before, with exactly where it goes wrong, and how the situation is > improved would help. Perhaps some of the greatest frustrations for Gentoo users stem from the lack of support for automatic rebuild of packages when necessary. Imagine how nice it would be if necessary rebuilds would automatically occur when appropriate, so that you wouldn't experience build failures that require you to manually intervene by running revdep-rebuild, perl-cleaner, or something like that. And there are other kinds of necessary rebuilds that don't trigger build failures, but lead to runtime failures that are noticed much later (like xorg driver failures after a major xorg-server update). Sub-slots can be used to solve the bulk of problems like these that our users have had to deal with manually. > Eh, no. Now it just always breaks when you perform a downgrade, and > revdev-rebuild or @preserved-libs won't help you. I prefer that you > give best practices how to use sub-slots to make Portage also able to do > a recompile of bar when libfnord in the same SLOT gets downgraded. > (Because minors are used for compatible changes -- additions -- to the > ABI.) > (And the recompile is preferably done against the headers of the > downgraded version, but with the newer version's lib still around, such > that if this is a vital binary such as Python, it will not break down -- > however, this is most likely too much to ask.) It might be worthwhile to try come up with some way to handle minor downgrades in some later EAPI, but it adds complexity. Meanwhile, sub-slots are a relatively simple extension to slot-operator deps, and they are poised to greatly improve user experience (via automatic rebuilds) if they are included in EAPI 5. -- Thanks, Zac
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On 07-09-2012 19:55:53 +0100, Ciaran McCreesh wrote: > > It appears slot-operator-deps do have some resemblance with ABI here > > (especially if :* would be written in PMS such that it only allows > > upgrades, no downgrades), but sub-slots are completely unrelated. > > Downgrades are a different, unrelated problem. If you're trying to > solve that, you'll need a different, orthogonal solution. Note, though, > that downgrade breakages are typically not covered by whatever you think > an ABI is. I don't really want to solve downgrades. I just want to fully understand what slot-deps is supposed to do, and what not. It seems I fail in doing so. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On 07-09-2012 12:03:16 -0700, Zac Medico wrote: > On 09/07/2012 11:17 AM, Fabian Groffen wrote: > > I guess real-life examples, more extensively described than you did > > before, with exactly where it goes wrong, and how the situation is > > improved would help. > > Perhaps some of the greatest frustrations for Gentoo users stem from the > lack of support for automatic rebuild of packages when necessary. > Imagine how nice it would be if necessary rebuilds would automatically > occur when appropriate, so that you wouldn't experience build failures > that require you to manually intervene by running revdep-rebuild, > perl-cleaner, or something like that. And there are other kinds of > necessary rebuilds that don't trigger build failures, but lead to > runtime failures that are noticed much later (like xorg driver failures > after a major xorg-server update). Sub-slots can be used to solve the > bulk of problems like these that our users have had to deal with manually. I like that! Kudos for making it work! I just wonder what the heck that has to do with SLOT. This discussion has been done before in this thread, and it somehow settled. > ... sub-slots are a relatively simple extension to slot-operator deps, > and they are poised to greatly improve user experience (via automatic > rebuilds) if they are included in EAPI 5. And we want it. But is it a good idea to add some feature that feels like just a hack? -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On Fri, 7 Sep 2012 21:25:22 +0200 Fabian Groffen wrote: > I like that! Kudos for making it work! > > I just wonder what the heck that has to do with SLOT. The correct fix for "not needing to rebuild" stuff is to SLOT libraries like crazy, and have a SLOT per thing-we-don't-call-ABI. This then needs := dependencies, so that packages can say "and remember which SLOT I was built against". However, there are some packages that cannot easily be SLOTted to the degree that we'd like. This is where sub-SLOTs come in. Given sub-SLOTted packages dep:1/a and dep:1/b, this says "I'd like to have slots 1a and 1b, but it's too difficult to allow 1a and 1b to be installed at the same time". So suppose the user has pkg with a dependency upon dep, with slot 1 and a := operator. They install pkg when dep:1/a is installed. The user then installs dep:1/b. In an ideal world, dep:1/a would remain installed in parallel with dep:1/b, but your friendly Gentoo developers have decided it's not worth their time to allow this. Thus, dep:1/a has to be uninstalled to allow dep:1/b to be installed. But this would break pkg, since pkg needs dep:1/a. However, a clever dependency resolver can note that reinstalling pkg would fix it, since dep:1/b also satisfies pkg's slot 1 := dependency (but not the locked 1/a dependency that the installed version of pkg has picked up). -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
On 07-09-2012 20:36:02 +0100, Ciaran McCreesh wrote: > The correct fix for "not needing to rebuild" stuff is to SLOT libraries > like crazy, and have a SLOT per thing-we-don't-call-ABI. This then > needs := dependencies, so that packages can say "and remember which > SLOT I was built against". > > However, there are some packages that cannot easily be SLOTted to the > degree that we'd like. This is where sub-SLOTs come in. Given > sub-SLOTted packages dep:1/a and dep:1/b, this says "I'd like to have > slots 1a and 1b, but it's too difficult to allow 1a and 1b to be > installed at the same time". > > So suppose the user has pkg with a dependency upon dep, with slot 1 and > a := operator. They install pkg when dep:1/a is installed. The user > then installs dep:1/b. In an ideal world, dep:1/a would remain > installed in parallel with dep:1/b, but your friendly Gentoo developers > have decided it's not worth their time to allow this. Thus, dep:1/a has > to be uninstalled to allow dep:1/b to be installed. But this would > break pkg, since pkg needs dep:1/a. However, a clever dependency > resolver can note that reinstalling pkg would fix it, since dep:1/b > also satisfies pkg's slot 1 := dependency (but not the locked 1/a > dependency that the installed version of pkg has picked up). Thanks. It seems we're there. At last. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-dev] Re: sub-slots (for EAPI 5)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 07/09/12 03:25 PM, Fabian Groffen wrote:
> On 07-09-2012 12:03:16 -0700, Zac Medico wrote:
>> On 09/07/2012 11:17 AM, Fabian Groffen wrote:
>>> I guess real-life examples, more extensively described than you
>>> did before, with exactly where it goes wrong, and how the
>>> situation is improved would help.
>>
>> Perhaps some of the greatest frustrations for Gentoo users stem
>> from the lack of support for automatic rebuild of packages when
>> necessary. Imagine how nice it would be if necessary rebuilds
>> would automatically occur when appropriate, so that you wouldn't
>> experience build failures that require you to manually intervene
>> by running revdep-rebuild, perl-cleaner, or something like that.
>> And there are other kinds of necessary rebuilds that don't
>> trigger build failures, but lead to runtime failures that are
>> noticed much later (like xorg driver failures after a major
>> xorg-server update). Sub-slots can be used to solve the bulk of
>> problems like these that our users have had to deal with
>> manually.
>
> I like that! Kudos for making it work!
>
> I just wonder what the heck that has to do with SLOT. This
> discussion has been done before in this thread, and it somehow
> settled.
>
>> ... sub-slots are a relatively simple extension to slot-operator
>> deps, and they are poised to greatly improve user experience (via
>> automatic rebuilds) if they are included in EAPI 5.
>
> And we want it. But is it a good idea to add some feature that
> feels like just a hack?
>
>
Originally the sub-slot idea came about because one of the ways
"around" all of this broken-and-requiring-afterthefact-rebuilding was
to just make everything slotted -- so there would always be multiple
slots of everything installed -- and use slot-operators to indicate
when things should be re-emerged
Although this would work, the end result would (imo at least) be
horrible on-disk.
Sub-slots allow the main part of SLOT to still specify what's
installed on disk, while allowing PMS to identify and trigger rebuilds
for SLOT changes based on slot-operators.
I see it akin to the '-r' portion of ${PV} -- Used by portage to
trigger updates but having very little meaning to the actual version
of the package that gets installed. (ok i might be stretching it with
this)
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
iF4EAREIAAYFAlBKUK4ACgkQ2ugaI38ACPDbCAEAiG+7hQch043se8ZfDE4qC52w
79ZImWn5jazqGQDN3zsA/3B1AJR+SWxUFDHZF1LArX0r0Gd7J2madTqP0m+llxuG
=7IEF
-END PGP SIGNATURE-
