Re: [gentoo-dev] don't rely on dynamic deps

[Jan Matejka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Wed, 23 Jul 2014 00:05:32 +0200
Tom Wijsman  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On Mon, 21 Jul 2014 16:41:39 -0400
> Ian Stakenvicius  wrote:
> 
> > I wonder if there may be some form of extension we could add to
> > portage, such that it could do a VDB-only "re-emerge" somehow, when
> > the in-tree ebuild doesn't match the in-VDB one.  If that could be
> > implemented properly (and i'm not sure that it could, tbh), maybe
> > that would help reduce issues with dynamic deps, too...
> 
> Hmm, dynamic dependencies ... dynamic rebuilds ... dynamic
> compilation; one day we will have hot code pushes where upstream
> changes immediately reflects itself upon one's system. Does such a
> gimmick succeed? Meteor!

https://en.wikipedia.org/wiki/KGraft

- --
Jan Matějka| Developer
https://gentoo.org | Gentoo Linux
GPG: A33E F5BC A9F6 DAFD 2021  6FB6 3EBF D45B EEB6 CA8B
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBCgAGBQJT/gT1AAoJEIN+7RD5ejahISgIAKbVgqLmuJADmJ37TddtmPPP
7WPq5UYdGT4wq9rElrU3GCu10u+pxQqOOKzVjnXiSlcCF1xqkG07caaYO/Xw/ccS
VclMPDngRU7YUSl9F6UeDkDK9l+48qE/1/ohrSL9f8giA5H5SZ3G3fanWN/oFp6W
FsGx7J6ddRmM/LaSHrso4ngbMU7XIMFZgUeR5X3t5bWef/Si9adxJzfbMoCQFUDN
7qXT8EkY9U9Po5/aAqtsriyGWZttuIvjGbmiNdrdJI/IT/Rshlqzqy/viqcK2aOX
bCO1dn5wRLxfV7JBgmg14UJqcDK6HgkSkl4rrl574M4arq7kZtbMEKbSvDcpurw=
=Wphr
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Mon, Jul 28, 2014 at 12:29 PM, Ian Stakenvicius  wrote:
> As has been mentioned or alluded to before, this is fine as long as
> end-users --sync when the dependency change is still in the tree.
> However, if that doesn't happen then we still end up with the issue.
>
> Of course, if that is the case, then #2 shouldn't happen either
> (because the end-user system wouldn't see B as having been removed and
> therefore --depclean won't remove it).

Agree.  Things stay consistent either way if everything is done properly.

Bottom line is that if you keep PVs installed which aren't in portage,
you're on your own.  They could contain known bugs that we fixed in a
revbump that you missed, or they could contain security issues that we
don't bother to check for since we don't have the PV in our
repository, etc.  If you keep such a PV installed then you're the
maintainer, so good luck!

If a more recent version is in portage, then your old ebuild will
probably uninstall cleanly (or at least as cleanly as it would with
static deps), and the upgrade will hopefully be in better shape.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 27/07/14 05:08 PM, Rich Freeman wrote:
> On Sun, Jul 27, 2014 at 4:24 PM, Michał Górny 
> wrote:
>> Dnia 2014-07-27, o godz. 10:42:19
>> 
>> Consider the following:
>> 
>> 1. A depends on B, both are installed,
>> 
>> 2. dependency on B is removed, emerge --depclean uninstalls B
>> thanks to dynamic-deps,
>> 
>> 3. B is treecleaned (nothing depends on it),
>> 
>> 4. old version of A is removed (user doesn't update it yet),
>> therefore dependency on B is restored from vdb.
>> 
>> So, now user has package A installed which has unsatisfied
>> dependency on non-available package.
> 
> I'd think that portage should update vdb as soon as it detects the 
> dependency change.  Then B would no longer depend on A in vdb.  It 
> shouldn't hold onto outdated information.  Basically a dependency 
> change should trigger a no-rebuild merge if it is safe to do so,
> and if not there should be a revbump anyway.

As has been mentioned or alluded to before, this is fine as long as
end-users --sync when the dependency change is still in the tree.
However, if that doesn't happen then we still end up with the issue.

Of course, if that is the case, then #2 shouldn't happen either
(because the end-user system wouldn't see B as having been removed and
therefore --depclean won't remove it).


...why do i feel like i'm getting the same headache i had in my 2nd
year databases course, when i was trying to wrap my head around ACID
compliance and transaction visibility
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlPWelAACgkQ2ugaI38ACPCokAEAvDNcn+kJ6WTpL+hMAjexRuJX
mbHoj9pGsuFQ2kqoL7YA/1n9mZ2zDpVBurXLflU2KpqNgGx3E/ujozBOveHzoII+
=0Zgq
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Arfrever Frehtes Taifersar Arahesis]

2014-07-27 23:33 Ciaran McCreesh napisał(a):
> On Sun, 27 Jul 2014 17:26:27 -0400
> Rich Freeman  wrote:
> > But, in that case you can put the necessary ebuilds into your overlay
> > and then portage can make everything right.
> 
> Oh? Please explain to us a) how the overlay interaction *actually* works
> with dynamic dependencies currently

Portage uses dynamic dependencies from ebuild in original repository of given 
installed ebuild.
Name of repository is stored in /var/db/pkg/${CATEGORY}/${PF}/repository file.

> and b) how it can work both in the case you describe

A user would have to do:
# echo ${new_repository} > /var/db/pkg/${CATEGORY}/${PF}/repository
# touch /var/db/pkg/${CATEGORY}/${PF}
# touch /var/db/pkg/${CATEGORY}
# touch /var/db/pkg

(It is possible that updating of timestamps is not yet strictly required...)

--
Arfrever Frehtes Taifersar Arahesis


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 5:50 PM, Kent Fredric  wrote:
>
> On 28 July 2014 09:34, Rich Freeman  wrote:
>>
>> and if it doesn't work for them,
>> they'll sync in the updates one way or another (using an overlay if
>> necessary).
>
>
> However, in the case the package gets removed from tree, an updates based
> approach would allow the dependencies to be cleaned up long after the
> package itself is gone.

Maybe, but is it really our goal to fix broken packages that aren't
even maintained any longer?  The latest version of the package will
always be in cvs/etc and users can always go fetch it, but do we need
a special updates mechanism simply for the purpose of fixing packages
that we've already decided are unsustainable?

If an updates-like approach is the best approach for active packages,
then I'd consider the side-benefit to treecleaned ones as being
beneficial.  However, I wouldn't really view this as a primary
concern.  At least, that is my sense of it right now.  The primary
focus needs to be on making dynamic deps work in a sensible way for
active packages, which we're apparently having problems with already.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Kent Fredric]

On 28 July 2014 09:34, Rich Freeman  wrote:

> and if it doesn't work for them,
> they'll sync in the updates one way or another (using an overlay if
> necessary).
>

However, in the case the package gets removed from tree, an updates based
approach would allow the dependencies to be cleaned up long after the
package itself is gone.

And this proves useful, because one of the questions people tend to ask (
or should ) when experiencing problems is "did you sync lately". If they
did sync lately, our best attempts at fixing their bug is employed.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 5:33 PM, Ciaran McCreesh
 wrote:
> On Sun, 27 Jul 2014 17:26:27 -0400
> Rich Freeman  wrote:
>> But, in that case you can put the necessary ebuilds into your overlay
>> and then portage can make everything right.
>
> Oh? Please explain to us a) how the overlay interaction *actually* works
> with dynamic dependencies currently,

No idea.  I doubt it is specified behavior.  Certainly we should learn
whatever lessons we can from what has been done already, but we aren't
constrained by it.

> and b) how it can work both in the
> case you describe, and in the case where an overlay has a substantially
> different ebuild for the same package.

I'd think that a change in repository should probably be treated like
a revbump.  There is no way to know that foo-1-r1 in one overlay is
the same as foo-1-r1 in another.  The package manager has to figure
out which overlay to use already, and if the same PV shows up in a
higher-priority overlay then it is treated as a revbump of whatever is
in the lower-priority overlay and it triggers a rebuild.  Maybe you
could get clever about checking for identical ebuilds/eclasses/etc and
avoid a rebuild if it literally is the same file, but I wouldn't go
further than that.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 5:27 PM, Kent Fredric  wrote:
>
> On 28 July 2014 08:56, Ciaran McCreesh 
> wrote:
>>
>> > To me it seems like a simple data model bug that vdb does not get
>> > updated to reflect the new situation after step 2 above.
>>
>> Rewriting VDB won't help if the user doesn't sync at the right time.
>
> Indeed. pkgmove has this problem solved with a mass of quarterly move files,
> but I'm not sure I'd want to have

Does it matter?  If a user never syncs again, they'll have a broken
package (though portage won't realize it).  If they wait until the
package is removed from the tree before syncing then they'll still
have a broken package.  If the broken package happens to work for
them, then the user won't care, and if it doesn't work for them,
they'll sync in the updates one way or another (using an overlay if
necessary).

As far as I can tell, the only thing that is needed for this to work
is for portage to be able to detect when an installed package
dependency changes in the repository, and this could be facilitated by
metadata.  Maintainers would then revbump when a change can't be
correctly handled, and they won't revbump when it can be.

Do we have to guarantee that users who don't sync before a PV is
removed get all updates to that PV if they have it installed when they
eventually do sync?  If I have some package installed that was
treecleaned a year ago and it depends on udev then obviously it won't
know anything about the new virtual - we live with that already
without too many problems.  Bottom line is that bad things happen if
you hang onto packages that aren't maintained in some repository
(possibly your overlay maintained by yourself).

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Sun, 27 Jul 2014 17:26:27 -0400
Rich Freeman  wrote:
> But, in that case you can put the necessary ebuilds into your overlay
> and then portage can make everything right.

Oh? Please explain to us a) how the overlay interaction *actually* works
with dynamic dependencies currently, and b) how it can work both in the
case you describe, and in the case where an overlay has a substantially
different ebuild for the same package.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Kent Fredric]

On 28 July 2014 08:56, Ciaran McCreesh 
wrote:

> > To me it seems like a simple data model bug that vdb does not get
> > updated to reflect the new situation after step 2 above.
>
> Rewriting VDB won't help if the user doesn't sync at the right time.
>

Indeed. pkgmove has this problem solved with a mass of quarterly move
files, but I'm not sure I'd want to have

a)  However many `depchange` entries required to make it happen linger on
for all eternity in some cruft file just in case people don't sync more
than once every 2 years:  ( Yes, we still have an updates/1Q-2009 file for
people stuck in a time warp who need change updates )

b) The burden of maintainers having to manually update that index. ( That's
effectively what the -r1.1 and INSTALL_FROM proposals amount to )

The only saving grace here if we applied this strategy, is we could
conceivably generate the index in an automated fashion due to ebuild edits
usually being more obvious to an SCM than a move is.  ( And you could
conceivably generate them by simply comparing snapshot diffs without any
SCM involvement )





-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 5:17 PM, Michał Górny  wrote:
> Dnia 2014-07-27, o godz. 17:08:27
> Rich Freeman  napisał(a):
>>
>> I'd think that portage should update vdb as soon as it detects the
>> dependency change.  Then B would no longer depend on A in vdb.  It
>> shouldn't hold onto outdated information.
>
> You just introduced the opposite breakage -- when a dependency on C was
> added, it ends up in vdb before C is installed. Now if C and current
> version of A are removed before C gets installed, you end up having
> broken dependency in vdb.
>

There is a possibility that you had the broken dependency before you
even synced - you just didn't realize it yet.  After all, if the dep
was added only as the result of an actual on-disk change, there should
have been a revbump.  So, having a missing dep in vdb just makes vdb
reflect reality.  You can get a missing dep in vdb by unmerging a
package without using depclean.

An unmet dependency is still a dependency.

> Plus, 'as soon' means you're making --pretend actually write something.
> That's bad.

This isn't all that different from package moves.  I believe that
modifies vdb as soon as it detects updates.

There is the issue of syncing and then not updating and then having
all the packages removed from the tree  - you're then missing a
package and have no easy way to install it.  But, in that case you can
put the necessary ebuilds into your overlay and then portage can make
everything right.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-27, o godz. 17:08:27
Rich Freeman  napisał(a):

> On Sun, Jul 27, 2014 at 4:24 PM, Michał Górny  wrote:
> > Dnia 2014-07-27, o godz. 10:42:19
> >
> > Consider the following:
> >
> > 1. A depends on B, both are installed,
> >
> > 2. dependency on B is removed, emerge --depclean uninstalls B thanks
> > to dynamic-deps,
> >
> > 3. B is treecleaned (nothing depends on it),
> >
> > 4. old version of A is removed (user doesn't update it yet), therefore
> > dependency on B is restored from vdb.
> >
> > So, now user has package A installed which has unsatisfied dependency
> > on non-available package.  
> 
> I'd think that portage should update vdb as soon as it detects the
> dependency change.  Then B would no longer depend on A in vdb.  It
> shouldn't hold onto outdated information.

You just introduced the opposite breakage -- when a dependency on C was
added, it ends up in vdb before C is installed. Now if C and current
version of A are removed before C gets installed, you end up having
broken dependency in vdb...

Plus, 'as soon' means you're making --pretend actually write something.
That's bad.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-27, o godz. 22:51:13
Peter Stuge  napisał(a):

> What is the purpose of keeping only dependencies as-they-were when
> the package was installed, if the package manager does not somehow
> benefit from that information in the future?

You have to ask the one who implemented that. Maybe it was for
performance reasons (dynamic-deps cause a lot of overhead
themselves...), maybe just because it was hard to implement it
properly.

Feel free to provide a patch. Just please don't forget that you
shouldn't write the updates before emerge finishes updates or starts
depclean on relevant packages, so that the installed package dependency
tree doesn't end up being broken.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 4:24 PM, Michał Górny  wrote:
> Dnia 2014-07-27, o godz. 10:42:19
>
> Consider the following:
>
> 1. A depends on B, both are installed,
>
> 2. dependency on B is removed, emerge --depclean uninstalls B thanks
> to dynamic-deps,
>
> 3. B is treecleaned (nothing depends on it),
>
> 4. old version of A is removed (user doesn't update it yet), therefore
> dependency on B is restored from vdb.
>
> So, now user has package A installed which has unsatisfied dependency
> on non-available package.

I'd think that portage should update vdb as soon as it detects the
dependency change.  Then B would no longer depend on A in vdb.  It
shouldn't hold onto outdated information.  Basically a dependency
change should trigger a no-rebuild merge if it is safe to do so, and
if not there should be a revbump anyway.

If A is removed before there is a sync, then A is still installed on
the user's system and portage still thinks that B depends on A, so it
remains consistent.

I acknowldege that this isn't how portage behaves today.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Sun, 27 Jul 2014 22:51:13 +0200
Peter Stuge  wrote:
> To me it seems like a simple data model bug that vdb does not get
> updated to reflect the new situation after step 2 above.

Rewriting VDB won't help if the user doesn't sync at the right time.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Peter Stuge]

Michał Górny wrote:
> Consider the following:
> 
> 1. A depends on B, both are installed,
> 
> 2. dependency on B is removed, emerge --depclean uninstalls B thanks
> to dynamic-deps,
> 
> 3. B is treecleaned (nothing depends on it),

So far I follow.


> 4. old version of A is removed (user doesn't update it yet),

ebuild removed from tree.


> therefore dependency on B is restored from vdb.

Why would removing the ebuild from the tree *add* a dependency to a
system where the ebuild is installed? That doesn't seem desirable.

To me it seems like a simple data model bug that vdb does not get
updated to reflect the new situation after step 2 above.


> So, now user has package A installed which has unsatisfied
> dependency on non-available package.

What is the purpose of keeping only dependencies as-they-were when
the package was installed, if the package manager does not somehow
benefit from that information in the future?


//Peter


pgp_7dQnFYlMN.pgp
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-27, o godz. 10:42:19
Rich Freeman  napisał(a):

> On Sun, Jul 27, 2014 at 10:05 AM, "Paweł Hajdan, Jr."
>  wrote:
> > On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
> >> Michał has documented the shortcomings of dynamic deps in our wiki[0].
> >> (Thank you!) [...]
> >> [0]  
> >
> > There's one more thing I'd like to ask about:
> >
> > For "Minor linking change w/ dependency change (unnecessary linking
> > removed)" the "dynamic deps" cell is red with "revbump + mostly
> > unnecessary rebuild", and "static deps" says "applied after rebuild".
> >
> > Arguably with dynamic deps one could also skip the revbump, and the
> > update would similarly be applied after rebuild.
> 
> One thing I would question in that table is "applied immediately (but
> can break hard when dynamic-deps stop working))."  How can dynamically
> removing an "unused dependency" cause something to break, setting
> aside bugs in the package manager?  If removing a dependency causes
> something to break, how can it be "unused?"

Consider the following:

1. A depends on B, both are installed,

2. dependency on B is removed, emerge --depclean uninstalls B thanks
to dynamic-deps,

3. B is treecleaned (nothing depends on it),

4. old version of A is removed (user doesn't update it yet), therefore
dependency on B is restored from vdb.

So, now user has package A installed which has unsatisfied dependency
on non-available package.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Sun, 27 Jul 2014 19:16:58 +0200
Peter Stuge  wrote:
> Ciaran McCreesh wrote:
> > Peter Stuge  wrote:
> > > Ciaran McCreesh wrote:
> > > > Rich Freeman  wrote:
> > > > > What would you do away with?  Being able to virtualize
> > > > > packages without recompiling everything that depends on them?
> > > > 
> > > > Well that's never worked properly or consistently to begin with
> > > 
> > > Please answer the question?
> > 
> > That does answer the question
> 
> It doesn't. The question is "What would you do away with?" and you
> didn't list the things you would do away with.
> 
> Please answer the question?

Dynamic dependencies, || dependencies (in favour of a heavily
restricted, non-abusable alternative) and REQUIRED_USE would be an
excellent start.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Peter Stuge]

Ciaran McCreesh wrote:
> Peter Stuge  wrote:
> > Ciaran McCreesh wrote:
> > > Rich Freeman  wrote:
> > > > What would you do away with?  Being able to virtualize packages
> > > > without recompiling everything that depends on them?
> > > 
> > > Well that's never worked properly or consistently to begin with
> > 
> > Please answer the question?
> 
> That does answer the question

It doesn't. The question is "What would you do away with?" and you
didn't list the things you would do away with.

Please answer the question?


//Peter


pgpiVMs7V6A6W.pgp
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Sun, 27 Jul 2014 19:02:05 +0200
Peter Stuge  wrote:
> Ciaran McCreesh wrote:
> > Rich Freeman  wrote:
> > > What would you do away with?  Being able to virtualize packages
> > > without recompiling everything that depends on them?
> > 
> > Well that's never worked properly or consistently to begin with
> 
> Please answer the question?

That does answer the question: as can be seen from the udev virtual
example earlier in this thread, we do not have a way of reliably
avoiding recompilations. Thus it is wrong to pretend that we have this
feature now, or that fully static dependencies would somehow be to blame
for recompilations.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Peter Stuge]

Ciaran McCreesh wrote:
> Rich Freeman  wrote:
> > What would you do away with?  Being able to virtualize packages
> > without recompiling everything that depends on them?
> 
> Well that's never worked properly or consistently to begin with

Please answer the question?


//Peter


pgpNaSXskEzkH.pgp
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Kent Fredric]

On 28 July 2014 03:52, Rich Freeman  wrote:

> Why?  Is this about removing an unused dependency?
>
> > 3. Gentoo simply tweaks the ebuild and doesn't bump [A]
>
> What is "[A]?"  What ebuild was tweaked, and how was it tweaked?
>

Here, "A" is the derived version of the ebuild of "Foo" the user installed,
in the present dynamic deps situation.

That is, Although user and tree have the same $PVR value, one has different
DEPENDS, and my impression based on past discussions is in that scenario,
the DEPENDS from /usr/portage/  take preceidence over VDB.

Though perhaps it depends on your definition of "used". If your definition
of "used" is "other things in portage needs it", then this case is
satisfied by it being "used" by the world file.

Users will likely know in such a scenario that they are "using" a thing
that is no longer in tree, but if it no longer being "in tree" changes how
its dependencies resolve for the worse.

*Most* examples of this you will see will be resolved by pkgmoves munging
the installed dependencies in VDB to cease to be broken.

I guess, in a way, pkgmove and dynamic deps are the same problem from
different perspectives.

pkgmove is an instruction that "set( X ) that requires Y needs to change"

and dynamic deps are "X  requires set ( Y ) and needs to change."

Just in the case of the former, the effect is made permanent as part of the
sync, and in the case of the latter ( based on my understanding ) the
effect is temporary ( bounded by the lifetime of the ebuild in /usr/portage
) and resolved at runtime for each and every merge 

( Do correct me if I'm wrong about this )





-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 11:44 AM, Kent Fredric  wrote:
>
> On 28 July 2014 02:42, Rich Freeman  wrote:
>>
>> One thing I would question in that table is "applied immediately (but
>> can break hard when dynamic-deps stop working))."  How can dynamically
>> removing an "unused dependency" cause something to break, setting
>> aside bugs in the package manager?  If removing a dependency causes
>> something to break, how can it be "unused?"
>
>
> My apologies if this scenario has been explained before, I saw things along
> these lines, but must may have missed their point:
>
> I get the impression that this happens:
>
> 1. User installs Foo
> 2. Gentoo needs to change what Foo depends on

Why?  Is this about removing an unused dependency?

> 3. Gentoo simply tweaks the ebuild and doesn't bump [A]

What is "[A]?"  What ebuild was tweaked, and how was it tweaked?

> 8. Shadowing effect of [A] is removed, and Foo is now back depending on the
> wrong thing.

What do you mean by "shadowing effect?"

You need to be a bit more clear on your scenario here.  I'm not really
getting much out of your example.  My question was how can removing an
unused dependency break things.  I'm not sure if your example includes
an unused dependency, whether it was removed, and whether it breaks
anything.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Kent Fredric]

On 28 July 2014 02:42, Rich Freeman  wrote:

> One thing I would question in that table is "applied immediately (but
> can break hard when dynamic-deps stop working))."  How can dynamically
> removing an "unused dependency" cause something to break, setting
> aside bugs in the package manager?  If removing a dependency causes
> something to break, how can it be "unused?"
>

My apologies if this scenario has been explained before, I saw things along
these lines, but must may have missed their point:

I get the impression that this happens:

1. User installs Foo
2. Gentoo needs to change what Foo depends on
3. Gentoo simply tweaks the ebuild and doesn't bump [A]
4. User syncs.
5. Subsequent emerges see dependencies from [A] which are fixed and works
in the interim
6. A gets removed.
7. User syncs.
8. Shadowing effect of [A] is removed, and Foo is now back depending on the
wrong thing.

=~ So although this problem will be resolved by also removing Foo, Foo
still exists on the system with bad dependencies which could lead to some
kind of problem, preventing emerge resolution from making sense.

Sure, Foo will get reaped by a depclean if it is no longer in world and
nothing depends on it,  but depclean tends to require world to be
deeply resolved first.



And more questions for the -r1.1 style "Ship a new ebuild" style proposal.

::gentoo ships Foo-1.0
::whatever *also* ships Foo-1.0 ( with different dependencies )

User installs from ::gentoo

::Whatever publishes a -r0.1 style ( or equivalent) dependency fix

What happens for the user, and why.

I'm of the mind ::Whatever should not tamper with ::gentoo here, though it
could be useful, it could also prove problematic.

Or maybe the idea is just too weird and quirky to implement this way.

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Sun, 27 Jul 2014 11:09:05 -0400
Rich Freeman  wrote:
> On Sun, Jul 27, 2014 at 10:59 AM, Ciaran McCreesh
>  wrote:
> > On Sun, 27 Jul 2014 16:56:17 +0200
> > ""Paweł Hajdan, Jr.""  wrote:
> >> It seems really tricky to correctly reason about dependency
> >> resolution.
> >
> > It's actually very easy if you do away with all the things that are
> > making it unnecessarily complicated... Nearly all of the complexity
> > is self-inflicted.
> 
> What would you do away with?  Being able to virtualize packages
> without recompiling everything that depends on them?

Well that's never worked properly or consistently to begin with, so all
we'd be doing away with is the pretence that we can get away with it.

> I do appreciate your argument, but at the same time for every complex
> problem there is an answer that is clear, simple, and wrong.

Unfortunately that's the answer Gentoo usually goes with.

> There are a lot of things in Gentoo that could be done in a simpler
> fashion, and 10 years ago Gentoo was a lot simpler than it is today.
> The thing is, all that complexity was added for a reason.

Most of that complexity was added due to "not thinking things through
fully" and "adding in a short-term hack with long-term consequences".
The reason was rarely "we need this complexity", and usually "we need
something, and on the surface of it this looks like it solves exactly
that something".

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 10:59 AM, Ciaran McCreesh
 wrote:
> On Sun, 27 Jul 2014 16:56:17 +0200
> ""Paweł Hajdan, Jr.""  wrote:
>> It seems really tricky to correctly reason about dependency
>> resolution.
>
> It's actually very easy if you do away with all the things that are
> making it unnecessarily complicated... Nearly all of the complexity is
> self-inflicted.

What would you do away with?  Being able to virtualize packages
without recompiling everything that depends on them?

I do appreciate your argument, but at the same time for every complex
problem there is an answer that is clear, simple, and wrong.

There are a lot of things in Gentoo that could be done in a simpler
fashion, and 10 years ago Gentoo was a lot simpler than it is today.
The thing is, all that complexity was added for a reason.  I'm all for
refactoring, but we need to be careful to not toss the baby with the
bathwater if it is at all possible.

It might be an interesting exercise if we could take something like
kde-meta+firefox+openoffice on the desktop/kde profile (or gnome if
you wish) and determine just how many rebuilds would have resulted in
the last six months if all the changes we're talking about actually
involved revbumps, and how much cpu time that would take on an
"average" system (I don't care what but distinguishing
firefox/openoffice/qt/kdelibs from some package with 1k lines of code
is useful).

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Sun, 27 Jul 2014 16:56:17 +0200
""Paweł Hajdan, Jr.""  wrote:
> It seems really tricky to correctly reason about dependency
> resolution.

It's actually very easy if you do away with all the things that are
making it unnecessarily complicated... Nearly all of the complexity is
self-inflicted.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Paweł Hajdan, Jr.]

On 7/27/14, 4:42 PM, Rich Freeman wrote:
> With dynamic deps you'd need to revbump if there is a linking change.
> Otherwise portage would just allow the dependency to be removed, and
> then linking will break, since the executable is unnecessarily linked
> to the dependency (in that scenario).

Right, I see - I think I got that right when first reading this, but got
confused after reading so many messages in this thread. Thanks for
patient explanation.

It seems really tricky to correctly reason about dependency resolution.

> One thing I would question in that table is "applied immediately (but
> can break hard when dynamic-deps stop working))."  How can dynamically
> removing an "unused dependency" cause something to break, setting
> aside bugs in the package manager?  If removing a dependency causes
> something to break, how can it be "unused?"

Yeah, I was also wondering about this.




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-27, o godz. 16:05:34
""Paweł Hajdan, Jr.""  napisał(a):

> On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
> > Michał has documented the shortcomings of dynamic deps in our wiki[0].
> > (Thank you!) [...]
> > [0]  
> 
> There's one more thing I'd like to ask about:
> 
> For "Minor linking change w/ dependency change (unnecessary linking
> removed)" the "dynamic deps" cell is red with "revbump + mostly
> unnecessary rebuild", and "static deps" says "applied after rebuild".
> 
> Arguably with dynamic deps one could also skip the revbump, and the
> update would similarly be applied after rebuild.

No, the update would be applied immediately. In other words, emerge
will remove the dependency and allow it to be depcleaned even though
the package still links to it. You need to revbump to avoid the broken
linkage.


-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Sun, Jul 27, 2014 at 10:05 AM, "Paweł Hajdan, Jr."
 wrote:
> On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
>> Michał has documented the shortcomings of dynamic deps in our wiki[0].
>> (Thank you!) [...]
>> [0]  
>
> There's one more thing I'd like to ask about:
>
> For "Minor linking change w/ dependency change (unnecessary linking
> removed)" the "dynamic deps" cell is red with "revbump + mostly
> unnecessary rebuild", and "static deps" says "applied after rebuild".
>
> Arguably with dynamic deps one could also skip the revbump, and the
> update would similarly be applied after rebuild.

With dynamic deps you'd need to revbump if there is a linking change.
Otherwise portage would just allow the dependency to be removed, and
then linking will break, since the executable is unnecessarily linked
to the dependency (in that scenario).

If you wanted to apply the change after rebuild then you'd need to
have a way to communicate to portage whether a dependency removal can
be applied immediately (extraneous unlinked dependency) or only after
rebuild (extraneous linked dependency).  The table speaks mostly to
portage as-is without additions like those posted in this thread.

I think that USE changes basically involve potentially-unnecessary
rebuilds no mater what in most real-life cases.  Most people use the
--newuse option when updating their system, so even if static
dependencies don't require rebuilds they're going to get them anyway.
Not using --newuse can cause different issues.

One thing I would question in that table is "applied immediately (but
can break hard when dynamic-deps stop working))."  How can dynamically
removing an "unused dependency" cause something to break, setting
aside bugs in the package manager?  If removing a dependency causes
something to break, how can it be "unused?"

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Paweł Hajdan, Jr.]

On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
> Michał has documented the shortcomings of dynamic deps in our wiki[0].
> (Thank you!) [...]
> [0]  

There's one more thing I'd like to ask about:

For "Minor linking change w/ dependency change (unnecessary linking
removed)" the "dynamic deps" cell is red with "revbump + mostly
unnecessary rebuild", and "static deps" says "applied after rebuild".

Arguably with dynamic deps one could also skip the revbump, and the
update would similarly be applied after rebuild.

Is my logic correct? I'm just trying to understand it better, and don't
intend an argument in favor of any of the options.

Paweł



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Sat, 26 Jul 2014 12:36:31 +0800
Patrick Lauer  wrote:

> On Wednesday 23 July 2014 01:06:15 Tom Wijsman wrote:
> > On Tue, 22 Jul 2014 08:10:20 +0300
> > 
> > Samuli Suominen  wrote:
> > > On 22/07/14 04:05, Rick "Zero_Chaos" Farina wrote:
> > > > And just for fun, since no one has mentioned it yet, dynamic
> > > > deps don't work at all on binpkgs since the Packages file
> > > > contains the deps (like vardb) and it doesn't get updated (just
> > > > like vardb).
> > > 
> > > Known long standing pitfall. It's managable.
> > 
> > It is one of the reasons I see binpkgs as breaking progress instead
> > of being part of the progress; it is manageable, but it could be
> > better.
> 
> So you'd rather have people not using Gentoo because you can't
> agilely pivot your strategy mixin?

No, that's not what I've said; the paragraph you quote is based on an
observation, not a statement. In that observation; if people want a
mixin, they would go for a full mixin and not for a half broken mixin.

> Without binpkg support I'd feel the need to hack it up, just to get
> things fast enough. It's one of the features that haven't been made
> popular enough (eh, we could easily provide binpkg-updates for
> @system for all profiles and the most common arches (amd64, x86,
> maybe arm) - I've been running a cronjob doing that for x86+amd64 for
> about a year now)
> 
> This perspective of "I don't need it, thus it shouldn't exist" is
> quite ... amusing.

+1; "It doesn't work, it shouldn't exist" is less amusing, more ideal
it should be "it doesn't work, let's fix or replace it" if we can...

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Taahir Ahmed]

My apologies for the top-reply.

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] don't rely on dynamic deps

[Taahir Ahmed]

It seems like a simple before/after comparison of active useflags and the text 
of the src_* functions (skipping build and install if they are completely 
identical) should catch the majority of unnecessary rebuilds.

On 25 July 2014 13:36 Ian Stakenvicius wrote:
> On 25/07/14 01:15 PM, Andreas K. Huettel wrote:
> >
> >>> Maybe this could be solved by having two kinds of revisions: -
> >>> One would rebuild all as usually (for example, -r1...) - The
> >>> other one would only regenerate VDB and wouldn't change the
> >>> installed files (for example, -r1.1)
> >>>
> >>> But I am not sure if it could be viable from a "technical"
> >>> point of view :(
> >>
> >> I'm afraid it couldn't. The major problem is not knowing *when*
> >> to migrate metadata, portage usually gets that right. The problem
> >> is in getting the correct output which is often near to
> >> impossible.
> >
> > I think we'd appreciate some more information here:
> >
> > * What exactly is the metadata that we're talking about? * How much
> > of it can be generated by sourcing the ebuild, without running
> > phases? * When does that exactly break? Example?
> >
> 
> It may help this overall discussion to step back even further here:
> 
> * What do we want to accomplish, exactly?
> 
>  - allow portage to emerge updated packages without rebuilding them
> (ie running src_* and pkg_*inst phases), when it can be determined
> from metadata changes (or for -rX.Y method perhaps it is assumed
> -just- on filename changes) that this is not necessary.
> 
> * What are all of the cases that would validly trigger this
> dont-bother-to-run-phases shortcut?
> 
> - addition of missing dependency [1]
> - new useflag not enabled by default
> - removed useflag that was not previously enabled
> - changing of dependency atom(s) -- ie swapping a specific atom to
> virtual one
> - EAPI bump (maybe??)
> -   <-- add more please!
> 
> 
> * For each of the above, in what cases is a full rebuild probably
> necessary anyhow??
> 
> - dependency atom minimum version is increased, and the in-VDB version
> of this dep is too old (??)
> - new dependency atom is missing from VDB (??)
> - EAPI bump adds new entries to VDB that need to be calculated from
> say, merge phase
> - 
> 
> 
> 
> ..i think from there we can perhaps determine what metadata could be
> updated and/or needs to be updated to maintain consistency in the
> dont-rebuild cases we want.
> 


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] don't rely on dynamic deps

[Ulrich Mueller]

> On Wed, 23 Jul 2014, Tom Wijsman wrote:

> Using an extension like -rX.Y seems odd; at the very least, I think
> an incremental variable or something along that line in the ebuild
> would work better.

It would also account for changes in eclasses, which any scheme bound
to the ebuild's filename cannot do.

> This allows for array usage like VERSION[dependencies]=1, thus
> allowing other variables to be dynamic as well; you compare that
> number against the one in the vdb, bingo...

Taking this one step further, I wonder if one couldn't directly
compare *DEPEND in the ebuild against the one in the VDB. Why would
one need to introduce another variable?

Ulrich


pgp5oc8HAFva1.pgp
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Patrick Lauer]

On Wednesday 23 July 2014 01:06:15 Tom Wijsman wrote:
> On Tue, 22 Jul 2014 08:10:20 +0300
> 
> Samuli Suominen  wrote:
> > On 22/07/14 04:05, Rick "Zero_Chaos" Farina wrote:
> > > And just for fun, since no one has mentioned it yet, dynamic deps
> > > don't work at all on binpkgs since the Packages file contains the
> > > deps (like vardb) and it doesn't get updated (just like vardb).
> > 
> > Known long standing pitfall. It's managable.
> 
> It is one of the reasons I see binpkgs as breaking progress instead of
> being part of the progress; it is manageable, but it could be better.
> 

So you'd rather have people not using Gentoo because you can't agilely pivot 
your strategy mixin?

Without binpkg support I'd feel the need to hack it up, just to get things 
fast enough. It's one of the features that haven't been made popular enough 
(eh, we could easily provide binpkg-updates for @system for all profiles and 
the most common arches (amd64, x86, maybe arm) - I've been running a cronjob 
doing that for x86+amd64 for about a year now)

This perspective of "I don't need it, thus it shouldn't exist" is quite ... 
amusing.

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Fri, Jul 25, 2014 at 11:01 AM, Ciaran McCreesh
 wrote:
> On Thu, 24 Jul 2014 21:45:58 -0400
> Rich Freeman  wrote:
>> Just a general comment not aimed at this particular part of the thread
>> - a solution doesn't have to be perfect to be useful.
>
> Wrong. The reason everything is such a mess at the moment is precisely
> because we've accumulated so much "good enough" and "not thinking your
> cunning plan all the way through" that nothing is actually correct any
> more.

I think we might be saying the same thing in different ways.  I wasn't
suggesting that we should implement solutions that fail in random
ways, but rather that if necessary we should focus more on simpler
solutions that we can get right, but which perhaps don't cover all of
our problems.

That is, I'm more for a perfect solution for a small problem rather
than a good-enough solution (which isn't) for a big problem.

For example, perhaps there is a way to safely add an unconditional
dependency to an installed package.  That won't solve every dependency
problem, but it could be helpful.

Another way to go about things would be to try to find ways to reduce
the chance of commiting a package that has an incorrect dependency in
the first place, so that we don't have to fix so many mistakes.  Then
perhaps the extra rebuilds when there is a rare mistake might be more
forgivable.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 25/07/14 01:36 PM, Ian Stakenvicius wrote:
> On 25/07/14 01:15 PM, Andreas K. Huettel wrote:
> 
 Maybe this could be solved by having two kinds of revisions:
 - One would rebuild all as usually (for example, -r1...) -
 The other one would only regenerate VDB and wouldn't change
 the installed files (for example, -r1.1)
 
 But I am not sure if it could be viable from a "technical" 
 point of view :(
>>> 
>>> I'm afraid it couldn't. The major problem is not knowing
>>> *when* to migrate metadata, portage usually gets that right.
>>> The problem is in getting the correct output which is often
>>> near to impossible.
> 
>> I think we'd appreciate some more information here:
> 
>> * What exactly is the metadata that we're talking about? * How
>> much of it can be generated by sourcing the ebuild, without
>> running phases? * When does that exactly break? Example?
> 
> 
> It may help this overall discussion to step back even further
> here:
> 
> * What do we want to accomplish, exactly?
> 
> - allow portage to emerge updated packages without rebuilding them 
> (ie running src_* and pkg_*inst phases), when it can be determined 
> from metadata changes (or for -rX.Y method perhaps it is assumed 
> -just- on filename changes) that this is not necessary.
> 
> * What are all of the cases that would validly trigger this 
> dont-bother-to-run-phases shortcut?
> 
> - addition of missing dependency [1] - new useflag not enabled by
> default - removed useflag that was not previously enabled -
> changing of dependency atom(s) -- ie swapping a specific atom to 
> virtual one - EAPI bump (maybe??) -   <-- add more please!
> 
> 
> * For each of the above, in what cases is a full rebuild probably 
> necessary anyhow??
> 
> - dependency atom minimum version is increased, and the in-VDB
> version of this dep is too old (??) - new dependency atom is
> missing from VDB (??) - EAPI bump adds new entries to VDB that need
> to be calculated from say, merge phase - 
> 
> 
> 
> ..i think from there we can perhaps determine what metadata could
> be updated and/or needs to be updated to maintain consistency in
> the dont-rebuild cases we want.
> 

I forgot: [1] since the package has merged prior to this anyhow,
whatever dependency it was must have been installed already when this
package was emerged.  So if it can still be found in vdb then we
should be safe to add the dependency to vdb of this package, otherwise
we may need to trigger a rebuild.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlPSllcACgkQ2ugaI38ACPCL3QD+ITe3bzJrJIORniniY9rzTEd5
W5nmL5zB+HiZuMZKdZQA/jrn88lx8bfn/f6DetPdrYiFjjShcXoeMbh5nCNmtcz9
=LEAi
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 25/07/14 01:15 PM, Andreas K. Huettel wrote:
> 
>>> Maybe this could be solved by having two kinds of revisions: -
>>> One would rebuild all as usually (for example, -r1...) - The
>>> other one would only regenerate VDB and wouldn't change the 
>>> installed files (for example, -r1.1)
>>> 
>>> But I am not sure if it could be viable from a "technical"
>>> point of view :(
>> 
>> I'm afraid it couldn't. The major problem is not knowing *when*
>> to migrate metadata, portage usually gets that right. The problem
>> is in getting the correct output which is often near to
>> impossible.
> 
> I think we'd appreciate some more information here:
> 
> * What exactly is the metadata that we're talking about? * How much
> of it can be generated by sourcing the ebuild, without running 
> phases? * When does that exactly break? Example?
> 

It may help this overall discussion to step back even further here:

* What do we want to accomplish, exactly?

 - allow portage to emerge updated packages without rebuilding them
(ie running src_* and pkg_*inst phases), when it can be determined
from metadata changes (or for -rX.Y method perhaps it is assumed
- -just- on filename changes) that this is not necessary.

* What are all of the cases that would validly trigger this
dont-bother-to-run-phases shortcut?

- - addition of missing dependency [1]
- - new useflag not enabled by default
- - removed useflag that was not previously enabled
- - changing of dependency atom(s) -- ie swapping a specific atom to
virtual one
- - EAPI bump (maybe??)
- -   <-- add more please!


* For each of the above, in what cases is a full rebuild probably
necessary anyhow??

- - dependency atom minimum version is increased, and the in-VDB version
of this dep is too old (??)
- - new dependency atom is missing from VDB (??)
- - EAPI bump adds new entries to VDB that need to be calculated from
say, merge phase
- - 



..i think from there we can perhaps determine what metadata could be
updated and/or needs to be updated to maintain consistency in the
dont-rebuild cases we want.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlPSlY0ACgkQ2ugaI38ACPBDHAD+POgCJlbPv9HHwhK4wxd8b2ir
ywM71Aemu6SPfCTE2MIBAKATag94jCHLlqwkYN2jrW23tYqTi5ajOwLzoZ/EqHx0
=qUpY
-END PGP SIGNATURE-

Re: Re: [gentoo-dev] don't rely on dynamic deps

[Andreas K. Huettel]

> > Maybe this could be solved by having two kinds of revisions:
> > - One would rebuild all as usually (for example, -r1...)
> > - The other one would only regenerate VDB and wouldn't change the
> > installed files (for example, -r1.1)
> > 
> > But I am not sure if it could be viable from a "technical" point of
> > view :(
> 
> I'm afraid it couldn't. The major problem is not knowing *when* to
> migrate metadata, portage usually gets that right. The problem is in
> getting the correct output which is often near to impossible.

I think we'd appreciate some more information here:

* What exactly is the metadata that we're talking about?
* How much of it can be generated by sourcing the ebuild, without running 
phases?
* When does that exactly break? Example?

Thanks!!!

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, council

Re: [gentoo-dev] don't rely on dynamic deps

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 25/07/14 17:01, Ciaran McCreesh wrote:
> The reason everything is such a mess at the moment is precisely 
> because we've accumulated so much "good enough" and "not thinking
> your cunning plan all the way through" that nothing is actually
> correct any more. It's one thing to get away with this
> occasionally, but quite another to build an entire system upon it.
> We can't afford more mistakes, and we have to actively fix existing
> ones.
This is my position as well.
- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlPSfWkACgkQRtClrXBQc7XOcwD/Vmzd/B4IysGbczDAxI/X6CLw
miOuIePVOSwIWRSMvx4A/0Kdy6SCxx2JCko9xMTcnAagqaoleWJeoPIwb1bUkE74
=2rq+
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[hasufell]

Ciaran McCreesh:
> On Fri, 25 Jul 2014 15:23:58 +
> hasufell  wrote:
>>> That's not really helpful advice: dynamic dependencies can't be
>>> fixed. Instead, you should say that anyone who thinks they have an
>>> idea on how to fix dynamic deps should think about it until they
>>> understand why it's wrong...
>>
>> I was rather talking about the "fix useless rebuilds" issue. It's a
>> valid point.
> 
> It's like saying "OK, the house is on fire, but I really like the
> wallpaper, so we should just stay in the building"...
> 
>> What would you suggest? Can the VDB be fixed in another way to avoid
>> such rebuilds?
> 
> There are ways of doing it, if you're prepared to make ebuild authors
> put in an awful lot of work for very little gain. But it shouldn't be a
> priority, and we need to fix existing breakages before doing something
> ambitious like that.
> 

I agree. After all, we have *-bin packages for stuff like libreoffice,
pypy, firefox etc anyway if rebuilding is really a problem for someone.

That said, it might maybe even make more sense to provide an official
binhost, although that is probably not less ambitious as fixing VDB on
the fly.

Sounds like that would be an interesting gentoo project. But afais PMS
doesn't really specify how binary packages should look like, so we will
hit incompatibility problems there as well.

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Fri, 25 Jul 2014 15:23:58 +
hasufell  wrote:
> > That's not really helpful advice: dynamic dependencies can't be
> > fixed. Instead, you should say that anyone who thinks they have an
> > idea on how to fix dynamic deps should think about it until they
> > understand why it's wrong...
> 
> I was rather talking about the "fix useless rebuilds" issue. It's a
> valid point.

It's like saying "OK, the house is on fire, but I really like the
wallpaper, so we should just stay in the building"...

> What would you suggest? Can the VDB be fixed in another way to avoid
> such rebuilds?

There are ways of doing it, if you're prepared to make ebuild authors
put in an awful lot of work for very little gain. But it shouldn't be a
priority, and we need to fix existing breakages before doing something
ambitious like that.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[hasufell]

Ciaran McCreesh:
> On Fri, 25 Jul 2014 15:09:55 +
> hasufell  wrote:
>> Everyone else who thinks got an idea on how to fix dynamic deps
>> support (or similar) should:
>> * write a PMS patch and get it merged
>> * join the portage team and volunteer to implement it instead of
>> yelling at them
> 
> That's not really helpful advice: dynamic dependencies can't be fixed.
> Instead, you should say that anyone who thinks they have an idea on how
> to fix dynamic deps should think about it until they understand why
> it's wrong...
> 

I was rather talking about the "fix useless rebuilds" issue. It's a
valid point.

What would you suggest? Can the VDB be fixed in another way to avoid
such rebuilds?

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Fri, 25 Jul 2014 15:09:55 +
hasufell  wrote:
> Everyone else who thinks got an idea on how to fix dynamic deps
> support (or similar) should:
> * write a PMS patch and get it merged
> * join the portage team and volunteer to implement it instead of
> yelling at them

That's not really helpful advice: dynamic dependencies can't be fixed.
Instead, you should say that anyone who thinks they have an idea on how
to fix dynamic deps should think about it until they understand why
it's wrong...

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[hasufell]

Ian Stakenvicius:
> Dynamic deps are the best solution outside of the (rather limited)
> profiles/updates functions we have right now to allow us to make
> whatever non-files-on-${ROOT} changes we need to make to the vdb.  So
> realistically what we should be doing is either trying to work out a
> better solution to dynamic deps (something that will failover nicely
> for PMs that don't support dynamic deps) or perhaps adding more
> functions to support VDB updating via profiles/updates/
> 
> Am I off-base here?  Thoughts?
> 
> 

Yes, as was already explained. Those are currently just dreams or
abstract thoughts.

Dynamics deps are already broken, not consistently enabled (e.g. when
subslots are in use), optional and not defined in PMS.

People really don't seem to understand what is going on here. We rely on
behavior that depends not only on a portage specific feature, but also
on the context and can pretty much be considered undefined.

I guess I have to repost
https://wiki.gentoo.org/wiki/Project:Portage/Dynamic_dependencies

Dynamic deps don't work for you, even if you think that. Coming up with
an alternative approach will probably take a lot of effort and shouldn't
be considered a blocker to fix a fundamental bug in dependency
calculation, which is already broken in portage in many ways.

If we already bikeshed about one of the simplest ways to improve
dependency calculation, I wonder what will happen if someone wants to
actually fix it.

Everyone else who thinks got an idea on how to fix dynamic deps support
(or similar) should:
* write a PMS patch and get it merged
* join the portage team and volunteer to implement it instead of yelling
at them

Re: [gentoo-dev] don't rely on dynamic deps

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 25/07/14 10:44 AM, Ian Stakenvicius wrote:
> On 22/07/14 06:44 PM, Tom Wijsman wrote:
>> On Tue, 22 Jul 2014 09:53:49 -0400 Ian Stakenvicius 
>>  wrote:
> 
>>> Using ${PVR} to detect how portage should update things would
>>> be asking for trouble, imo.
> 
>> This entire sub thread reads like a dynamic dependencies 
>> alternative in disguise, the difference lies in an increase of
>> the level of control and in the place where this then gets 
>> reimplemented.
> 
> 
> It is.
> 
> Here's the situation as I see it -- the portage tree needs to be 
> consistent at snapshot time.  But things can change all over the 
> place, deps are moved, virtuals replace single or groups of atoms, 
> packages get split, etc. etc. etc.
> 
> Dynamic deps are the best solution outside of the (rather limited) 
> profiles/updates functions we have right now to allow us to make 
> whatever non-files-on-${ROOT} changes we need to make to the vdb.
> So realistically what we should be doing is either trying to work
> out a better solution to dynamic deps (something that will failover
> nicely for PMs that don't support dynamic deps) or perhaps adding
> more functions to support VDB updating via profiles/updates/
> 
> Am I off-base here?  Thoughts?
> 

Ignore this, i should've read the rest of the thread first before posting.

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlPScN4ACgkQ2ugaI38ACPBS5gD+MXU3VUvwhp1u/0wIDHeXEQdX
TmJXhvDhuhuE+7ehee0A/1HGASXipYsejfJxPesQFO4Egs1Yzj20PXlVmil9H8FY
=WwNJ
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Thu, 24 Jul 2014 21:45:58 -0400
Rich Freeman  wrote:
> Just a general comment not aimed at this particular part of the thread
> - a solution doesn't have to be perfect to be useful.

Wrong. The reason everything is such a mess at the moment is precisely
because we've accumulated so much "good enough" and "not thinking your
cunning plan all the way through" that nothing is actually correct any
more. It's one thing to get away with this occasionally, but quite
another to build an entire system upon it. We can't afford more
mistakes, and we have to actively fix existing ones.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 22/07/14 06:44 PM, Tom Wijsman wrote:
> On Tue, 22 Jul 2014 09:53:49 -0400 Ian Stakenvicius
>  wrote:
> 
>> Using ${PVR} to detect how portage should update things would be
>> asking for trouble, imo.
> 
> This entire sub thread reads like a dynamic dependencies
> alternative in disguise, the difference lies in an increase of the
> level of control and in the place where this then gets
> reimplemented.


It is.

Here's the situation as I see it -- the portage tree needs to be
consistent at snapshot time.  But things can change all over the
place, deps are moved, virtuals replace single or groups of atoms,
packages get split, etc. etc. etc.

Dynamic deps are the best solution outside of the (rather limited)
profiles/updates functions we have right now to allow us to make
whatever non-files-on-${ROOT} changes we need to make to the vdb.  So
realistically what we should be doing is either trying to work out a
better solution to dynamic deps (something that will failover nicely
for PMs that don't support dynamic deps) or perhaps adding more
functions to support VDB updating via profiles/updates/

Am I off-base here?  Thoughts?

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlPSbVgACgkQ2ugaI38ACPBDpAEAnqx8hBGkmmiVGE6Pz7Rh+BE9
ed5KuWwihJdjPGjXdjoA/ifwGD8oUO8epWIq4rahW+egUFhklKtPu57jIYSjY90y
=cZb0
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Pacho Ramos]

El mié, 23-07-2014 a las 14:33 +0100, Ciaran McCreesh escribió:
> On Mon, 21 Jul 2014 23:06:07 +0200
> Pacho Ramos  wrote:
> > Maybe this could be solved by having two kinds of revisions:
> > - One would rebuild all as usually (for example, -r1...)
> > - The other one would only regenerate VDB and wouldn't change the
> > installed files (for example, -r1.1)
> > 
> > But I am not sure if it could be viable from a "technical" point of
> > view :(
> 
> This in no way solves the problem. Consider the following course of
> events:
> 
> User installs foo-1.1-r1
> Developer makes foo-1.1-r1.1
> foo-1.1* is removed from the tree
> User syncs
> 

Isn't there a way for PMs to know that they need to not rebuild full if
user has 1.1-r1 *installed* in his system and pretends to update to
-r1.1? 

Re: [gentoo-dev] don't rely on dynamic deps

[Pacho Ramos]

El vie, 25-07-2014 a las 00:06 +0200, Michał Górny escribió:
[...]
> > Maybe this could be solved by having two kinds of revisions:
> > - One would rebuild all as usually (for example, -r1...)
> > - The other one would only regenerate VDB and wouldn't change the
> > installed files (for example, -r1.1)
> > 
> > But I am not sure if it could be viable from a "technical" point of
> > view :(
> 
> I'm afraid it couldn't. The major problem is not knowing *when* to
> migrate metadata, portage usually gets that right. The problem is in
> getting the correct output which is often near to impossible.
> 

But, isn't it something up to developer in that case? I mean, we should
have a list of things that deserve that VDB regeneration and, then, make
the -r1.1 revbump instead of the major one.

Re: [gentoo-dev] don't rely on dynamic deps

[Pacho Ramos]

El mar, 22-07-2014 a las 23:56 +0200, Tom Wijsman escribió:
[...]
> Useless triggers are the problem; why are the rev bumps needed, why are
> dependencies forgotten, ...? Sounds like a developer work flow issue...
> 
> https://bugs.gentoo.org/show_bug.cgi?id=499852
> 

There are lots of cases of upstream forgetting to update configure
checks and changing without noticing the real requirements. For example,
moving from gtk+-2.90.0 to gtk+-3.3. Since we don't such old setups, we
don't notice a newer gtk+ is needed. Sometimes, a user with a really old
stable setup tries to update, find the problem and we need to update the
dependency

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-21, o godz. 21:34:10
Alexandre Rostovtsev  napisał(a):

> On Mon, 2014-07-21 at 22:56 +0200, Michał Górny wrote:
> > Yes, it does. I'm not sure if it leads anywhere, though. Dynamic deps
> > are a pipe dream. You can't implement them properly, so we're using
> > half-working implementation as an excuse to be lazy.
> 
> Why not adapt the updates mechanism for modifying rdepends? Perhaps
> something like
> 
> rdepends-add foo-bar/blah-3.14 "wombat? ( >=dev-libs/wombat-1.0 )"
> 
> This would give the package manager all the benefits of static dep
> resolution while allowing us to dynamically make simple changes (like
> adding a missing dependency to an ebuild) without forcing users to
> rebuild the package.

At the moment updates are stateless. In other words, PM has no idea if
the update has been applied or not, and the update should be defined so
that it can't be applied multiple times.

IOW, if you do a pkgmove via updates, the originating package ebuild
must no longer exist so that the update can be only applied once. If
you do a slotmove, the originating slot must no longer be used
in affected ebuilds.

Now, with dependency updates you lack this guarantee. Package manager
can add the dependency... and then add it again... and again...
and again. It could supposedly try to match dependencies and find out
whether a particular dependency has been added already but that sounds
like something that could easily cause issues in the future.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Rich Freeman]

On Wed, Jul 23, 2014 at 9:33 AM, Ciaran McCreesh
 wrote:
> On Mon, 21 Jul 2014 23:06:07 +0200
> Pacho Ramos  wrote:
>> Maybe this could be solved by having two kinds of revisions:
>> - One would rebuild all as usually (for example, -r1...)
>> - The other one would only regenerate VDB and wouldn't change the
>> installed files (for example, -r1.1)
>>
>> But I am not sure if it could be viable from a "technical" point of
>> view :(
>
> This in no way solves the problem. Consider the following course of
> events:
>
> User installs foo-1.1-r1
> Developer makes foo-1.1-r1.1
> foo-1.1* is removed from the tree
> User syncs

An updates-like mechanism would help here, since the updates could
persist longer.

Also, the user is probably going to end up uninstalling foo anyway or
updating it to a newer revision, which means that whatever was broken
with -r1 will tend to become a bit of a moot issue.  Portage doesn't
really support hanging onto PVs that aren't in the tree all that well
to begin with.

Just a general comment not aimed at this particular part of the thread
- a solution doesn't have to be perfect to be useful.  If we come up
with a good clean solution that avoids rebuilds in a half-dozen
specific circumstances and we agree to only use it in those
circumstances, there is no reason we can't use it, even if there is
some other circumstance that will still require a revbump.  I'm
sensing in this thread that we're forcing ourselves to choose between
a hack that can be applied 100% of the time but which can break
randomly, or a hypothetical perfect solution that never breaks but
which will probably never exist either.  A solution that works 80% of
the time and never breaks as long as it is properly applied is an
acceptable compromise.

Rich

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Mon, 21 Jul 2014 23:06:07 +0200
Pacho Ramos  wrote:
> Maybe this could be solved by having two kinds of revisions:
> - One would rebuild all as usually (for example, -r1...)
> - The other one would only regenerate VDB and wouldn't change the
> installed files (for example, -r1.1)
> 
> But I am not sure if it could be viable from a "technical" point of
> view :(

This in no way solves the problem. Consider the following course of
events:

User installs foo-1.1-r1
Developer makes foo-1.1-r1.1
foo-1.1* is removed from the tree
User syncs

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Tue, 22 Jul 2014 20:01:55 -0400
Ian Stakenvicius  wrote:
> The thing about -rX.Y is that it allows this new-dynamic-deps thing
> to act like a regular rev bump to any PM that doesn't bother to
> implement it (or dynamic deps for that matter).  Instant
> backwards-compatibility is a handy feature.

...but it doesn't actually solve the problem.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Ian Stakenvicius]

Sent from an iPhone, sorry for the HTML...

> On Jul 22, 2014, at 6:44 PM, Tom Wijsman  wrote:
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On Tue, 22 Jul 2014 09:53:49 -0400
> Ian Stakenvicius  wrote:
> 
>> Using ${PVR} to detect how portage should update things
>> would be asking for trouble, imo.
> 
> This entire sub thread reads like a dynamic dependencies alternative in
> disguise, the difference lies in an increase of the level of control
> and in the place where this then gets reimplemented.
> 
> The increase of control comes from the maintainer being able to decide
> whether the dependencies in the vdb are updated or not; however, this
> gives rise to a mindset where you consider this level of control for
> other variables as well (which syntax we'll [ab]use for that?) as well
> as end up with more ebuilds for the sake of updating vdb dependencies.
> 
> Using an extension like -rX.Y seems odd; at the very least, I think an
> incremental variable or something along that line in the ebuild would
> work better. This allows for array usage like VERSION[dependencies]=1,
> thus allowing other variables to be dynamic as well; you compare that
> number against the one in the vdb, bingo...
> 
> Or is it just a figment?
> 
> Please think a design through and don't take a cheap shot with -rX.Y.
> 

The thing about -rX.Y is that it allows this new-dynamic-deps thing to act like 
a regular rev bump to any PM that doesn't bother to implement it (or dynamic 
deps for that matter).  Instant backwards-compatibility is a handy feature.

Re: [gentoo-dev] don't rely on dynamic deps

[Alexandre Rostovtsev]

On Wed, 2014-07-23 at 01:13 +0200, Tom Wijsman wrote:
> On Mon, 21 Jul 2014 21:34:10 -0400
> Alexandre Rostovtsev  wrote:
> > Why not adapt the updates mechanism for modifying rdepends? Perhaps
> > something like
> > 
> > rdepends-add foo-bar/blah-3.14 "wombat? ( >=dev-libs/wombat-1.0 )"
> > 
> > This would give the package manager all the benefits of static dep
> > resolution while allowing us to dynamically make simple changes (like
> > adding a missing dependency to an ebuild) without forcing users to
> > rebuild the package.
> 
> Thinking this through:
> 
>  1) What about rdepends-change and rdepends-del? If you only support
>  addition; you get the same problem as with things like pkgmove,
>  changing and/or removing it could become somewhat problematic.

rdepends-add is easy to implement: just append a string to RDEPENDS and
reparse it. Deletion is trickier. It's clear what it means to delete an
atom from a flat list of atoms. But specifying what it means to delete a
part of a complex expression (conditionals, disjunctions) requires some
careful thought.

>  2) This needs two commits every time you want to do this; one commit
>  for the updates/, another to keep the ebuild recent for (rev)bumps.

Yes, but that's just more incentive to migrate to git :)

>  3) It'll be a lot of fun to attempt to support this in Repoman.

Ideally, repoman would suggest what you should add to updates/ when it
detects that you are committing a change to an existing ebuild.

>  4) How do we clean up these entries? Doesn't this info grow fast?

The point is to *not* clean up these entries for months/years. I want to
address the situation where a users installs from an ebuild with wrong
dependencies, the next day the ebuild gets its dependencies fixed in
cvs, two weeks later the ebuild gets deleted from cvs, and only then the
user resyncs. With our current dynamic depends, as well as with the
proposed "minor revisions" mechanism, the user will still have a broken
vdb. I want to make sure the user's vdb gets updated even when the
ebuild in question is gone from portage.

>  5) The first paramater: Should that point to a single ebuild? Should
>  that support ranges?

Just like everything else in profiles/, so for now, that means a single
dependency specification. If you want version ranges, they should be
allowed in package.mask too :)


signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] don't rely on dynamic deps

[hasufell]

Samuli Suominen:
> 
> On 22/07/14 10:25, "Paweł Hajdan, Jr." wrote:
>> On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
>>> 2. Remove dynamic-deps. This is what I think currently makes sense.
>> +1 I also think it's the best option.
>>
>>
> 
> Not before someone has implemented an alternative way to avoid useless
> rebuilding.
> The quality of the distribution doesn't improve by killing one of the
> most important
> features the package manager has.
> The quality of the distribution improves by providing an alternative
> with less problems.
> 
> Sounds like to me, that the people who want to remove the feature so
> badly, are the
> ones volunteering for the job as well.
> 


There seems to be a misunderstanding.

The feature is already _optional_ and not even active in all
circumstances (did you read the wiki entry?). If your ebuilds
assume that random portage features are enabled, then that's pretty much
undefined behavior.

We can debate whether there are dependency changes not worth a revbump.
We can debate how to reinstate dynamic deps support or how to update the
VDB.

But considering any of that as a blocker to fix a fundamental bug in
dependency calculation, handling of VDB and PMS compatibility is close
to being silly, I'm sorry.

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Tue, 22 Jul 2014 22:05:54 +0300
Samuli Suominen  wrote:
> The quality of the distribution doesn't improve by killing one of the
> most important features the package manager has.

Uh, that's a bit of an odd claim, given that dynamic deps often doesn't
do what you're after anyway...

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-21, o godz. 23:06:07
Pacho Ramos  napisał(a):

> El lun, 21-07-2014 a las 20:55 +0100, Ciaran McCreesh escribió:
> > On Mon, 21 Jul 2014 21:53:04 +0200
> > "Andreas K. Huettel"  wrote:
> > > Revision must be bumped when the on-disk files installed by the
> > > ebuild are changed. 
> > > Nothing about dependencies.
> > > 
> > > This has been policy for a LONG time, and we're not going to change
> > > it overnight just because you protest.
> > 
> > Policy used to be that you'd do a revbump when you wanted users to
> > reinstall stuff, and you wouldn't otherwise. The only complication is
> > that sometimes you want users to reinstall stuff so that there's
> > accurate dependency information available, rather than because
> > something has changed.
> > 
> 
> Maybe this could be solved by having two kinds of revisions:
> - One would rebuild all as usually (for example, -r1...)
> - The other one would only regenerate VDB and wouldn't change the
> installed files (for example, -r1.1)
> 
> But I am not sure if it could be viable from a "technical" point of
> view :(

I'm afraid it couldn't. The major problem is not knowing *when* to
migrate metadata, portage usually gets that right. The problem is in
getting the correct output which is often near to impossible.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, 22 Jul 2014 09:53:49 -0400
Ian Stakenvicius  wrote:

> Using ${PVR} to detect how portage should update things
> would be asking for trouble, imo.

This entire sub thread reads like a dynamic dependencies alternative in
disguise, the difference lies in an increase of the level of control
and in the place where this then gets reimplemented.

The increase of control comes from the maintainer being able to decide
whether the dependencies in the vdb are updated or not; however, this
gives rise to a mindset where you consider this level of control for
other variables as well (which syntax we'll [ab]use for that?) as well
as end up with more ebuilds for the sake of updating vdb dependencies.

Using an extension like -rX.Y seems odd; at the very least, I think an
incremental variable or something along that line in the ebuild would
work better. This allows for array usage like VERSION[dependencies]=1,
thus allowing other variables to be dynamic as well; you compare that
number against the one in the vdb, bingo...

Or is it just a figment?

Please think a design through and don't take a cheap shot with -rX.Y.

- -- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBAgAGBQJTzulZAAoJEPWZc8roOL/QzSMH/0wrGF+6joDtUlmUiNuTZBHB
Y0aYkr+Je7R4jj+NQxMY08j+odyImgnT+IrNrQcs7VEboXkrKS0s7ZEmQqCpNvmp
vVLvGUeAlzPgGz31aKIzMBhe5TuyCTwOvArU+DVcxDEktcbHP+sDBPTojQAr932e
qJtf6fdXDaUklu+MPlNofroREd2hjUrkS34ll6W5E+KizNMfRO7n4SAN38pkkE+C
4t3elp1I2Eei7HQT/cNUY78ab8Sgiy6N5CryN73zx6jyw9EwShLFV/8VN3M9SJ1B
jvBmD01EjsW6FLz6fvUPy2dz4GBKaC4YY0qhK5XocBwROUu2yCGV/w/es1JROB0=
=xuKt
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Tue, 22 Jul 2014 09:25:45 +0200
""Paweł Hajdan, Jr.""  wrote:

> One question: why for "Removal of a USE flag along with the relevant
> dependencies" dynamic deps say "revbump + unnecessary rebuild"? What
> would happen without the revbump?

Assuming dynamic dependencies don't exist, another package would still
depend on the USE flag in the vdb; the only way to force that USE flag
dependency to go away is with an unnecessarily rebuilding rev bump, as
without it it would complain that the USE flag doesn't exist.

(We're talking about RDEPEND="some/pkg[useflag]" --> RDEPEND="some/pkg")


-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Mon, 21 Jul 2014 19:37:17 +
hasufell  wrote:

> afaiu dynamic deps are broken and not defined in PMS

It goes a step further than that!

The PMS imposes certain limits on dependencies; it states that DEPEND
must be present before executing src_* phases, that RDEPEND must be
present before treating the results as usable and that PDEPEND must be
done before finishing the batch of installs.

http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-750008.1

If you attempt to fit in dynamic dependencies in that specification;
the src_* phases could never run, the results can never be considered
usable and the batch of installs can never finish.

> still... people seem to fix deps without revbumping, causing users who
> either don't use dynamic deps (it's optional for portage through
> --dynamic-deps=y, although it's on by default) or who use a different
> PM to not get the fix, at worst resulting in broken dependency
> calculation

Why do we have a PMS if we don't take into account other PMs?

Is Gentoo still a meta distribution? How does the Portage tree portage?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, 21 Jul 2014 16:41:39 -0400
Ian Stakenvicius  wrote:

> I wonder if there may be some form of extension we could add to
> portage, such that it could do a VDB-only "re-emerge" somehow, when
> the in-tree ebuild doesn't match the in-VDB one.  If that could be
> implemented properly (and i'm not sure that it could, tbh), maybe that
> would help reduce issues with dynamic deps, too...

Hmm, dynamic dependencies ... dynamic rebuilds ... dynamic compilation;
one day we will have hot code pushes where upstream changes immediately
reflects itself upon one's system. Does such a gimmick succeed? Meteor!

- -- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBAgAGBQJTzuAsAAoJEPWZc8roOL/QNYsH/1uJrazScukDYnGj3lsPkl7P
5l7l7caugq5CwFfJ+VLZR5byV6MePBff+rJsO6Ch8v4yM+h+IFnOn7pLS27Oqm71
LRaGHeTH/pge3jq9mm53b7ABi2IjuNSKIr69/loYMJaNgHQLZCtR/wFVxKR3XFrA
dhJN7WKhHAG0+1HRlNWCdYpHllG+cmayLARlwfWGbZii/OZWh0eAVEUV0pFdZwlP
WaeOcnLebn+TFWPyKEkGKkmz7yI7fFMaoKBueEAZ9PwEUmhdSK5PEeY2U/OpLOA7
7wOYDe9J1k04q5DwLzZe9LvqjwjYtGIP4sL1/b+9TCw59+akKC+DmnDv67vuTHs=
=wV/h
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Mon, 21 Jul 2014 21:34:10 -0400
Alexandre Rostovtsev  wrote:

> On Mon, 2014-07-21 at 22:56 +0200, Michał Górny wrote:
> > Yes, it does. I'm not sure if it leads anywhere, though. Dynamic
> > deps are a pipe dream. You can't implement them properly, so we're
> > using half-working implementation as an excuse to be lazy.
> 
> Why not adapt the updates mechanism for modifying rdepends? Perhaps
> something like
> 
> rdepends-add foo-bar/blah-3.14 "wombat? ( >=dev-libs/wombat-1.0 )"
> 
> This would give the package manager all the benefits of static dep
> resolution while allowing us to dynamically make simple changes (like
> adding a missing dependency to an ebuild) without forcing users to
> rebuild the package.

Thinking this through:

 1) What about rdepends-change and rdepends-del? If you only support
 addition; you get the same problem as with things like pkgmove,
 changing and/or removing it could become somewhat problematic.

 2) This needs two commits every time you want to do this; one commit
 for the updates/, another to keep the ebuild recent for (rev)bumps.

 3) It'll be a lot of fun to attempt to support this in Repoman.

 4) How do we clean up these entries? Doesn't this info grow fast?

 5) The first paramater: Should that point to a single ebuild? Should
 that support ranges?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Tue, 22 Jul 2014 23:11:37 +0200
Michał Górny  wrote:

> [citation needed].
> 
> In other words, please support such claims with evidence. Because
> honestly I didn't see very much people complaining about unnecessary
> rebuilds, except in this specific thread.
> 
> But I guess they're indeed a larger issue than, for example, portage
> forcing wrong branches of || dependencies or other dependency
> calculation errors that result in people being unable to update their
> systems. But I don't really visit Forums often.

[citation needed].

Actually, please try to go a step further and reproduce it; surprise!

> If you ever happen to change EAPI in a stable ebuild without revbump,
> please be kind enough to revoke your own commit rights. Thanks.

So much revocations. Thanks.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Tue, 22 Jul 2014 22:05:54 +0300
Samuli Suominen  wrote:

> Not before someone has implemented an alternative way to avoid useless
> rebuilding.
> The quality of the distribution doesn't improve by killing one of the
> most important
> features the package manager has.
> The quality of the distribution improves by providing an alternative
> with less problems.

Those are apples with eggs; what you claim to be most important in
itself also has problems as stated in this thread, it doesn't stop there
as even beyond that it is a hack that deals with triggered problems. So,
I don't think it is right to discuss replacing problems with problems.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Tue, 22 Jul 2014 23:11:37 +0200
Michał Górny  wrote:
> But I guess they're indeed a larger issue than, for example, portage
> forcing wrong branches of || dependencies or other dependency
> calculation errors that result in people being unable to update their
> systems. But I don't really visit Forums often.

Users are so used to broken dependency resolution that they don't
complain about it very much, and even when they do, they're rarely able
to identify the actual cause of the problem.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, 22 Jul 2014 20:50:55 +0200
Alexander Berntsen  wrote:

> On 22/07/14 20:44, Kent Fredric wrote:
> > So we'll probably need a repoman check that is smart enough to know
> >  "X is modified" and compare the DEPEND fields with the previous 
> > incarnation prior to commit, and then at very least, warn people 
> > doing `repoman full` that they've modified the dependencies, and 
> > that a -r1 bump is thus highly recommended.
> > 
> > And that check can be added *now* prior to banning/disabling
> > dynamic deps.
> > 
> > And people who want to pay attention to that warning can start
> > doing it before policy dictates they must.
> This would be a good thing to do. Would you like to try implementing
> it?

Just a side note, this would need VCS diff support in Repoman; there
are some other feature requests on b.g.o that are also pending this
VCS diff support, but it takes some understanding of the Repoman code.

This becomes easier to implement after the refactor of Repoman
completes; checks are organized and consolidated into several files
due to the refactor, rather than mixed together in a giant single file.

Having repoman help with the developer workflow sounds like a neat idea.

- -- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBAgAGBQJTzvOZAAoJEPWZc8roOL/QVcoH/0JhGPYQg5yPYDafriSIxpD+
ydS0jwCZudHiX7YZk/L5XiVLKTiwEPNLUzEmmYtkCnPXtJAZTFUYlTmn9sY9GUEh
dV9Gr6gG5LpwjGDF2E9F5ivkSdUqGbx8ztqKibSAm4POy+uLm7EDEBtRJe095VVo
k/kKukGSpa9hlnLB43hP9u1rs0vuwx0YB6FwK+dRVxGJAyPipgxg1jt6uRqOO9+a
sBOzc910js8rpO1bkL4vGbrRViVUoFFOylWrXDxEuuyoaAWROZbItqe4xK2XagI3
wYJa/aLzGO9b3oTDQPVSEdIpgqfuJCI39BH4UrlAUZgT4GIuB8VroOx9MAOtHZc=
=jVQd
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Mon, 21 Jul 2014 17:52:51 -0500
William Hubbs  wrote:

> My concern about doing a revbump just because the deps change is that
> the new revision has to be committed in ~arch, so we then have to hit
> the arch teams, which we know are overworked anyway, with stable
> requests just because we changed the dependencies. Isn't that causing
> a lot of possibly unnecessary work for our arch teams?

Rev bumps that only add dependencies shouldn't need separate STABLEREQs.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Mon, 21 Jul 2014 23:01:58 +0200
Jeroen Roovers  wrote:

> So you suggest we work around a bug in the PM which would be a single
> fix. Everywhere.

Which bugs? Which fixes? Where? ... Did this thread spawn from nothing?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Mon, 21 Jul 2014 22:42:23 +0300
Samuli Suominen  wrote:

> Revision bumping for dependency change that doesn't cause the
> package's file content
> to change doesn't make sense; triggers useless rebuilds for users.

A merged ebuild that misses a dependency needs an useless extra emerge.
Think about the triggers instead of the extra rebuilds or extra emerges.
 
> Portage is the official package manager, and has dynamic deps enabled
> by default.

Is it a feature or is it a hack?

> And it's already in ebuild-quiz.txt to ensure knowing when to, or not
> to, revbump:
> 
> *** Ebuild technical/policy questions
> 
> 1.You change a package's ebuild to install an init script.
> Previously, the package had no init script at all.
>   Is a revision bump necessary? Why? What about when adding a
> patch?

That's not about dynamic dependencies; but yes, too much rev bumps.

> So, -1, useless rebuilds is one of the biggest problems lately, it's
> an relatively new problem,
> people are revbumping packages for the simplest things like EAPI4->5

Useless triggers are the problem; why are the rev bumps needed, why are
dependencies forgotten, ...? Sounds like a developer work flow issue...

https://bugs.gentoo.org/show_bug.cgi?id=499852

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Mon, 21 Jul 2014 21:53:04 +0200
"Andreas K. Huettel"  wrote:

> Actually the quizzes are pretty much clear on that. 
> 
> Revision must be bumped when the on-disk files installed by the
> ebuild are changed. 
> Nothing about dependencies.
> 
> This has been policy for a LONG time, and we're not going to change
> it overnight just because you protest.

As suggested, later PMS features (eg. (+) / (-), ...) and PM features
(eg. binpkgs, ...) can change that; so, a policy adaption or a clearly
described workflow may at some point be necessary to stay actual. 

> Now... whether dynamic deps are technically the right thing to do is
> another question. It merits discussion, but we need to be really sure
> about the consequences of any change.

To deal with that one needs a clear workflow; given that it replaces an
automatic feature or hack without much rebuilds, by something that
needs you to decide whether or not to rev bump and cause more rebuilds.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Tom Wijsman]

On Tue, 22 Jul 2014 08:10:20 +0300
Samuli Suominen  wrote:

> On 22/07/14 04:05, Rick "Zero_Chaos" Farina wrote:
> >
> > And just for fun, since no one has mentioned it yet, dynamic deps
> > don't work at all on binpkgs since the Packages file contains the
> > deps (like vardb) and it doesn't get updated (just like vardb).
> 
> Known long standing pitfall. It's managable.

It is one of the reasons I see binpkgs as breaking progress instead of
being part of the progress; it is manageable, but it could be better.

> > Revbump or make dynamic deps actually work (ha).
> 
> If they are really as broken people claim, why is the feature enabled
> by default in the
> official package manager?

Why are these discussions resounding more strongly with each occurrence?

> As in, why I'm not seeing a bug with title "sys-apps/portage: disable
> dynamic deps by default"

https://bugs.gentoo.org/show_bug.cgi?id=516612

> with a Comment #0 listing all the culprits why it should be disabled
> by default?

They've been gathered throughout this thread; yes, they could use a
summary, but that doesn't change that all the culprits exist out there.

> Or do people think it works well enough, so that it's pros overweight
> the cons? I do, 
> and many others seem to think so as well.

Depends on what the pros and cons are; it makes the difference between
merely thoughts and an actual reality, to construct towards a decision.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-21, o godz. 22:42:23
Samuli Suominen  napisał(a):

> So, -1, useless rebuilds is one of the biggest problems lately,

[citation needed].

In other words, please support such claims with evidence. Because
honestly I didn't see very much people complaining about unnecessary
rebuilds, except in this specific thread.

But I guess they're indeed a larger issue than, for example, portage
forcing wrong branches of || dependencies or other dependency
calculation errors that result in people being unable to update their
systems. But I don't really visit Forums often.

> it's an
> relatively new problem,
> people are revbumping packages for the simplest things like EAPI4->5

If you ever happen to change EAPI in a stable ebuild without revbump,
please be kind enough to revoke your own commit rights. Thanks.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[William Hubbs]

On Tue, Jul 22, 2014 at 11:42:30AM +0200, Alexander Berntsen wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> On 22/07/14 02:36, hasufell wrote:
> > William Hubbs:
> >> My concern about doing a revbump just because the deps change is
> >> that the new revision has to be committed in ~arch, so we then
> >> have to hit the arch teams, which we know are overworked anyway,
> >> with stable requests just because we changed the dependencies.
> >> Isn't that causing a lot of possibly unnecessary work for our
> >> arch teams?
> > Procedure over logic?
> > 
> > Just commit it straight to arch if repoman doesn't complain.
> William,
> 
> this is, as Julian pointed out, a problem you can solve by changing
> your policies. This is not a problem related to the Portage software,
> in which dynamic-deps are broken.

s/your/our/

Repoman refuses to commit if you try to go directly to stable without
using --force.

I'm just being cautious; I'm not sure whether this qualifies as the type
of emergency situation where commiting directly to stable is a good
thing or not.

William



signature.asc
Description: Digital signature

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-22, o godz. 09:25:45
""Paweł Hajdan, Jr.""  napisał(a):

> On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
> > Michał has documented the shortcomings of dynamic deps in our wiki[0].
> > (Thank you!) This documentation also includes two of our possible
> > solutions.
> >
> > [0]    
> 
> Thank you, this is very useful. I didn't understand the issue much
> before reading that page.
> 
> One question: why for "Removal of a USE flag along with the relevant
> dependencies" dynamic deps say "revbump + unnecessary rebuild"? What
> would happen without the revbump?

Well, for completeness I'll start by listing what would happen with
static deps. Though nothing will actually happen. Already-built
packages may have the flag enabled along with deps. When people rebuild
-- through --newuse, --changed-use or otherwise -- they will get
the functionality and dependencies removed along with the flag.

How dynamic-deps would handle that are pretty much a matter of
implementation choice. I can think of two major possibilities:

1. dynamic-deps are applied nevertheless. Since the relevant
dependencies are removed along with the flag, dynamic-deps causes
portage to no longer pull in needed libraries even though package was
built with the flag enabled and still links to them. Long story short,
linkage is broken.

2. PM notices IUSE no longer matches and doesn't apply dynamic-deps.
While this prevents the breakage from happening, it's one additional
point to the list of cases when dynamic-deps are disabled. Which means
that no further dependency changes to the ebuild have dynamic effect
and -- with current portage implementation -- the dependencies are
'reverted' to the state when the package was installed, even though
just before the change portage used ebuild dependencies.

Of course, you could try to invent some smart logic that would handle
this specific case better. But it makes the dependency resolution even
more complex and hard to understand, plus someone needs to actually do
it. And then, you're going to hit even more corner cases and implement
additional workarounds for each one. And then each of those workarounds
will create new corner cases...

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Samuli Suominen]

On 22/07/14 10:25, "Paweł Hajdan, Jr." wrote:
> On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
>> 2. Remove dynamic-deps. This is what I think currently makes sense.
> +1 I also think it's the best option.
>
>

Not before someone has implemented an alternative way to avoid useless
rebuilding.
The quality of the distribution doesn't improve by killing one of the
most important
features the package manager has.
The quality of the distribution improves by providing an alternative
with less problems.

Sounds like to me, that the people who want to remove the feature so
badly, are the
ones volunteering for the job as well.

- Samuli

Re: [gentoo-dev] don't rely on dynamic deps

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 22/07/14 20:44, Kent Fredric wrote:
> So we'll probably need a repoman check that is smart enough to know
>  "X is modified" and compare the DEPEND fields with the previous 
> incarnation prior to commit, and then at very least, warn people 
> doing `repoman full` that they've modified the dependencies, and 
> that a -r1 bump is thus highly recommended.
> 
> And that check can be added *now* prior to banning/disabling
> dynamic deps.
> 
> And people who want to pay attention to that warning can start
> doing it before policy dictates they must.
This would be a good thing to do. Would you like to try implementing it?
- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlPOso8ACgkQRtClrXBQc7V7BgEAgElHG5RcxpJWS2eTQ7YVFdDX
NZ55itqGeMngocAoitUA+QF4N0w07NrQSpPecPaF5AeuLOspGI7xjfaU/2BNFLGO
=1H0R
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Kent Fredric]

On 22 July 2014 19:25, "Paweł Hajdan, Jr."  wrote:

> On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
> > Michał has documented the shortcomings of dynamic deps in our wiki[0].
> > (Thank you!) This documentation also includes two of our possible
> > solutions.
> >
> > [0]  
>
> Thank you, this is very useful. I didn't understand the issue much
> before reading that page.
>
> One question: why for "Removal of a USE flag along with the relevant
> dependencies" dynamic deps say "revbump + unnecessary rebuild"? What
> would happen without the revbump?
>
> > 1. Improve dynamic-deps. This is, as Michał pointed out earlier in
> > this thread a pipe dream.
>
> Agreed.
>
> > 2. Remove dynamic-deps. This is what I think currently makes sense.
>
> +1 I also think it's the best option.
>
> Paweł
>
>
Ok, we can side step this discussion partially:

Lets pretend for a moment dynamic deps get banned.

People will still unconsciously make that mistake and things will still
break when they do.

So we'll probably need a repoman check that is smart enough to know "X is
modified" and compare the DEPEND fields with the previous incarnation prior
to commit, and then at very least, warn people doing `repoman full` that
they've modified the dependencies, and that a -r1 bump is thus highly
recommended.

And that check can be added *now* prior to banning/disabling dynamic deps.

And people who want to pay attention to that warning can start doing it
before policy dictates they must.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] don't rely on dynamic deps

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 21/07/14 05:06 PM, Pacho Ramos wrote:
> El lun, 21-07-2014 a las 20:55 +0100, Ciaran McCreesh escribió:
>> On Mon, 21 Jul 2014 21:53:04 +0200 "Andreas K. Huettel"
>>  wrote:
>>> Revision must be bumped when the on-disk files installed by
>>> the ebuild are changed. Nothing about dependencies.
>>> 
>>> This has been policy for a LONG time, and we're not going to
>>> change it overnight just because you protest.
>> 
>> Policy used to be that you'd do a revbump when you wanted users
>> to reinstall stuff, and you wouldn't otherwise. The only
>> complication is that sometimes you want users to reinstall stuff
>> so that there's accurate dependency information available, rather
>> than because something has changed.
>> 
> 
> Maybe this could be solved by having two kinds of revisions: - One
> would rebuild all as usually (for example, -r1...) - The other one
> would only regenerate VDB and wouldn't change the installed files
> (for example, -r1.1)
> 
> But I am not sure if it could be viable from a "technical" point
> of view :(
> 
> 

eww, no.  Using ${PVR} to detect how portage should update things
would be asking for trouble, imo.  Besides, I don't think detection of
when to just update VDB is the issue.  The main issue that I see is
- -how- VDB should be adjusted based on what changes are made to the
ebuilds.  For instance, if minimum versions of deps are adjusted
in-place, should vdb be updated to match?  what happens if the minimum
version of the currently-installed dep is below the new one?  etc. etc.

Also, in theory an EAPI bump with nothing else changing should be
re-generatable in the VDB, but i have a gut feeling (no evidence, just
a feeling) that going from say, EAPI2 to EAPI5 without doing some of
the phase functions again (ie 'merge', maybe there are others that can
affect VDB?) will result in a different VDB from a regular rebuild.


-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlPObO0ACgkQ2ugaI38ACPCerQEAgTgQOvCDl0dbB5sOOZ4diBNs
cheQR18XFo7MsVBX3uUA/0zP1cAiWy1zAF+crrfCC42krtvGmSSiU4JG0dFo4452
=iNmo
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[hasufell]

Alexander Berntsen:
> 
> Julian,
> 
> would you like to share your experiences with Paludis? My guess is
> that Paludis is more predictable in this respect. I.e., instead of
> breaking stuff, I expect Paludis to simply give up.
> 

Relying on dynamic deps as they are currently implemented simply causes
the vdb to enter a broken state after some time.

If you switch from portage to paludis then, you will encounter a lot of
"unsuitable candidates" messages during dependency resolving and will
have to figure out that the broken vdb is the reason. Afais paludis does
not allow you to randomly break the vdb (or I haven't found the --nodeps
option yet).

Re: [gentoo-dev] don't rely on dynamic deps

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 22/07/14 02:36, hasufell wrote:
> William Hubbs:
>> My concern about doing a revbump just because the deps change is
>> that the new revision has to be committed in ~arch, so we then
>> have to hit the arch teams, which we know are overworked anyway,
>> with stable requests just because we changed the dependencies.
>> Isn't that causing a lot of possibly unnecessary work for our
>> arch teams?
> Procedure over logic?
> 
> Just commit it straight to arch if repoman doesn't complain.
William,

this is, as Julian pointed out, a problem you can solve by changing
your policies. This is not a problem related to the Portage software,
in which dynamic-deps are broken.
- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlPOMgYACgkQRtClrXBQc7VuuwD/UNQzX5aHSBbfXhyYRxH4oYzK
N9aEf88WLVJK2JVKJBkA/iDF6ozQ9I0WKWpi/jvZa/W7yxKeZsXu5ACa5mbgM88+
=9/RG
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Paweł Hajdan, Jr.]

On 7/21/14, 11:52 PM, Alexander Berntsen wrote:
> Michał has documented the shortcomings of dynamic deps in our wiki[0].
> (Thank you!) This documentation also includes two of our possible
> solutions.
>
> [0]  

Thank you, this is very useful. I didn't understand the issue much
before reading that page.

One question: why for "Removal of a USE flag along with the relevant
dependencies" dynamic deps say "revbump + unnecessary rebuild"? What
would happen without the revbump?

> 1. Improve dynamic-deps. This is, as Michał pointed out earlier in
> this thread a pipe dream.

Agreed.

> 2. Remove dynamic-deps. This is what I think currently makes sense.

+1 I also think it's the best option.

Paweł



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] don't rely on dynamic deps

[Samuli Suominen]

On 22/07/14 04:05, Rick "Zero_Chaos" Farina wrote:
>
> And just for fun, since no one has mentioned it yet, dynamic deps don't
> work at all on binpkgs since the Packages file contains the deps (like
> vardb) and it doesn't get updated (just like vardb).

Known long standing pitfall. It's managable.

>
> Revbump or make dynamic deps actually work (ha).
>

If they are really as broken people claim, why is the feature enabled by
default in the
official package manager?
As in, why I'm not seeing a bug with title "sys-apps/portage: disable
dynamic deps by default"
with a Comment #0 listing all the culprits why it should be disabled by
default?
Or do people think it works well enough, so that it's pros overweight
the cons? I do,
and many others seem to think so as well.

Re: [gentoo-dev] don't rely on dynamic deps

[Alexandre Rostovtsev]

On Mon, 2014-07-21 at 22:56 +0200, Michał Górny wrote:
> Yes, it does. I'm not sure if it leads anywhere, though. Dynamic deps
> are a pipe dream. You can't implement them properly, so we're using
> half-working implementation as an excuse to be lazy.

Why not adapt the updates mechanism for modifying rdepends? Perhaps
something like

rdepends-add foo-bar/blah-3.14 "wombat? ( >=dev-libs/wombat-1.0 )"

This would give the package manager all the benefits of static dep
resolution while allowing us to dynamically make simple changes (like
adding a missing dependency to an ebuild) without forcing users to
rebuild the package.

Re: [gentoo-dev] don't rely on dynamic deps

[Rick "Zero_Chaos" Farina]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/21/2014 06:52 PM, William Hubbs wrote:
> I'm picking a random msg to reply to.
> 
> My concern about doing a revbump just because the deps change is that
> the new revision has to be committed in ~arch, so we then have to hit
> the arch teams, which we know are overworked anyway, with stable
> requests just because we changed the dependencies. Isn't that causing a
> lot of possibly unnecessary work for our arch teams?

If you edit an ebuild in place, and that ebuild has stable keywords, how
is that any different than revbumping it and keeping it with stable
keywords? The two are functionally identical, people get a changed
ebuild, and the arch teams do nothing. The only difference is that 1000
different ways dynamic deps are broken.

And just for fun, since no one has mentioned it yet, dynamic deps don't
work at all on binpkgs since the Packages file contains the deps (like
vardb) and it doesn't get updated (just like vardb).

Revbump or make dynamic deps actually work (ha).

- -Zero
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTzbjHAAoJEKXdFCfdEflKVL4P/2Sb88YYcbHJZ1pfnVukxvvi
zYD00knmGiQOcmSxZMPnhVXjV+Z1SzyOPtmZa5KXLd2xwmVvNwJdUV0ssjfDi6Gg
fxnAEWWuZ6xEvpIlm7WIXq5VFGL/AO4HUso6mHqSVZbbgYLPiFCooSAn291Epmjn
vGIkUf5pNtRp6gLQFTqM2ksDzJWaOF9bmuapcCSumv3uhKK2Empy+6kHTKYVd/m5
Mvo10p7W8amozV+pl7Si7tWGOf/U+Xo3N0gt9VJzz8oxIpooXZ+nSB3p9bXSbKZN
sspz9khHLKFyhOAfoBbtLB9cqwPfPqgMlt6h4n9l2uJ+E0nMz0vRIlQnMarKD2FE
/3DG6zrvWU0eRuYM7c5iqOrlL1WYgAA3p6CEBo5U9+h+8eF2bgxAJt/c7txJG0Ws
EpSlwOIFyfVOWaJeb1WpBt72sMH+URF/YRl6K0ZKmp7+X5Ga/39ZznRfhRblRp9h
nBMyIer9Ai7wNbc7A1qMvjedqOknOwSJjKEDkaUAIn5RTFa5VmLAGURSeuQnIvjE
ogel9ENjdoZJW0i8bEO8ArCS9X5vgdWiwZHcjZL62KNYoqc6ySrsTUm0Eo3vDevo
wr782AzzLPVftPXNHMnEthBs5ztbLr6YlXv/V5jfCc1YRQGUeEMjBkny/Nxm1VLi
/l1lbtaTcm5FcqSjen4Q
=asgW
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[hasufell]

William Hubbs:
> I'm picking a random msg to reply to.
> 
> My concern about doing a revbump just because the deps change is that
> the new revision has to be committed in ~arch, so we then have to hit
> the arch teams, which we know are overworked anyway, with stable
> requests just because we changed the dependencies. Isn't that causing a
> lot of possibly unnecessary work for our arch teams?
> 

Procedure over logic?

Just commit it straight to arch if repoman doesn't complain.

Re: [gentoo-dev] don't rely on dynamic deps

[William Hubbs]

I'm picking a random msg to reply to.

My concern about doing a revbump just because the deps change is that
the new revision has to be committed in ~arch, so we then have to hit
the arch teams, which we know are overworked anyway, with stable
requests just because we changed the dependencies. Isn't that causing a
lot of possibly unnecessary work for our arch teams?

William



signature.asc
Description: Digital signature

Re: [gentoo-dev] don't rely on dynamic deps

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Friends,

Michał has documented the shortcomings of dynamic deps in our wiki[0].
(Thank you!) This documentation also includes two of our possible
solutions.

1. Improve dynamic-deps. This is, as Michał pointed out earlier in
this thread a pipe dream.

2. Remove dynamic-deps. This is what I think currently makes sense.

3. This is undocumented in the Wiki, but it is certainly an option: do
nothing. It is of course the worst option; but it is perhaps the most
probably course of action as well...


dynamic-deps do not work, and nobody is stepping up to fix them. PMS
defines a static dependency model, and this is what other package
managers use as far as I know.


Julian,

would you like to share your experiences with Paludis? My guess is
that Paludis is more predictable in this respect. I.e., instead of
breaking stuff, I expect Paludis to simply give up.


Sebastian,

I CC'd you because I would love to hear your opinion on this.


Trofi,

please share your opinion too!



To sum up: My vote is disable dynamic-deps. And I would be happy to
apply a patch that does this with the information I have today.


[0]  
- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlPNi7oACgkQRtClrXBQc7UP6gEAnnWR7L7hDqvaL64ygDwqaBV/
4upsbo6z2FJK4BehajgA/30wolmft/L9vTR/RzH9Wlsu6+NoUBTBMeJGNuIdBCIl
=++4v
-END PGP SIGNATURE-

Re: [gentoo-dev] don't rely on dynamic deps

[Maxim Kammerer]

On Tue, Jul 22, 2014 at 12:25 AM, Michał Górny  wrote:
> The funny thing is, almost none of the Gentoo developers even know that
> slot operators disable dynamic dependencies completely in portage.

So *that's* why I now have to change RDEPENDs in both the source
ebuild and in VDB in order to augment package's dependencies before
depclean...

-- 
Maxim Kammerer
Liberté Linux: http://dee.su/liberte

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-21, o godz. 22:21:42
Ciaran McCreesh  napisał(a):

> On Mon, 21 Jul 2014 23:15:41 +0200
> Jeroen Roovers  wrote:
> > On Mon, 21 Jul 2014 23:06:07 +0200
> > Pacho Ramos  wrote:
> > > Maybe this could be solved by having two kinds of revisions:
> > > - One would rebuild all as usually (for example, -r1...)
> > > - The other one would only regenerate VDB and wouldn't change the
> > > installed files (for example, -r1.1)
> > 
> > Or the package manager looks at changed in *DEPEND between the repo
> > and vdb and resolves those.
> 
> ...assuming that the ebuild hasn't been removed, and that it can be
> associated correctly when overlays are involved, and that the change
> wasn't a change where a saved pkg_prerm uses the old dependency, not
> the new one, or all the other ways this breaks.

You forgot about slot operators.

The funny thing is, almost none of the Gentoo developers even know that
slot operators disable dynamic dependencies completely in portage.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Michał Górny]

Dnia 2014-07-22, o godz. 00:13:13
Samuli Suominen  napisał(a):

> 
> On 21/07/14 23:56, Michał Górny wrote:
> > Now... whether dynamic deps are technically the right thing to do is 
> > another 
> > question. It merits discussion, but we need to be really sure about the 
> > consequences of any change.
> > Yes, it does. I'm not sure if it leads anywhere, though. Dynamic deps
> > are a pipe dream. You can't implement them properly, so we're using
> > half-working implementation as an excuse to be lazy.
> 
> What's lazy is maintainer doing revision bump without thinking
> if it's really required, spreading his laziness upon every users
> machine (by triggering revision bump driven rebuild)

Yes, users much more prefer random breakage over time. And debugging
the issues save us a lot of time!

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Mon, 21 Jul 2014 23:15:41 +0200
Jeroen Roovers  wrote:
> On Mon, 21 Jul 2014 23:06:07 +0200
> Pacho Ramos  wrote:
> > Maybe this could be solved by having two kinds of revisions:
> > - One would rebuild all as usually (for example, -r1...)
> > - The other one would only regenerate VDB and wouldn't change the
> > installed files (for example, -r1.1)
> 
> Or the package manager looks at changed in *DEPEND between the repo
> and vdb and resolves those.

...assuming that the ebuild hasn't been removed, and that it can be
associated correctly when overlays are involved, and that the change
wasn't a change where a saved pkg_prerm uses the old dependency, not
the new one, or all the other ways this breaks.

You need to think your cunning plan the whole way through.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Samuli Suominen]

On 21/07/14 23:56, Michał Górny wrote:
> Now... whether dynamic deps are technically the right thing to do is another 
> question. It merits discussion, but we need to be really sure about the 
> consequences of any change.
> Yes, it does. I'm not sure if it leads anywhere, though. Dynamic deps
> are a pipe dream. You can't implement them properly, so we're using
> half-working implementation as an excuse to be lazy.

What's lazy is maintainer doing revision bump without thinking
if it's really required, spreading his laziness upon every users
machine (by triggering revision bump driven rebuild)

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Mon, 21 Jul 2014 23:13:06 +0200
Jeroen Roovers  wrote:
> On Mon, 21 Jul 2014 22:06:08 +0100
> Ciaran McCreesh  wrote:
> > On Mon, 21 Jul 2014 23:01:58 +0200
> > Jeroen Roovers  wrote:
> > > So you suggest we work around a bug in the PM which would be a
> > > single fix. Everywhere.
> > 
> > Dynamic dependencies is not fixable. It's an irredeemably broken
> > concept.
> 
> It works fine for me.

You should read the relevant bugs before making such a bold claim...
You're illustrating why we have the problem to begin with: you're
confusing "I've not noticed it go wrong, or realised that breakages
I've seen are because of this" with "works fine".

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] don't rely on dynamic deps

[Jeroen Roovers]

On Mon, 21 Jul 2014 23:06:07 +0200
Pacho Ramos  wrote:

> Maybe this could be solved by having two kinds of revisions:
> - One would rebuild all as usually (for example, -r1...)
> - The other one would only regenerate VDB and wouldn't change the
> installed files (for example, -r1.1)

Or the package manager looks at changed in *DEPEND between the repo
and vdb and resolves those. Oh wait, portage already does. And we've
been talking about this for something like six years now? Probably
longer.


 jer

Re: [gentoo-dev] don't rely on dynamic deps

[Jeroen Roovers]

On Mon, 21 Jul 2014 22:06:08 +0100
Ciaran McCreesh  wrote:

> On Mon, 21 Jul 2014 23:01:58 +0200
> Jeroen Roovers  wrote:
> > So you suggest we work around a bug in the PM which would be a
> > single fix. Everywhere.
> 
> Dynamic dependencies is not fixable. It's an irredeemably broken
> concept.

It works fine for me.


 jer

Re: [gentoo-dev] don't rely on dynamic deps

[Ciaran McCreesh]

On Mon, 21 Jul 2014 23:01:58 +0200
Jeroen Roovers  wrote:
> So you suggest we work around a bug in the PM which would be a single
> fix. Everywhere.

Dynamic dependencies is not fixable. It's an irredeemably broken
concept.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature