[gentoo-dev] Re: Things one could be upset about

[Duncan]

Róbert Čerňanský posted on Tue, 20 Jan 2015 06:51:01 +0100 as excerpted:

> On Mon, 19 Jan 2015 20:51:31 + Ciaran McCreesh
>  wrote:
> 
>> On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský
>>  wrote:
>> > From my point of view it would do much help if portage resolves USE
>> > dependencies automatically instead of telling the user to change USE
>> > flags manually (I am talking about bug #258371).
>> 
>> This is only possible in carefully selected circumstances, and to get
>> it to work more generally would require a lot of hinting from package
>> maintainers.
> 
> But portage already knows that.  It tells the user which USE flags needs
> to be changed in order to emerge a package.  It should just go one step
> further - to make the proposed change happen by itself.

Actually, current portage (2.2.15 is what I have installed here ATM) does 
exactly that, making changes to the appropriate package.* files as 
necessary, mediated only by the usual CONFIG_PROTECT variables.

Since /etc/portage is CONFIG_PROTECTed by default, these changes normally 
first appear in that feature's .* files, to be merged by the admin using 
etc_update or whatever, but if you CONFIG_PROTECT_MASK /etc/portage, 
portage will (I believe, I've never been stupid enough to try it) happily 
write those changes without admin mediation.

And a number of users, including me, objected and still don't like that 
feature, tho with the CONFIG_PROTECT mediation it's at least tolerable.  
As others in-thread have stated, we don't believe that's something 
portage should be messing with.

In particular, I have a specific scheme for my package.* directories and 
the files under them, and portage always wants to write the changes to 
the wrong one... based on my experience so far, apparently the last 
existing file in the appropriate package.* directory, when it should go 
in some other file.

Picking an example out of the air, I have USE=pdf set globally, with
USE=-pdf set for specific packages in a file named after the flag (not 
the package it's set for) and to reflect the fact that I'm disabling it:

/etc/portage/package.use/0-pdf

But if portage were to need to set USE=-pdf on another package, instead 
of writing the changes to apply to that file, it would write the changes 
to apply to...

/etc/portage/package.use/zzpkg-ncmpc

... which is there because ncmpc has several very specific USE flags that 
will never apply to anything else and are best grouped together, and 
which shouldn't contain a USE=-pdf setting for ANY package, INCLUDING 
ncmpc, because that's a more generic USE flag, that belongs in the 
previously mentioned /etc/portage/package.use/0-pdf file.

Of course in practice, no automated ruleset is ever going to make sense 
of the way I have things organized, and I don't expect it to.  Which is 
my point.  Portage shouldn't be editing those files AT ALL.  It should 
spit out the suggested change, and let me edit what I, as the *ADMIN*, 
consider the appropriate file to make that change, assuming of course 
that I actually agree with the changed flag in the first place, and don't 
want to make some other adjustment that I consider more appropriate to 
the specific situation.

At least with CONFIG_PROTECT, tho, portage doesn't just edit the file if 
I accidentally hit an enter when I run my usual post-sync
emerge --ask --update --newuse ... @world.  It simply creates a dot-file 
which I have to delete later, after I've decided the appropriate action 
and taken it.  Which is why I characterize it as at least tolerable.  But 
were that feature to have never been merged, I'd be /much/ happier.


Meanwhile, if you're not seeing that feature yet, then you're not running 
current portage, maybe because you're on stale^h^hble portage or 
something.  But it's certainly there, to my unhappiness if toleration 
because at /least/ it obeys CONFIG_PROTECT.

So this subthread could go away... perhaps to be replaced with one 
griping about portage trying to mess with its own config and screwing up 
badly, with only CONFIG_PROTECT stopping it from /really/ screwing up an 
admin's painstaking portage config.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: Moving CPU flags into USE_EXPAND

[Duncan]

Andrew Savchenko posted on Tue, 20 Jan 2015 23:59:23 +0300 as excerpted:

> On Tue, 20 Jan 2015 12:17:35 -0800 Christopher Head wrote:
>> On January 20, 2015 12:47:03 AM PST, Alexis Ballier
>>  wrote:
>> >So, you're telling me that if you have a list of 90 cpu extensions,
>> >you will from time to time open that list to see if there is a 91st
>> >one added ? I think most people won't even notice, at best they'll
>> >look for the changelog.
>> 
>> No, actually, I’m advocating the exact opposite. I’m saying that, as
>> long as the list file is kept up to date, then I will look at those 90
>> flags when I first install and never again. If a 91st flag appears some
>> day, then as long as the file was maintained as I described in an
>> earlier message (i.e. flags are added as soon as manufacturers announce
>> features), I already know I can reliably ignore the new flag. After
>> all,
>> if the flag didn’t exist when I installed the system, then my CPU must
>> necessarily not have that feature—unless CPUs are in the habit of
>> sprouting new instructions after you buy them!
> 
> Not exactly. CPUs are not in a habit, but software is. Some brand new
> instuction set may be supported in (any of) packages with some delay.
> Thus it is possible that instruction set supported by your CPU will
> appear in the list of cpu flags after your ininial install.

PMFJI...

chead's idea is (I believe) simply to have the description file updated 
with all current hardware feature flags as soon as they are known (he 
said announced, but sometimes they change between announcement and 
actually appearing in hardware, so "known", as in "known to actually 
appear in hardware", would seem to be better).

That way, no matter what the software supported at the time and what 
flags were thus actually used in packages, when someone first installs 
gentoo on a new machine, or when they first upgrade their CPU to 
something with new features, they can run the script and update their 
use_expand to match their hardware _ONCE_, without worrying about whether 
or when a package with support for it might appear.

If no package with that support ever appears, no harm done, that entry in 
the use_expand is simply never used.

OTOH when some package /does/ get support for new hardware instructions 
and adds the appropriate flag, it'll appear in portage's output, but 
because the use_expand was already set when gentoo was installed or the 
cpu upgraded, the user won't have to worry about needing to look it up 
and decide whether to set it, again, it'll already be done, back when the 
_hardware_ changed, _not_ sometime later, when the _software_ changed to 
support the new hardware.

Of course if the user upgrades hardware after a package supports a 
feature, they'll have to upgrade their use_expand setting appropriately 
or miss support for the new instructions, but that's always the case.  
Just if handled as chead suggests, it'd be the case ONLY when the 
hardware is updated, instead of every time a package upgrades its own 
support.

Correct, chead?  Does that make things clearer, aballier and bircoph?

=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Moving CPU flags into USE_EXPAND

[Gordon Pettey]

On Tue, Jan 20, 2015 at 2:17 PM, Christopher Head  wrote:

> On January 20, 2015 12:47:03 AM PST, Alexis Ballier 
> wrote:
> >So, you're telling me that if you have a list of 90 cpu extensions, you
> >will from time to time open that list to see if there is a 91st one
> >added ? I think most people won't even notice, at best they'll look for
> >the changelog.
>
> No, actually, I’m advocating the exact opposite. I’m saying that, as long
> as the list file is kept up to date, then I will look at those 90 flags


Why do you keep insisting on 90 flags? Nothing is going to have alternate
code paths for whether you've got pse, pse36, pat, fpu, or ept. The vast
majority of the flags in cpuinfo aren't ever going to be in anything
userspace. The kernel might care about them, but even then, most of them
aren't going to be related to any options you can change.

Re: [gentoo-dev] keywording gcc-4.9 branch

[Luca Barbato]

On 17/01/15 18:43, Anthony G. Basile wrote:

Hi everyone,

It looks like there aren't too many bugs against gcc-4.9 (bug #495124).
There are a couple having to do with lto and one which is hardened
specific, but its probably a good idea to start keywording on the
various arches so we can get more bug reports and not trail behind.  So
I opened bug #536874.

I've followed Ryan's workflow where he has one bug for gcc 4.9 porting
and another for keywording/stabilization.  I'm not sure why he did it
this way, but it may be less confusing in the future to have just one
which track the state of the ebuild from keyword masked, to keyworded to
stabilization.

@rhill, any advice here :)




Word of warning, asan was still terminally broken on nontrivial code 
last time I tested it.


lu

[gentoo-dev] Re: Review: news item and script for CPU_FLAGS_X86

[Duncan]

Michał Górny posted on Tue, 20 Jan 2015 10:40:17 +0100 as excerpted:

> Display-If-Keyword: amd64 ~amd64 x86 ~x86
> 
> The USE flags corresponding to intruction sets and other features
> specific to the x86 architecture are being moved into a separate USE
> flag group called CPU_FLAGS_X86.

So based on this preliminary news item, I decided to get ahead of the 
game and add the use_expand to the appropriate make.conf sourced file...

... and IMMEDIATELY ran into this:

CPU_FLAGS_X86 ??  I'm ~amd64, do I leave it as-is, change it to 
CPU_FLAGS_X86_64 (kernel style), or change it to CPU_FLAGS_AMD64 (gentoo 
style)?

Knowing what I know about gentoo's use of use_expand, I'm almost certain 
I leave it as-is: CPU_FLAGS_X86 .  But if it's triggering the slightest 
doubt and hesitation here, and it is, it's certain to be a rather larger 
source of confusion to many.  So...

Please update the news item to specify that yes, it's _X86 for amd64 
users as well (assuming my near-certainty above is correct, else make 
explicit whatever actually applies).

Here's a second-draft pass at more explicit wording (also corrects a 
typo, "intruction", that spell-check flagged when I select-pasted):

The USE flags corresponding to instruction sets and other features
specific to the x86 (and amd64) architecture are being moved into a 
separate USE flag group called CPU_FLAGS_X86.  (Because amd64 is an 
extension of x86, amd64 users use the same group, CPU_FLAGS_X86.)


Additionally, unless the change is going to be atomic across the tree, 
including existing stable packages /and/ the news item at exactly the 
same time, I'd suggest an additional paragraph in the news item, 
suggesting keeping the existing USE flags for some period (the 
traditional 1-year supported update time period?) as well.  First-draft 
wording (please suggest updates, I'm not entirely comfortable with this 
but it suffices for first-draft):

In ordered to cover the transition period, you may wish to keep existing 
copies of any USE flags transfered to CPU_FLAGS_X86 for a suggested 
period of one year, or until you've merged updates with the new 
use_expand form of all affected packages, after which you can remove the 
old form from your normal USE.


Thanks.  The extra work that this whole change entails is appreciated! 
=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Michał Górny]

Dnia 2015-01-20, o godz. 09:13:19
Alexis Ballier  napisał(a):

> > For this reason, I would like to employ the solution used by Exherbo.
> > More specifically, use:
> > 
> >   libav? ( media-libs/libav:= )
> >   !libav? ( media-libs/ffmpeg:= )
> > 
> > This has two advantages:
> > 
> > 1. gives users more explicit control over whether they want to use
> > libav or ffmpeg. Since the two have mutual blockers, right now random
> > packages could have tried to force you to switch from one to the
> > other. However, most often Portage would just give you terribly
> > unreadable blockers.
> 
> This sounds good to me; there are a few things to consider though:
> 
> - Whatever default useflag is enabled, people will have "terribly
>   unreadable blockers" if they use the other implementation. It is even
>   worse if you want to follow upstreams in their preferred
>   implementation: some want libav, others ffmpeg...

It's no better than it is now. I don't see any obvious solution, except
maybe for fake flags that would shout loudly at users, like USE=libav
with matching pkg_pretend() telling to change the global flag. As I see
it, this problem has no solution unless we're going to make the two
co-installable.

> - This is hidden by the proposal but it enforces that a package has
>   equal visibility of both implementations (while the || requires only
>   one); libav tends to have a faster code cleaning rate and thus a
>   slower unmasking rate in gentoo. I agree with that requirement but
>   this might end-up being unpractical since this means packages
>   keywording and stabilization will have the slowest pace: mpv has been
>   masked for 4 months and counting for this reason.

It will *at least* make those issues explicit and visible to repoman.
Rather than discovered via users hitting impossible dep trees.

> - What to do with packages like mplayer that haven't been building
>   against libav for a year or so ?

I don't know. pkg_pretend() like suggested above?

> - ffmpeg dep is media-video/ffmpeg:0= (not media-libs but more
>   importantly the :0=)

Those were just proof-of-concept examples, I didn't take care to write
them correctly ;P.

> > 2. Subslots work correctly. Rebuilds are forced when the chosen
> > library is upgraded. Moreover, USE flag change causes a rebuild when
> > user decides to change the ffmpeg provider.
> 
> No offense, but this argument is complete crap. You should rather fix
> portage bugs than propose to introduce tree-wide changes to hide them...
> More precisely: || ( a:= b c:= d ) is perfectly defined (in the "what
> it means" sense, not in PMS sense). When the package is built, if 'a' is
> satisfied then a (and its subslot) is added to the subslot list of the
> package; ditto for c. You end up with a list of subslot deps, that you
> can store in vdb or whatever, and use that to decide when to rebuild
> the package.

Following a specification is not a bug. I could argue here a lot but...
just implement it. And return to us when you do. We had an item for
this for EAPI 6, and it was removed because nobody volunteered to
implement it. In fact, nobody even were able to fully specify what it
should do... but if you want to discuss that, please start a new
thread, possibly at the -pms list.

--
Best regards,
Michał Górny

Re: [gentoo-dev] Moving CPU flags into USE_EXPAND

[Andrew Savchenko]

On Tue, 20 Jan 2015 12:17:35 -0800 Christopher Head wrote:
> On January 20, 2015 12:47:03 AM PST, Alexis Ballier  
> wrote:
> >So, you're telling me that if you have a list of 90 cpu extensions, you
> >will from time to time open that list to see if there is a 91st one
> >added ? I think most people won't even notice, at best they'll look for
> >the changelog.
> 
> No, actually, I’m advocating the exact opposite. I’m saying that, as long as 
> the list file is kept up to date, then I will look at those 90 flags when I 
> first install and never again. If a 91st flag appears some day, then as long 
> as the file was maintained as I described in an earlier message (i.e. flags 
> are added as soon as manufacturers announce features), I already know I can 
> reliably ignore the new flag. After all, if the flag didn’t exist when I 
> installed the system, then my CPU must necessarily not have that 
> feature—unless CPUs are in the habit of sprouting new instructions after you 
> buy them!

Not exactly. CPUs are not in a habit, but software is. Some brand
new instuction set may be supported in (any of) packages with some
delay. Thus it is possible that instruction set supported by your
CPU will appear in the list of cpu flags after your ininial install.

Best regards,
Andrew Savchenko


pgp_gy7a6jRcb.pgp
Description: PGP signature

Re: [gentoo-dev] Things one could be upset about

[Jorge Manuel B. S. Vicetto]

On Sun, 18 Jan 2015, Patrick Lauer wrote:


On Saturday 17 January 2015 14:00:34 William Hubbs wrote:

On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote:

On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer 

wrote:

* Stage3 archives are too fat

See https://bugs.gentoo.org/show_bug.cgi?id=531632
We're now shipping three python versions and glib for extra fun!
Fix: Motivate releng to stop being silly


Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
is a great option if that remains the default after installation
(although it would be fine for just the initial stages).


I'm going to be very blunt. I am sick of the finger being pointed
only at releng for this.

The issue is package dependencies.

If even one package in the tree has a dumb dependency on python, e.g.
dev-lang/python, it will pull in all stable versionf of python.


Only if you absolutely insist that releng can never deviate from tree.


What RelEng has insisted in is building what we have in the tree - which, 
curiously enough, is a good way to test the tree.
Anyway, I've started working on a portdir config for the default stages, 
not because of this, but because of the caps issue (bug 531788).



Which is a silly idea, see bindist useflag, which is locally enabled for stage
building and then removed. Oh wait, it's not removed because we can't deviate
while deviating. So users regularly find ssl 'broken' ...


Building stages with USE="bindist" isn't a question of being silly but of 
making sure we don't give anyone a reason to "sue" Gentoo. We don't want 
to force USE="bindist" in the released stages, but that means we add one 
more workaround to catalyst code or we provide a way to specify which use 
flags we want to use for building the stage and which we want to be kept 
in make.conf.



It took me about 2h of fiddling around to find a few spots where stage3 has
useless bloat:
- pkgconfig pulling in 30MB of glib
- python installing tests (that's 3x 25MB now ...)


internal-glib is not something that should be used in the stages.


- having more than one python installed (which is not really absolutely
necessary, and could easily be reduced to one)


If your system has 3 python versions installed because the tree deps make 
portage install 3 versions, you should expect that to happen in your 
stages.
Since I'll have to work with portdir to address the caps issue above, I'll 
also take care of this, but if you want to blame someone about this, 
releng doesn't "maintain the tree".
Also, by adding this to the releng repo, that means we're going to have 
more work as we'll have to track new python versions.



Out of 700MB on-disk I could prune about 150MB - about 20%, without affecting
functionality

It's not about pointing a finger, it's about fixing issues when they are easy to
fix and not hiding behind a fake complexity argument.
(I would fix things, if I had access and/or certainty that patches provided
would be integrated ...)


I don't recall ever having seen a patch from you to catalyst, so before 
you "suggest" we don't want to accept your patches, you may want to send 
one ;-)



Have fun,

Patrick


Have fun,
Jorge

Re: [gentoo-dev] Moving CPU flags into USE_EXPAND

[Christopher Head]

On January 20, 2015 12:47:03 AM PST, Alexis Ballier  wrote:
>So, you're telling me that if you have a list of 90 cpu extensions, you
>will from time to time open that list to see if there is a 91st one
>added ? I think most people won't even notice, at best they'll look for
>the changelog.

No, actually, I’m advocating the exact opposite. I’m saying that, as long as 
the list file is kept up to date, then I will look at those 90 flags when I 
first install and never again. If a 91st flag appears some day, then as long as 
the file was maintained as I described in an earlier message (i.e. flags are 
added as soon as manufacturers announce features), I already know I can 
reliably ignore the new flag. After all, if the flag didn’t exist when I 
installed the system, then my CPU must necessarily not have that feature—unless 
CPUs are in the habit of sprouting new instructions after you buy them!

>Isn't it better to have a script, e.g. in gentoo-x86/scripts (that
>would be the first of this kind here I think), that would
>"parse" /proc/cpuinfo and output 'CPU_FLAGS_...="..."' so that for a
>first install you can simply send its output to make.conf and, if you
>are paranoid, can use it to check if this has changed in a postsync
>hook ?

I see having a script to select flag values as orthogonal to when the flag 
values need to be looked at. I also agree that having a script would be a good 
thing.

-- 
Christopher Head

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Zac Medico]

On 01/20/2015 12:15 PM, Ciaran McCreesh wrote:
> On Tue, 20 Jan 2015 12:12:32 -0800
> Zac Medico  wrote:
 Regardless of whether or not (or how) we choose to apply
 REQUIRED_USE to various cases, I think we should keep REQUIRED_USE
 around, since having a machine-readable representation of these
 constraints can potentially be extremely useful to dependency
 resolvers.
>>>
>>> [evidence needed]
>>
>> REQUIRED_USE is already successfully used to apply constraints that
>> would otherwise need to be enforced by executing shell code in
>> pkg_pretend. I think it's obvious that the declarative and
>> machine-readable nature of REQUIRED_USE make if preferable to using
>> pkg_pretend in many cases.
> 
> It's not obvious at all. The evidence so far suggests that good human
> readable messages are important, and that if we're ever going to get
> automated resolution then we'll need a new not-SAT-based format anyway.

Nothing stops us from using pkg_pretend right now. Since REQUIRED_USE
remains useful, I would suggest that we keep it around until we have
implemented a replacement that allows for automated resolution.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Ciaran McCreesh]

On Tue, 20 Jan 2015 12:12:32 -0800
Zac Medico  wrote:
> >> Regardless of whether or not (or how) we choose to apply
> >> REQUIRED_USE to various cases, I think we should keep REQUIRED_USE
> >> around, since having a machine-readable representation of these
> >> constraints can potentially be extremely useful to dependency
> >> resolvers.
> > 
> > [evidence needed]
> 
> REQUIRED_USE is already successfully used to apply constraints that
> would otherwise need to be enforced by executing shell code in
> pkg_pretend. I think it's obvious that the declarative and
> machine-readable nature of REQUIRED_USE make if preferable to using
> pkg_pretend in many cases.

It's not obvious at all. The evidence so far suggests that good human
readable messages are important, and that if we're ever going to get
automated resolution then we'll need a new not-SAT-based format anyway.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Zac Medico]

On 01/20/2015 12:05 PM, Ciaran McCreesh wrote:
> On Tue, 20 Jan 2015 12:02:38 -0800
> Zac Medico  wrote:
>> On 01/20/2015 09:25 AM, Ciaran McCreesh wrote:
>>> On Tue, 20 Jan 2015 13:41:21 +0100
>>> Ulrich Mueller  wrote:
 Seriously? You expect users to figure out [1] what combinations of
 USE flags will work for such an ebuild?
>>>
>>> Why don't we just admit that Brian was horribly wrong, scrap
>>> REQUIRED_USE in the next EAPI, and go back to the sensible,
>>> tried-and-tested way of doing it that I proposed to begin with?
>>
>> Regardless of whether or not (or how) we choose to apply REQUIRED_USE
>> to various cases, I think we should keep REQUIRED_USE around, since
>> having a machine-readable representation of these constraints can
>> potentially be extremely useful to dependency resolvers.
> 
> [evidence needed]
> 

REQUIRED_USE is already successfully used to apply constraints that
would otherwise need to be enforced by executing shell code in
pkg_pretend. I think it's obvious that the declarative and
machine-readable nature of REQUIRED_USE make if preferable to using
pkg_pretend in many cases.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Ciaran McCreesh]

On Tue, 20 Jan 2015 12:02:38 -0800
Zac Medico  wrote:
> On 01/20/2015 09:25 AM, Ciaran McCreesh wrote:
> > On Tue, 20 Jan 2015 13:41:21 +0100
> > Ulrich Mueller  wrote:
> >> Seriously? You expect users to figure out [1] what combinations of
> >> USE flags will work for such an ebuild?
> > 
> > Why don't we just admit that Brian was horribly wrong, scrap
> > REQUIRED_USE in the next EAPI, and go back to the sensible,
> > tried-and-tested way of doing it that I proposed to begin with?
> 
> Regardless of whether or not (or how) we choose to apply REQUIRED_USE
> to various cases, I think we should keep REQUIRED_USE around, since
> having a machine-readable representation of these constraints can
> potentially be extremely useful to dependency resolvers.

[evidence needed]

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Zac Medico]

On 01/20/2015 09:25 AM, Ciaran McCreesh wrote:
> On Tue, 20 Jan 2015 13:41:21 +0100
> Ulrich Mueller  wrote:
>> Seriously? You expect users to figure out [1] what combinations of
>> USE flags will work for such an ebuild?
> 
> Why don't we just admit that Brian was horribly wrong, scrap
> REQUIRED_USE in the next EAPI, and go back to the sensible,
> tried-and-tested way of doing it that I proposed to begin with?

Regardless of whether or not (or how) we choose to apply REQUIRED_USE to
various cases, I think we should keep REQUIRED_USE around, since having
a machine-readable representation of these constraints can potentially
be extremely useful to dependency resolvers.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Zac Medico]

On 01/20/2015 10:20 AM, Alexis Ballier wrote:
> On Tue, 20 Jan 2015 09:28:21 -0800
> Zac Medico  wrote:
> 
>> On 01/20/2015 01:11 AM, Alexis Ballier wrote:
>>> I think we can only make the safest assumption. Even without
>>> subslot, if you consider this: || ( a b c d ), with a and c
>>> installed but package automagically deciding to use only a, how can
>>> a PM decide whether it is safe to remove a or not after the package
>>> has been merged ?
>>
>> Right, this demonstrates that || deps are ambiguous. So, maybe we
>> should look at alternatives that are not so ambiguous, such as USE
>> conditionals and REQUIRED_USE constraints.
> 
> if you assume there are no automagic deps, what is wrong and/or
> ambiguous with '|| ( a b c d ) means "any of them, and at least all
> that were available when the package was merged"' ?
> in the above example this avoids PM breaking people systems by removing
> 'a' (e.g. if another packages pulls in 'd' that has a blocker against
> 'a') and as a side effect makes := deps work...

Sure, but when you start using words like "at least all that were
available when the package was merged", it shows a lack of precision in
your model. I tend to agree with Ciaran, that || deps are best suited to
dependencies that can be switched at runtime. For := deps, USE
conditionals and REQUIRED_USE constraints seem like a better fit.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Alexis Ballier]

On Tue, 20 Jan 2015 09:28:21 -0800
Zac Medico  wrote:

> On 01/20/2015 01:11 AM, Alexis Ballier wrote:
> > On Tue, 20 Jan 2015 01:01:41 -0800
> > Zac Medico  wrote:
> > 
> >> On 01/20/2015 12:13 AM, Alexis Ballier wrote:
> >>> On Mon, 19 Jan 2015 20:31:45 +0100
> >>> Michał Górny  wrote:
>  2. Subslots work correctly. Rebuilds are forced when the chosen
>  library is upgraded. Moreover, USE flag change causes a rebuild
>  when user decides to change the ffmpeg provider.
> >>>
> >>>
> >>> No offense, but this argument is complete crap. You should rather
> >>> fix portage bugs than propose to introduce tree-wide changes to
> >>> hide them... More precisely: || ( a:= b c:= d ) is perfectly
> >>> defined (in the "what it means" sense, not in PMS sense). When the
> >>> package is built, if 'a' is satisfied then a (and its subslot) is
> >>> added to the subslot list of the package; ditto for c. You end up
> >>> with a list of subslot deps, that you can store in vdb or
> >>> whatever, and use that to decide when to rebuild the package.
> >>
> >> That's an interesting proposal, but I immediately find myself
> >> questioning how closely it models reality. For example, maybe the
> >> package links to both the a:= package and c:= package, or maybe
> >> just to one of them. Shouldn't our model match reality as closely
> >> as possible, as long as it's practical?
> > 
> > Do you have any such example ?
> 
> Well, I think this demonstrates the sorts of ambiguities that may
> arise when using || deps to model reality. We may find that replacing
> || deps with USE conditionals and REQUIRED_USE constraints gives us a
> more accurate and practical model for some kinds of dependencies.

for the above example, yes of course, but this is because it is
another instance of automagic deps since you don't control to what the
package links to.

> > I think we can only make the safest assumption. Even without
> > subslot, if you consider this: || ( a b c d ), with a and c
> > installed but package automagically deciding to use only a, how can
> > a PM decide whether it is safe to remove a or not after the package
> > has been merged ?
> 
> Right, this demonstrates that || deps are ambiguous. So, maybe we
> should look at alternatives that are not so ambiguous, such as USE
> conditionals and REQUIRED_USE constraints.

if you assume there are no automagic deps, what is wrong and/or
ambiguous with '|| ( a b c d ) means "any of them, and at least all
that were available when the package was merged"' ?
in the above example this avoids PM breaking people systems by removing
'a' (e.g. if another packages pulls in 'd' that has a blocker against
'a') and as a side effect makes := deps work...

Alexis.

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Zac Medico]

On 01/20/2015 01:11 AM, Alexis Ballier wrote:
> On Tue, 20 Jan 2015 01:01:41 -0800
> Zac Medico  wrote:
> 
>> On 01/20/2015 12:13 AM, Alexis Ballier wrote:
>>> On Mon, 19 Jan 2015 20:31:45 +0100
>>> Michał Górny  wrote:
 2. Subslots work correctly. Rebuilds are forced when the chosen
 library is upgraded. Moreover, USE flag change causes a rebuild
 when user decides to change the ffmpeg provider.
>>>
>>>
>>> No offense, but this argument is complete crap. You should rather
>>> fix portage bugs than propose to introduce tree-wide changes to
>>> hide them... More precisely: || ( a:= b c:= d ) is perfectly
>>> defined (in the "what it means" sense, not in PMS sense). When the
>>> package is built, if 'a' is satisfied then a (and its subslot) is
>>> added to the subslot list of the package; ditto for c. You end up
>>> with a list of subslot deps, that you can store in vdb or whatever,
>>> and use that to decide when to rebuild the package.
>>
>> That's an interesting proposal, but I immediately find myself
>> questioning how closely it models reality. For example, maybe the
>> package links to both the a:= package and c:= package, or maybe just
>> to one of them. Shouldn't our model match reality as closely as
>> possible, as long as it's practical?
> 
> Do you have any such example ?

Well, I think this demonstrates the sorts of ambiguities that may arise
when using || deps to model reality. We may find that replacing || deps
with USE conditionals and REQUIRED_USE constraints gives us a more
accurate and practical model for some kinds of dependencies.

> I think we can only make the safest assumption. Even without subslot,
> if you consider this: || ( a b c d ), with a and c installed but
> package automagically deciding to use only a, how can a PM decide
> whether it is safe to remove a or not after the package has been
> merged ?

Right, this demonstrates that || deps are ambiguous. So, maybe we should
look at alternatives that are not so ambiguous, such as USE conditionals
and REQUIRED_USE constraints.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Ciaran McCreesh]

On Tue, 20 Jan 2015 13:41:21 +0100
Ulrich Mueller  wrote:
> Seriously? You expect users to figure out [1] what combinations of
> USE flags will work for such an ebuild?

Why don't we just admit that Brian was horribly wrong, scrap
REQUIRED_USE in the next EAPI, and go back to the sensible,
tried-and-tested way of doing it that I proposed to begin with?

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Ciaran McCreesh]

On Tue, 20 Jan 2015 09:13:19 +0100
Alexis Ballier  wrote:
> More precisely: || ( a:= b c:= d ) is perfectly defined (in
> the "what it means" sense, not in PMS sense).

It really isn't... || ( a b c ) only "works" currently if a, b and c can
be switched at runtime. If you're using it for anything else, it only
works if you don't look too closely.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

On Tue, 20 Jan 2015 12:01:13 +0100
Luca Barbato  wrote:
> On 17/01/15 16:03, Ciaran McCreesh wrote:
> > On Sat, 17 Jan 2015 22:59:08 +0800
> > Patrick Lauer  wrote:
> >>> The problem isn't the constants, though. The problem is the
> >>> resolution algorithm. There's not much point tweaking performance
> >>> until the resolver is fixed to produce a correct answer...
> >>
> >> Patches welcome :)
> >
> > If I send you a patch that updates all the documentation to use
> > Paludis rather than Portage, will you welcome it?
> >
> 
> If you rewrite paludis in C99 or equally language with non-brittle 
> runtime (libc++ isn't really viable yet =/) I would seriously
> consider it.

There's always -static-libstdc++. Although Gentoo is still the only
distribution which has trouble with having more than one libstdc++
visible at a time...

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Michael Orlitzky]

On 01/20/2015 05:31 AM, Luca Barbato wrote:
>>
>> This triggered a repressed memory of a bug once filed against me:
>>
>> https://blog.flameeyes.eu/2009/01/tinderboxing-problems-missing-default-use-flags
>>
>> I vaguely agree, but please address any hate mail to Diego.
>>
>>
> 
> Why?
> 

It was a joke, but I thought it was funnier without the THIS IS A JOKE
end-of-line smiley face. So.

Seriously though: I think he's right, but I don't feel that strongly
about it.

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Alec Ten Harmsel]

On 01/20/2015 07:46 AM, Andreas K. Huettel wrote:
> Am Dienstag 20 Januar 2015, 08:57:43 schrieb Michał Górny:
>> So a package supporting both providers has:
>>
>>   IUSE="ffmpeg libav"
>>   RDEPEND="
>>   ffmpeg? ( media-video/ffmpeg:= )
>>   libav? ( media-video/libav:= [media-libs/libpostproc:=] )"
>>   REQUIRED_USE="^^ ( ffmpeg libav )"
>>
>> And a package with optional support for both ends up like:
>>
>>   IUSE="avcodec postproc ffmpeg libav"
>>   RDEPEND="
>>   avcodec? (
>>   ffmpeg? ( media-video/ffmpeg:= )
>>   libav? ( media-video/libav:= )
>>   )
>>   postproc? (
>>   ffmpeg? ( media-video/ffmpeg:= )
>>   libav? ( media-libs/libpostproc:= )
>>   )"
>>   REQUIRED_USE="
>>   avcodec? ( ^^ ( ffmpeg libav ) )
>>   postproc? ( ^^ ( ffmpeg libav ) )
>>   ffmpeg? ( || ( avcodec postproc ) )
>>   libav? ( || ( avcodec postproc ) )"
>>
> No.
>

As a user, I completely agree with Andreas. What's wrong with a
dependency on virtual/ffmpeg? Let's say I want to install mpv; if I
completely ignore the upstream's recommendation of ffmpeg and instead
use libav to satisfy a virtual/ffmpeg dependency, that's my problem, not
the Gentoo developers (in my opinion).

I've written a few ebuilds, and by no means am even close to competent.
If they get more and more complex like this, why should I even bother
ever learning? I feel like ebuilds should be as simple as possible, to
keep devs sane and to interest more people. Another downside is that a
much more verbose solution like this will make dependency resolution
take longer imnho.

Alec

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Andreas K. Huettel]

Am Dienstag 20 Januar 2015, 08:57:43 schrieb Michał Górny:
> 
> So a package supporting both providers has:
> 
>   IUSE="ffmpeg libav"
>   RDEPEND="
>   ffmpeg? ( media-video/ffmpeg:= )
>   libav? ( media-video/libav:= [media-libs/libpostproc:=] )"
>   REQUIRED_USE="^^ ( ffmpeg libav )"
> 
> And a package with optional support for both ends up like:
> 
>   IUSE="avcodec postproc ffmpeg libav"
>   RDEPEND="
>   avcodec? (
>   ffmpeg? ( media-video/ffmpeg:= )
>   libav? ( media-video/libav:= )
>   )
>   postproc? (
>   ffmpeg? ( media-video/ffmpeg:= )
>   libav? ( media-libs/libpostproc:= )
>   )"
>   REQUIRED_USE="
>   avcodec? ( ^^ ( ffmpeg libav ) )
>   postproc? ( ^^ ( ffmpeg libav ) )
>   ffmpeg? ( || ( avcodec postproc ) )
>   libav? ( || ( avcodec postproc ) )"
> 

No.

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, council

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Ulrich Mueller]

> On Tue, 20 Jan 2015, Michał Górny wrote:

>   REQUIRED_USE="
>   avcodec? ( ^^ ( ffmpeg libav ) )
>   postproc? ( ^^ ( ffmpeg libav ) )
>   ffmpeg? ( || ( avcodec postproc ) )
>   libav? ( || ( avcodec postproc ) )"

Seriously? You expect users to figure out [1] what combinations of
USE flags will work for such an ebuild?

There are good reasons why the devmanual [2] advises against such
overuse of REQUIRED_USE:

| Note: In order to avoid forcing users to micro-manage flags too
| much, REQUIRED_USE should be used sparingly. Follow the normal
| policy whenever it is possible to do a build that will presumably
| suit the user's needs.

Ulrich

[1] https://en.wikipedia.org/wiki/Zebra_Puzzle
[2] 
http://devmanual.gentoo.org/general-concepts/use-flags/index.html#conflicting-use-flags


pgpMVDGQmmfud.pgp
Description: PGP signature

Re: [gentoo-dev] Things one could be upset about

[Luca Barbato]

On 19/01/15 16:47, hasufell wrote:

I think you forgot an important point:

* lack of practical QA: no review workflow and no appropriate tools for
reviewing

I could start a long text block about why reviewing is mandatory for QA,
but let's just think about it this way:
What do you think would happen if the linux kernel switched to CVS and
gave the most active 250 collaborators direct push access to the main
Linus repository?

I hope greg k-h does not read this. He'd probably get a heart attack.

Also: people seem to think we don't have enough manpower for a review
workflow. No, it's really the other way around. If you make
collaboration difficult, then you need a lot more manpower.



I already pointed out that there are _not_ good review tools. There are 
not for a by-email workflow we have in Libav, there aren't really for a 
tool-mediated workflow we could have in Gentoo.


I have no problems in devoting some time on preparing a tool suited for 
our purpose (once we switch to git), but I'd need more volunteers to 
help me with it.


lu

Re: [gentoo-dev] Things one could be upset about

[Luca Barbato]

On 17/01/15 16:03, Ciaran McCreesh wrote:

On Sat, 17 Jan 2015 22:59:08 +0800
Patrick Lauer  wrote:

The problem isn't the constants, though. The problem is the
resolution algorithm. There's not much point tweaking performance
until the resolver is fixed to produce a correct answer...


Patches welcome :)


If I send you a patch that updates all the documentation to use Paludis
rather than Portage, will you welcome it?



If you rewrite paludis in C99 or equally language with non-brittle 
runtime (libc++ isn't really viable yet =/) I would seriously consider it.


lu

Re: [gentoo-dev] Packages up for grabs

[Pacho Ramos]

El mar, 20-01-2015 a las 11:21 +0100, Hanno Böck escribió:
[...]
> maintainer-wanted or if they're likely broken anyway just removed.

s/-wanted/-needed/ :)

Re: [gentoo-dev] Things one could be upset about

[Róbert Čerňanský]

On Tue, 20 Jan 2015 11:08:19 +0300
Andrew Savchenko  wrote:

> On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote:
> > On Tue, 20 Jan 2015 00:14:29 +0300
> > Andrew Savchenko  wrote:
> > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
> > > > From my point of view it would do much help if portage resolves
> > > > USE dependencies automatically
[...]
> > > As the user the last thing I want is to have some USE flags
> > > changed without my permission
[...]
> > Is is of course perfectly fine to have that option disabled by
> > default.  But I am afraid that I do not quite understand yours and
> > Daniel's concerns.  If I paraphrase your statement with regards to
> > current dependency handling, it is as if you were saying: "I do not
> > want portage to install any package automatically by its own, I want
> > to install all the dependencies explicitly."
> 
> The paraphrasing above is wrong. What I want to say is "I don't want
> portage to automagically change _functionality_ of my packages on
> its own, even in order to solve dependencies."
>  
> > Why we allow portage to install dependencies and still have things
> > under control?  Well we have --pretend and --ask options and we can
> > see what would/will be done.  This would be the same with USE flags.
> > You would see in the --pretend output which flags would be changed.
> 
> No this wouldn't be the same. USE flags are _not_ equal to package
> dependencies, sometimes they will not trigger dependency change at
> all. USE flags are about _functionality_, dependencies are about
> the _means_ to implement that functionality (as well as base
> functionality of a package).

I see your point about functionality.  There are USE flags that
_changes_ functionality (like threads) and there are others which _adds_
functionality (like for example speex adds possibility to play files in
that particular format in mplayer).  The former I consider similar to
package dependencies because they too can add functionality to the
system (for example if ruby is installed as dependency it adds
possibility to execute ruby scripts) same as those USE flags adds
functionality to an application.

However, even if portage wants me to change USE flag that will change
functionality, in 99% of time I end up adding to my package.use what it
wants, since my goal is to install some application or update my
applications.  So just reviewing the changes that portage wants to do
and hit Y.  And in those rare cases when I really do not want some flag
enabled or some package installed I hit N and tweak things.

> > To match the package dependency handling, there should be also an
> > option equivalent to --depclean.  It would revert the USE changes
> > which were done because of dependencies requirements if no longer
> > needed.
> 
> This will dramatically increase complexity of dependencies metadata
> as well as of algorithms to handle it (and they are already slow),
> with no clear benefits therefore. No, thanks.

Are you talking about proposed new USE specific depclean option or
emerge in general?  If it is the new depclean command that would run
slow then I am quite sure it will still be faster than me or anyone else
going manually through package.use and for each and every USE flag there
trying to remove it, run emerge, see if it passes, if not, add it back.

If it is emerge in general that would be slower that it is still better
than run it, see it fail and telling you what USE to change, you making
the change, and run it again.

> > For example, lets have following packages:
> > 
> > - libbar
> > - libfoo with IUSE=bar, which depends on: bar? ( libbar )
> > - foo, which depends on: libfoo[bar]
[...]
> > New behaviour with automatic USE dependencies resolution:
> > 
> > emerge -av foo
> > [ebuild  N ] libbar
> > [ebuild  N ] libfoo +bar
> > [ebuild  N ] foo
> > 
> > Now, you can hit Y if you agree what portage wants to do or N and
> > not to install foo at all.
> 
> And if I don't agree? What if for some reason I don't want to
> have libfoo[bar] on my system. What If preferred solution will
> be not to use libbar at all and to use libclue instread? 

In this example, if you do not agree, you have no other option how to
install foo (with or without automatic USE deps).  Because foo depends
on libfoo[bar] unconditionally.

If however foo would depend either on libfoo[bar] or libclue then the
portage would do the same thing as today (I guess it would take the
first dependency as mandatory or the one which is already installed if
any.) In this case, yes, your preference might be libclue instead what
portage chooses.  But I see no difference in the way how user choose it
comparing to todays '|| ( libfoo libclue )' -like dependencies.

> World update on my systems usually involves about 2000 of packages.
> It's already hard to read emerge -DNuav output in order to check
> for new packages, dependencies, downgraded and removed packages.
> Automagick dependency chan

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Luca Barbato]

On 20/01/15 03:07, Michael Orlitzky wrote:

On 01/19/2015 05:44 PM, Pacho Ramos wrote:


I agree with your suggestion but I would prefer the Remi's approach of
letting people to know if they want "ffmpeg" or "libav", otherwise it is
not so obvious to know what disabling/enabling one of that USE flags
will end up causing without reading each ebuild :/ (also, maybe some
ebuilds will use one logic while others the inverse)



This triggered a repressed memory of a bug once filed against me:

https://blog.flameeyes.eu/2009/01/tinderboxing-problems-missing-default-use-flags

I vaguely agree, but please address any hate mail to Diego.




Why?

[gentoo-dev] Packages up for grabs

[Hanno Böck]

Hi,

I'm listed on a number of packages as the maintainer that I don't
really care about any more.
Please if you want to take any of them add yourself as maintainer and
remove me. What's left will be re-assigned to herds if they exist,
maintainer-wanted or if they're likely broken anyway just removed.


app-editors/bluefish
Editor for programming and HTML. Recently bumped to the latest version.
Some minor bugs open in bugzilla that could need some attention.

app-laptop/hdaps-gl
Trivial tool to detect movement sensor in thinkpad laptops. Didn't have
a release in a long time, but also no issues, so probably can stay like
it is.

app-portage/portage-mod_jabber
I'm pretty sure this doesn't work any more. Dead upstream. Will
p.mask/lastrite if nobody steps up (stepping up means likely being the
new upstream).

dev-perl/Math-Vec
random perl package, will re-assign to perl herd if nobody else takes
it.

dev-ruby/dbf
database import for ruby. Already assigned to ruby and geoscience herd
where it can stay.

dev-ruby/GeoRuby
Already assigned to ruby and geoscience herd where it can stay.

media-gfx/pngnq
There's a trivial automake issue open in bugzilla which I will likely
just fix before giving it away, other than that will re-assign to
maintainer-wanted if nobody takes it.

media-sound/moodbar
This was used by amarok to display a colorful "moodbar" for audio files.
It depends on gstreamer 0.10, don't know if this will go away anyway at
some point.


cu,
-- 
Hanno Böck
http://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: BBB51E42

[gentoo-dev] Packages up for grabs

[Pacho Ramos]

Due to mduft retirement the following packages are up for grabs now:
app-emulation/rex-client
net-proxy/cntlm

Thanks

[gentoo-dev] Review: news item and script for CPU_FLAGS_X86

[Michał Górny]

Dnia 2015-01-18, o godz. 21:44:05
Michał Górny  napisał(a):

> Hello,
> 
> I would like to commit the following flags as cpu_flags_x86_desc.
> The list combines global USE flags with some local USE flags I've been
> able to find.

Following your suggestions, I'm attaching three files:

1. updated cpu_flags_x86.desc, with:

 a. fma3/fma4 distinction,

 b. aes-ni renamed to aes for consistency with cpuinfo, we no longer
 have to worry about USE flag collision with other USE=aes uses,

 c. complete and machine-parseable listing of cpuinfo flags.

2. cpuinfo2cpuflags-x86.py script that parses cpu_flags_x86.desc
and /proc/cpuinfo and generates a nice CPU_FLAGS_X86 value for you. We
need to decide where to put it (scripts/?).

3. Initial text of the news item for review.

-- 
Best regards,
Michał Górny


cpu_flags_x86.desc
Description: Binary data
#!/usr/bin/env python

import collections, errno, os, os.path, re, sys

def main(*argv):
	if 'PORTDIR' in os.environ:
		portdir = os.environ['PORTDIR']
	else:
		try:
			import portage
		except ImportError:
			return 'Unable to determine PORTDIR, please set it before calling this script'
		else:
			trees = portage.create_trees(
	config_root = os.environ.get('PORTAGE_CONFIGROOT'),
	target_root = os.environ.get('ROOT'))
			tree = trees[max(trees)]
			portdir = tree['porttree'].dbapi.repositories.get_location_for_name('gentoo')

	# dict of cpuinfo -> flag mappings
	flag_dict = collections.defaultdict(list)

	desc = os.path.join(portdir, 'profiles', 'desc', 'cpu_flags_x86.desc')
	# flag - description ; discards empty lines and comments
	l_regexp = re.compile(r'^(?P[a-z0-9_]+)\s+-\s+(?P.+)$')
	# [cpuinfo-flag] ; [] are discarded for easy use
	cpuinfo_regexp = re.compile(r'(?<=\[).+?(?=\])')
	try:
		with open(desc, 'r') as f:
			for l in f:
m = l_regexp.match(l)
if m:
	# grep for any [cpuinfo] or [cpuinfo2]
	flags = cpuinfo_regexp.findall(l)
	for fl in flags:
		flag_dict[fl].append(m.group('flag'))
	# fallback to exact flag match
	if not flags:
		flag_dict[m.group('flag')].append(m.group('flag'))
	except OSError as e:
		if e.errno == errno.ENOENT:
			return '%s does not exist, set PORTDIR to correct gentoo repository location' % desc

	out = set()

	flags_regexp = re.compile(r'^flags\s+:\s+(?P.*)$')
	with open('/proc/cpuinfo', 'r') as f:
		for l in f:
			m = flags_regexp.match(l)
			if m:
for fl in m.group('flags').split():
	if fl in flag_dict:
		out.update(flag_dict[fl])
break

	print('CPU_FLAGS_X86="%s"' % ' '.join(sorted(out)))
	return 0

if __name__ == '__main__':
	sys.exit(main(*sys.argv[1:]))
Title: CPU_FLAGS_X86 introduction
Author: Michał Górny 
Content-Type: text/plain
Posted: 2015-01-xx
Revision: 1
News-Item-Format: 1.0
Display-If-Keyword: amd64 ~amd64 x86 ~x86

The USE flags corresponding to intruction sets and other features
specific to the x86 architecture are being moved into a separate USE
flag group called CPU_FLAGS_X86.

In order not to lose CPU-specific optimizations, users will be required
to update their make.conf (and package.use) file. For example, if
the following USE flags were present:

  USE="mmx mmxext sse sse2 sse3"

Those flags need to be copied into:

  CPU_FLAGS_X86="mmx mmxext sse sse2 sse3"

When in doubt, consult profiles/desc/cpu_flags_x86.desc. Most of
the flag names match /proc/cpuinfo names, with the notable exception
of SSE3 which is called 'pni' in /proc/cpuinfo (and which isn't the same
as SSSE3).

To help you enable correct USE flags, you can use the
cpuinfo2cpuflags-x86.py script provided in scripts/ directory of the
repository. It will parse your /proc/cpuinfo and print an accurate
CPU_FLAGS_X86 value for it.


pgpIPsI4I4KEW.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Alexis Ballier]

On Tue, 20 Jan 2015 01:01:41 -0800
Zac Medico  wrote:

> On 01/20/2015 12:13 AM, Alexis Ballier wrote:
> > On Mon, 19 Jan 2015 20:31:45 +0100
> > Michał Górny  wrote:
> >> 2. Subslots work correctly. Rebuilds are forced when the chosen
> >> library is upgraded. Moreover, USE flag change causes a rebuild
> >> when user decides to change the ffmpeg provider.
> > 
> > 
> > No offense, but this argument is complete crap. You should rather
> > fix portage bugs than propose to introduce tree-wide changes to
> > hide them... More precisely: || ( a:= b c:= d ) is perfectly
> > defined (in the "what it means" sense, not in PMS sense). When the
> > package is built, if 'a' is satisfied then a (and its subslot) is
> > added to the subslot list of the package; ditto for c. You end up
> > with a list of subslot deps, that you can store in vdb or whatever,
> > and use that to decide when to rebuild the package.
> 
> That's an interesting proposal, but I immediately find myself
> questioning how closely it models reality. For example, maybe the
> package links to both the a:= package and c:= package, or maybe just
> to one of them. Shouldn't our model match reality as closely as
> possible, as long as it's practical?

Do you have any such example ?

I think we can only make the safest assumption. Even without subslot,
if you consider this: || ( a b c d ), with a and c installed but
package automagically deciding to use only a, how can a PM decide
whether it is safe to remove a or not after the package has been
merged ?

Alexis.

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Zac Medico]

On 01/20/2015 12:13 AM, Alexis Ballier wrote:
> On Mon, 19 Jan 2015 20:31:45 +0100
> Michał Górny  wrote:
>> 2. Subslots work correctly. Rebuilds are forced when the chosen
>> library is upgraded. Moreover, USE flag change causes a rebuild when
>> user decides to change the ffmpeg provider.
> 
> 
> No offense, but this argument is complete crap. You should rather fix
> portage bugs than propose to introduce tree-wide changes to hide them...
> More precisely: || ( a:= b c:= d ) is perfectly defined (in the "what
> it means" sense, not in PMS sense). When the package is built, if 'a' is
> satisfied then a (and its subslot) is added to the subslot list of the
> package; ditto for c. You end up with a list of subslot deps, that you
> can store in vdb or whatever, and use that to decide when to rebuild
> the package.

That's an interesting proposal, but I immediately find myself
questioning how closely it models reality. For example, maybe the
package links to both the a:= package and c:= package, or maybe just to
one of them. Shouldn't our model match reality as closely as possible,
as long as it's practical?
-- 
Thanks,
Zac

Re: [gentoo-dev] Review: desc/cpu_flags_x86.desc

[Alexis Ballier]

On Tue, 20 Jan 2015 09:52:40 +0100
Michał Górny  wrote:

> Dnia 2015-01-19, o godz. 11:38:26
> Alexis Ballier  napisał(a):
> 
> > On Sun, 18 Jan 2015 21:44:05 +0100
> > Michał Górny  wrote:
> > 
> > > Hello,
> > > 
> > > I would like to commit the following flags as cpu_flags_x86_desc.
> > > The list combines global USE flags with some local USE flags I've
> > > been able to find.
> > > 
> > > 
> > > 3dnow - Use the 3DNow! instruction set
> > > 3dnowext - Use the Enhanced 3DNow! instruction set
> > > aes-ni - Enable support for Intel's AES instruction set (aes in
> > > cpuinfo) avx - Adds support for Advanced Vector Extensions
> > > instructions avx2 - Adds support for Advanced Vector Extensions 2
> > > instructions fma - Use the Fused Multiply Add instruction set
> > > mmx - Use the MMX instruction set
> > > mmxext - Use the Extended MMX instruction set (intersection of
> > > Enhanced 3DNow! and SSE instruction sets) (3dnowext or sse in
> > > cpuinfo) padlock - Use VIA padlock instructions popcnt - Enable
> > > popcnt instruction support sse - Use the SSE instruction set
> > > sse2 - Use the SSE2 instruction set
> > > sse3 - Use the SSE3 instruction set (pni in cpuinfo)
> > > sse4 - Enable SSE4 instruction support
> > > sse4_1 - Enable SSE4.1 instruction support
> > > sse4_2 - Enable SSE4.2 instruction support
> > > sse4a - Enable SSE4a instruction support
> > > ssse3 - Use the SSSE3 instruction set
> > > 
> > 
> > e... are you aware that these descriptions are close to
> > useless ? 'foo - enable foo' -> thanks for the information I
> > couldn't have guessed...
> 
> It's not my fault we have to put some description there!

I know; I'm just ranting because you could grep use.local.desc to get
more informative ones for some of them; then we can start discussing on
how to improve the other ones :)

Alexis.

Re: [gentoo-dev] Review: desc/cpu_flags_x86.desc

[Michał Górny]

Dnia 2015-01-19, o godz. 11:38:26
Alexis Ballier  napisał(a):

> On Sun, 18 Jan 2015 21:44:05 +0100
> Michał Górny  wrote:
> 
> > Hello,
> > 
> > I would like to commit the following flags as cpu_flags_x86_desc.
> > The list combines global USE flags with some local USE flags I've been
> > able to find.
> > 
> > 
> > 3dnow - Use the 3DNow! instruction set
> > 3dnowext - Use the Enhanced 3DNow! instruction set
> > aes-ni - Enable support for Intel's AES instruction set (aes in
> > cpuinfo) avx - Adds support for Advanced Vector Extensions
> > instructions avx2 - Adds support for Advanced Vector Extensions 2
> > instructions fma - Use the Fused Multiply Add instruction set
> > mmx - Use the MMX instruction set
> > mmxext - Use the Extended MMX instruction set (intersection of
> > Enhanced 3DNow! and SSE instruction sets) (3dnowext or sse in
> > cpuinfo) padlock - Use VIA padlock instructions popcnt - Enable
> > popcnt instruction support sse - Use the SSE instruction set
> > sse2 - Use the SSE2 instruction set
> > sse3 - Use the SSE3 instruction set (pni in cpuinfo)
> > sse4 - Enable SSE4 instruction support
> > sse4_1 - Enable SSE4.1 instruction support
> > sse4_2 - Enable SSE4.2 instruction support
> > sse4a - Enable SSE4a instruction support
> > ssse3 - Use the SSSE3 instruction set
> > 
> 
> e... are you aware that these descriptions are close to useless ?
> 'foo - enable foo' -> thanks for the information I couldn't have
> guessed...

It's not my fault we have to put some description there!

> you already have more useful ones available:
> 
> Enables FMA3 optimizations: AMD processors starting
>   with Piledriver architecture and Intel Haswell based processors or
>   later.
> Enables FMA4 optimizations: AMD processors starting
> with Bulldozer architecture.
> Enables SSE4.2 optimizations: Nehalem-based Intel
> Core i7 or later.
> Faster floating point optimization for SSSE3 capable
> chips (Intel Core 2 and later chips)
> 
> and so on...

Feel free to update them yourself. As far as I'm concerned, those
descriptions are worse than useless because they end up being
confusing. Many CPU lines had weird models with more or less
instruction sets, so brand name aren't really helpful. It's better to
focus on /proc/cpuinfo.

-- 
Best regards,
Michał Górny


pgpwEywupN5yb.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Moving CPU flags into USE_EXPAND

[Alexis Ballier]

On Tue, 20 Jan 2015 00:29:22 -0800
Christopher Head  wrote:

> On Tue, 20 Jan 2015 09:21:54 +0100
> Alexis Ballier  wrote:
> 
> > you will not see it if no package use it.
> 
> I guess you mean I wouldn’t see it in emerge output if no package uses
> it, even if it is USE-expanded? Yes, I’m aware of that. But if all the
> flags are listed together in one place (I forget what the file was
> called—cpu.flags.use.desc or something?), then I can just open that
> file to see the list, hence why case 2 is better IMO—put all the
> entries in that file for users to look at, even if no package uses
> them yet.

So, you're telling me that if you have a list of 90 cpu extensions, you
will from time to time open that list to see if there is a 91st one
added ? I think most people won't even notice, at best they'll look for
the changelog.

Isn't it better to have a script, e.g. in gentoo-x86/scripts (that
would be the first of this kind here I think), that would
"parse" /proc/cpuinfo and output 'CPU_FLAGS_...="..."' so that for a
first install you can simply send its output to make.conf and, if you
are paranoid, can use it to check if this has changed in a postsync
hook ?

Alexis.

Re: [gentoo-dev] Moving CPU flags into USE_EXPAND

[Christopher Head]

On Tue, 20 Jan 2015 09:21:54 +0100
Alexis Ballier  wrote:

> you will not see it if no package use it.

I guess you mean I wouldn’t see it in emerge output if no package uses
it, even if it is USE-expanded? Yes, I’m aware of that. But if all the
flags are listed together in one place (I forget what the file was
called—cpu.flags.use.desc or something?), then I can just open that
file to see the list, hence why case 2 is better IMO—put all the
entries in that file for users to look at, even if no package uses them
yet.
-- 
Christopher Head


signature.asc
Description: PGP signature

Re: [gentoo-dev] Moving CPU flags into USE_EXPAND

[Alexis Ballier]

On Mon, 19 Jan 2015 23:43:19 -0800
Christopher Head  wrote:

> On Wed, 14 Jan 2015 11:01:16 -0800
> Zac Medico  wrote:
> 
> > Why should we have to foresee the future? We can easily add support
> > for new flags in CPU_FLAGS_* variables at any time.
> 
> Ah, what I meant was that whoever maintains this flag list only needs
> to forsee the present—when AMD or Intel adds a new instruction set
> extension, you add a flag for it to the variable immediately, whether
> or not any package actually uses it yet. Why? Here’s why:
> 
> Case 1: flags are added only as packages need them. This is pretty
> much what we have today, only without the USE-expand feature. Every
> time a package adds support for a new CPU feature, it gets a new USE
> flag, and I see it in my emerge output.
[...]

you will not see it if no package use it.


Alexis.

Re: [gentoo-dev] RFCv2: USE=avcodec (+ USE=ffmpeg/libav)

[Alexis Ballier]

On Tue, 20 Jan 2015 08:57:43 +0100
Michał Górny  wrote:

> Hi,
> 
> Since you didn't like the previous hacky idea, here's a new one.
> 
> The basic flags correspond to features and are used if the relevant
> support is optional:
> 
>   avcodec - Enables audio/video decoding support via libavcodec
> (ffmpeg/libav)
>   postproc - Enable image post-processing via libpostproc
> (ffmpeg/libpostproc)
> 

I crearly prefer the other prop. We have: libavcodec, libavdevice,
libavfilter, libavformat, libavresample, libavutil, libpostproc,
libswresample, libswscale. Do you want a useflag for each of them ? Can
you tell me on top of your head which one will force ffmpeg and which
one will force libav ?

Also, avcodec is just a codec library that, alone, is mostly useless for
e.g. a video player: you need a demuxer to read your files, which
demuxer is in libavformat...

Alexis.

Re: [gentoo-dev] RFC: USE=libav as replacement for broken || ( libav:= ffmpeg:= )

[Alexis Ballier]

On Mon, 19 Jan 2015 20:31:45 +0100
Michał Górny  wrote:

> Hello,
> 
> As we've discussed multiple times, the following kind of dependencies
> is completely broken and can't work:
> 
>   || ( media-libs/libav:= media-libs/ffmpeg:= )

See end of the email.

> For this reason, I would like to employ the solution used by Exherbo.
> More specifically, use:
> 
>   libav? ( media-libs/libav:= )
>   !libav? ( media-libs/ffmpeg:= )
> 
> This has two advantages:
> 
> 1. gives users more explicit control over whether they want to use
> libav or ffmpeg. Since the two have mutual blockers, right now random
> packages could have tried to force you to switch from one to the
> other. However, most often Portage would just give you terribly
> unreadable blockers.


This sounds good to me; there are a few things to consider though:

- Whatever default useflag is enabled, people will have "terribly
  unreadable blockers" if they use the other implementation. It is even
  worse if you want to follow upstreams in their preferred
  implementation: some want libav, others ffmpeg...
- This is hidden by the proposal but it enforces that a package has
  equal visibility of both implementations (while the || requires only
  one); libav tends to have a faster code cleaning rate and thus a
  slower unmasking rate in gentoo. I agree with that requirement but
  this might end-up being unpractical since this means packages
  keywording and stabilization will have the slowest pace: mpv has been
  masked for 4 months and counting for this reason.
- What to do with packages like mplayer that haven't been building
  against libav for a year or so ?
- ffmpeg dep is media-video/ffmpeg:0= (not media-libs but more
  importantly the :0=)
- It is a good opportunity to kill virtual/ffmpeg and at the same time
  workaround a subslot and := deps limitation with virtuals.


> 2. Subslots work correctly. Rebuilds are forced when the chosen
> library is upgraded. Moreover, USE flag change causes a rebuild when
> user decides to change the ffmpeg provider.


No offense, but this argument is complete crap. You should rather fix
portage bugs than propose to introduce tree-wide changes to hide them...
More precisely: || ( a:= b c:= d ) is perfectly defined (in the "what
it means" sense, not in PMS sense). When the package is built, if 'a' is
satisfied then a (and its subslot) is added to the subslot list of the
package; ditto for c. You end up with a list of subslot deps, that you
can store in vdb or whatever, and use that to decide when to rebuild
the package.


Alexis.

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

On Tue, 20 Jan 2015 06:51:01 +0100 Róbert Čerňanský wrote:
> On Mon, 19 Jan 2015 20:51:31 +
> Ciaran McCreesh  wrote:
> 
> > On Mon, 19 Jan 2015 21:44:25 +0100
> > Róbert Čerňanský  wrote:
> > > From my point of view it would do much help if portage resolves USE
> > > dependencies automatically instead of telling the user to change USE
> > > flags manually (I am talking about bug #258371).
> > 
> > This is only possible in carefully selected circumstances, and to get
> > it to work more generally would require a lot of hinting from package
> > maintainers.
> 
> But portage already knows that.  It tells the user which USE flags
> needs to be changed in order to emerge a package.  It should just go
> one step further - to make the proposed change happen by itself.

And ofter it proposes multiple alternative ways to fix this. How do
you suppose to select between multiple possible alternative
solutions then? Another issues is that sometimes it will be
preferred by the user to disable offending functionality at all.

Good example here is sci-libs/hdf5. Set USE="cxx threads mpi" in
make.conf. Have fun. Especially if in one application (depending on
hdf5) you really need cxx support and in another one (also
depending on hdf5) mpi support is really needed. In some cases it is
preferred to disable hdf5 support at all.

Best regards,
Andrew Savchenko


pgpcWxk85uSeF.pgp
Description: PGP signature

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote:
> On Tue, 20 Jan 2015 00:14:29 +0300
> Andrew Savchenko  wrote:
> 
> > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
> > > From my point of view it would do much help if portage resolves USE
> > > dependencies automatically instead of telling the user to change USE
> > > flags manually (I am talking about bug #258371).
> > 
> > As the user the last thing I want is to have some USE flags changed
> > without my permission ending up stuff I need to be omitted or stuff
> > I don't want to see on my system to be installed. Of course if
> > someone prefers USE flags to be randomly changed I don't mind if
> > such option will exist (as proposed in bug #258371) as long as it
> > is disabled by default.
> 
> Is is of course perfectly fine to have that option disabled by
> default.  But I am afraid that I do not quite understand yours and
> Daniel's concerns.  If I paraphrase your statement with regards to
> current dependency handling, it is as if you were saying: "I do not
> want portage to install any package automatically by its own, I want
> to install all the dependencies explicitly."

The paraphrasing above is wrong. What I want to say is "I don't want
portage to automagically change _functionality_ of my packages on
its own, even in order to solve dependencies."
 
> Why we allow portage to install dependencies and still have things
> under control?  Well we have --pretend and --ask options and we can
> see what would/will be done.  This would be the same with USE flags.
> You would see in the --pretend output which flags would be changed.

No this wouldn't be the same. USE flags are _not_ equal to package
dependencies, sometimes they will not trigger dependency change at
all. USE flags are about _functionality_, dependencies are about
the _means_ to implement that functionality (as well as base
functionality of a package).

> To match the package dependency handling, there should be also an
> option equivalent to --depclean.  It would revert the USE changes
> which were done because of dependencies requirements if no longer
> needed.

This will dramatically increase complexity of dependencies metadata
as well as of algorithms to handle it (and they are already slow),
with no clear benefits therefore. No, thanks.

> For example, lets have following packages:
> 
> - libbar
> - libfoo with IUSE=bar, which depends on: bar? ( libbar )
> - foo, which depends on: libfoo[bar]
> 
> USE flag bar is not enabled.  You want to install application foo.
> 
> Current behaviour:
> 
> # emerge -av foo
> ...
> Required USE changes:
> libfoo +bar
> ... (sorry for not exact emerge output but you get the idea)
> 
> Now, you either fire up your favorite editor and add "libfoo +bar" to
> /etc/portage/package.use, re-run emerge and have libbar installed even
> you do not want it.  The other option is not to install application
> foo at all.
> 
> If you choose to emerge it and after year or so you decide to unmerge
> it because you do not need it anymore you still will have a leftover
> in package.use file - the line "libfoo +foo" even if you run emerge
> --depclean.
> 
> New behaviour with automatic USE dependencies resolution:
> 
> emerge -av foo
> [ebuild  N ] libbar
> [ebuild  N ] libfoo +bar
> [ebuild  N ] foo
> 
> Now, you can hit Y if you agree what portage wants to do or N and not
> to install foo at all.

And if I don't agree? What if for some reason I don't want to
have libfoo[bar] on my system. What If preferred solution will
be not to use libbar at all and to use libclue instread? 

World update on my systems usually involves about 2000 of packages.
It's already hard to read emerge -DNuav output in order to check
for new packages, dependencies, downgraded and removed packages.
Automagick dependency change will make this even harder, much
harder because it will involve multiple additional emerge runs in
order to deal with issues properly. And each run takes about half
an hour.

Yet again, Gentoo is all about choise. If someone wants that
automagick to be implemented, go ahead and send patches for
developers to review. But please keep it disabled by default, in
most cases it will provide bizarre results anyway. (Think about
conflicting alternatives, no sane way to prefer one above another
automatically and a chain of such issues during world update.) 

Best regards,
Andrew Savchenko


pgpeQjP_x8K4S.pgp
Description: PGP signature