Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead
Drive-by trolling comment but I wish the effort to keep porkage alive would have instead been directed towards pkgcore.
Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 13/01/14 09:39, C. Bergström wrote: Drive-by trolling comment but I wish the effort to keep porkage alive would have instead been directed towards pkgcore. Realistically, we have to keep updating them both in parallel. pkgcore needs to be brought up to portage-level functionality, and we need to keep fixing portage bugs for its many users. We could have a technical discussion on the merits of pkgcore vs. portage, but in the end, it's about the users. For the record, I would be very happy if you hacked on pkgcore. Just as happy as if you hacked on portage, even. So please do. :-) - -- Alexander alexan...@plaimi.net http://plaimi.net/~alexander -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLTpx8ACgkQRtClrXBQc7UXtAEAp+ZlhX1E7NGp4rF0x8yEVzFk Ynf1W9rr0Q2XZ3nBsk4A/RsrKd82zgYPaBizcs2toz/U9pjmbOkS9PUQPJAFe4ze =EKI+ -END PGP SIGNATURE-
Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead
On Mon, Jan 13, 2014 at 9:39 AM, C. Bergström cbergst...@pathscale.com wrote: Drive-by trolling comment but I wish the effort to keep porkage alive would have instead been directed towards pkgcore. If you know your email is going to be drive-by trolling, maybe just hold it in next time?
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/13/14 03:43 PM, Alexander Berntsen wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 13/01/14 09:39, C. Bergström wrote:
Drive-by trolling comment but I wish the effort to keep porkage
alive would have instead been directed towards pkgcore.
Realistically, we have to keep updating them both in parallel. pkgcore
needs to be brought up to portage-level functionality, and we need to
keep fixing portage bugs for its many users. We could have a technical
discussion on the merits of pkgcore vs. portage, but in the end, it's
about the users.
For the record, I would be very happy if you hacked on pkgcore. Just
as happy as if you hacked on portage, even. So please do. :-)
Where I work uses pkgcore[1], but not the areas which are generally
beneficial to the whole community. (We use it as part of a web
application to handle testsuites which have build dependencies.) We can
blah blah about performance of resolving package dependencies all day
long, but it's clearly not a game changer feature for users. (or they
just don't know) Long term the API to pkgcore could be beneficial, but
again I'm not sure it's a game changer for users. Dare I turn this from
a trolling comment into a laundry list of pros/cons of pkgcore and why
now may be a good time to invest in the future. At the end of the day we
have one codebase which is engineered and another which has evolved.
If gentoo only ever wants to fetch tarballs and build source.. who
cares.. If you ever want to build something on top of that - you will
painfully realize that there's only 1 choice.
On 01/13/14 03:59 PM, Dirkjan Ochtman wrote:
If you know your email is going to be drive-by trolling, maybe just
hold it in next time?
Yeah I know better, but this time was like a fart - (half attempt joking)
[1] /* Side details */
In a nutshell we make it possible to track the results of make check
or equivalent test coverage for every source package. There's a little
work involved for setting up each package, but it gives the easy ability
to *know* and prove that between xyz ${FLAGS} there's no regressions.
For example: How do you *know* that fftw or ssl is regression free when
you enable avx, fma or some other higher level of optimization (-O3). By
knowing I don't mean just result correctness, but also potentially
runtime performance, code size and potentially compile times. (If those
are things you care about)
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 10:15 AM, C. Bergström cbergst...@pathscale.com wrote: On 01/13/14 03:43 PM, Alexander Berntsen wrote: Where I work uses pkgcore[1], but not the areas which are generally beneficial to the whole community. (We use it as part of a web application to handle testsuites which have build dependencies.) We can blah blah about performance of resolving package dependencies all day long, [...] Not sure about what you mean with blah blah. But given the amount of both disk caches (metadata, vdb cache) and memory caches (the in-memory aux_db cache that portage loads using pickle (it's a dict) takes like 70-100Mb of RAM on an average desktop system), Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Ironically, launching the same emerge command twice, will take more or less the same time. Yeah, this is probably bad design... -- Fabio Erculiani
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/13/14 04:31 PM, Fabio Erculiani wrote: On Mon, Jan 13, 2014 at 10:15 AM, C. Bergström cbergst...@pathscale.com wrote: On 01/13/14 03:43 PM, Alexander Berntsen wrote: Where I work uses pkgcore[1], but not the areas which are generally beneficial to the whole community. (We use it as part of a web application to handle testsuites which have build dependencies.) We can blah blah about performance of resolving package dependencies all day long, [...] Not sure about what you mean with blah blah. But given the amount of both disk caches (metadata, vdb cache) and memory caches (the in-memory aux_db cache that portage loads using pickle (it's a dict) takes like 70-100Mb of RAM on an average desktop system), Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Ironically, launching the same emerge command twice, will take more or less the same time. Yeah, this is probably bad design... ack - I know the benefits (and downsides) of pkgcore compared to portage, but I leave that up to others who would like to voice their opinion. It would be great to get pkgcore up to feature parity with portage, but I don't have the resources to help with that. (In the future, possibly next month, I will try to put some bounties)
[gentoo-dev] Re: [OT] pkgcore bikeshed
On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote:
On 01/13/14 03:43 PM, Alexander Berntsen wrote:
Realistically, we have to keep updating them both in parallel. pkgcore
needs to be brought up to portage-level functionality,
Yeah but it already outshines under the hood: all you're talking about is
EAPI and radhermit is working on it; I'm sure he and dol-sen would be
happy for more help as well, so long as it's supportive. ISTR TomWij is
involved too.
Updating both in parallel isn't hard: once pkgcore is up to EAPI-5, EAPI-6
isn't that much work (mostly bash afair.) At that point, put portage into
feature-freeze, and only bugfix it. Call a hiatus on new EAPIs for 6 months
and put all effort into making damn sure pkgcore is a drop-in replacement.
There's certainly enough devs to do that, and definitely enough interest
in finally moving to portage-NG.
and we need to keep fixing portage bugs for its many users.
Sure.
We could have a technical
discussion on the merits of pkgcore vs. portage, but in the end, it's
about the users.
I don't think anyone who's serious can come down on any side but pkgcore
in that debate, so I agree we skip that discussion.
You won't get any disagreement from me on your last point.
For the record, I would be very happy if you hacked on pkgcore. Just
as happy as if you hacked on portage, even. So please do. :-)
Good good :-)
We can blah blah about performance of resolving package dependencies
all day long, but it's clearly not a game changer feature for users.
(or they just don't know)
They just don't know: and it really is a game-changer, once you've tried it.
We must be able to finally deliver pkgcore speed, 5 years after the event.
Long term the API to pkgcore could be beneficial, but again I'm not sure
it's a game changer for users.
Doesn't matter: it's good to have something the devs can get hyped about too.
snakeoil is a lovely bit of code.
[1] /* Side details */
In a nutshell we make it possible to track the results of make check
or equivalent test coverage for every source package. There's a little
work involved for setting up each package, but it gives the easy ability
to *know* and prove that between xyz ${FLAGS} there's no regressions.
For example: How do you *know* that fftw or ssl is regression free when
you enable avx, fma or some other higher level of optimization (-O3). By
knowing I don't mean just result correctness, but also potentially
runtime performance, code size and potentially compile times. (If those
are things you care about)
OFC they are, or we wouldn't be using Gentoo ;) And that does sound like
an interesting project, of wider interest to the community. (hint hint;)
--
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)
Re: [gentoo-dev] Re: [OT] pkgcore bikeshed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 13/01/14 12:02, Steven J. Long wrote: Yeah but it already outshines under the hood: all you're talking about is EAPI and radhermit is working on it; I'm sure he and dol-sen would be happy for more help as well, so long as it's supportive. ISTR TomWij is involved too. I know all this. But until the EAPI is brought up to date, I don't see a lot of devs migrating their work there. (Kudos to radhermit for his work.) Furthermore, bringing the EAPI up to date is not of ignorable complexity. Updating both in parallel isn't hard: once pkgcore is up to EAPI-5, EAPI-6 isn't that much work (mostly bash afair.) If it is trivial: show us the code. At that point, put portage into feature-freeze, and only bugfix it. Call a hiatus on new EAPIs for 6 months and put all effort into making damn sure pkgcore is a drop-in replacement. I don't see this happening as it stands right now. Let's revisit this when pkgcore is more up to date. There's certainly enough devs to do that, and definitely enough interest in finally moving to portage-NG. These are two big statements, and I'm not convinced either of them holds true. Feel free to gather data on the latter. (The former is not immediately quantifiable.) - -- Alexander alexan...@plaimi.net http://plaimi.net/~alexander -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLT290ACgkQRtClrXBQc7XeuAD+Jj4Oj73br8pvcKdkDSVFfE3J tG6ajS3wzQHNmkbxWPoA+gMVyzgm5SSDRAjqHysocgvvDe6Xwa2CA+/ON98u+2xs =mBxD -END PGP SIGNATURE-
Re: Re: [gentoo-dev] Re: [OT] pkgcore bikeshed
Am Montag 13 Januar 2014, 13:28:13 schrieb Alexander Berntsen: Updating both in parallel isn't hard: once pkgcore is up to EAPI-5, EAPI-6 isn't that much work (mostly bash afair.) So far, I dont know of any work on the exact EAPI-6 feature set yet. We should maybe open a new thread on that, *if* there is already something. -- Andreas K. Huettel Gentoo Linux developer kde, council
Re: [gentoo-dev] Re: [OT] pkgcore bikeshed
On Mon, 13 Jan 2014, Andreas K Huettel wrote: So far, I dont know of any work on the exact EAPI-6 feature set yet. We should maybe open a new thread on that, *if* there is already something. I've started collecting things already some months ago: http://wiki.gentoo.org/wiki/Future_EAPI/EAPI_6_tentative_features Ulrich pgpzCtBa7Qe2A.pgp Description: PGP signature
Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead
On Mon, 13 Jan 2014 15:39:17 +0700 C. Bergström cbergst...@pathscale.com wrote: Drive-by trolling comment but I wish the effort to keep porkage alive would have instead been directed towards pkgcore. If we still have users left by the time pkgcore is finished... Moving the work efforts from one PM to another and expecting the future to go well is too optimistic, of course it would make pkgcore a great thing eventually but it can come at the cost of Gentoo itself; *danger*. While Portage's performance might have dropped a little (though is still quite fast [~30s] on --backtrack=0), the development on it didn't stop as Arfrever and few_ have been committing over the past weeks to months; there just hasn't been a release because the lead position was MIA... And before anyone pulls a pkgcore almost has EAPI 5 support comment; you should note that EAPI 6 is around the corner, and many of its would-be features have already been implemented in Portage. I'm in doubt if the same hold true for pkgcore; which was known to be dead for a while as its founder left development, and it took some time for two of our developers to pikc it up again. So, yes, we need more people on pkgcore; no, we can't just leave Portage behind, as it still is the beating heart of Gentoo for now. Who knows, as we get up to speed, Portage might even see some neat refactoring and performance improvements; or well, that's what I expect to see some of us work towards to. We're already putting measures in place to clean up the source code (short variable names, file length); but for now, it's not that huge of a problem to stop using it at all. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. Yes, a side effect of not having that is that a lot of tools would need to be rewritten too if we would make a sudden move to pkgcore; or at least, it would mean that Portage would still need to be around to serve as an API, but that would also mean it still needs to be maintained. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead
Tom Wijsman wrote: So, yes, we need more people on pkgcore; no, we can't just leave Portage behind, as it still is the beating heart of Gentoo for now. I guess the point is that it might continue beating long enough mostly on its own. //Peter pgp_E4oOCUrAm.pgp Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: At the end of the day we have one codebase which is engineered and another which has evolved. Too broad generalization, too much assumption; both can be held as meaning nothing compared to what engineered and evolved could really be, but as with doing that, it gets a subjective nature. In other words, the lack of context makes this statement meaningless. What we're looking for is a well-engineered _and_ well-evolving distribution, not just a well-engineered _or_ well-evolving package; and thus there's a lot more to it to really use this as an argument. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 13/01/14 09:46 AM, Tom Wijsman wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. ..you mean, define the API and include it in PMS? how much bureaucracy will that need to occur? -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlLT/o0ACgkQ2ugaI38ACPBIlwD/UhmM7jiu0c3kbczm/p0eA0nE QklKzRWkYxLpUpWEwegA/0GzTsFmUL/9C+xVvh1rpKjk23L6LAR/mWuiB34Vmnb0 =vpX0 -END PGP SIGNATURE-
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. Yeah, this is probably bad design... Too broad generalization with no fundamental argument about the design. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [OT] pkgcore bikeshed
On Mon, 13 Jan 2014 11:02:10 +
Steven J. Long sl...@rathaus.eclipse.co.uk wrote:
Yeah but it already outshines under the hood: all you're talking
about is EAPI and radhermit is working on it;
pkgcore's response to EAPI 6 is something to hold your breath for.
I'm sure he and dol-sen would be happy for more help as well, so long
as it's supportive. ISTR TomWij is involved too.
Yes and no, in the short run I'm going to do some repoman work as to
benefit both the QA and Portage teams (and the community as a whole);
from that point on I'll try to look at Portage and pkgcore from a broad
view (software re-engineering style) to see how I want to continue.
As stated in other mails, I'll definitely help keep Portage alive for
as long as possible regardless; I will thus have to see if remaining
time permits to help out with pkgcore as well.
So, in the short run; no, also, I'm devaway for the next two weeks.
Like Portage made out a call for people, I think pkgcore should in the
near future make out a call for more people; there definitely are some
people here and there that have deep interest in it, thus contributions
to it might just be waiting around the corner...
Updating both in parallel isn't hard: once pkgcore is up to EAPI-5,
EAPI-6 isn't that much work (mostly bash afair.)
Its feature set is unknown afaik, thus it could become easy but just
as well become huge; this depends on the approach that the PMS team
takes going forward. But for all I know I've seen some things get
implemented in Portage, is the same true for pkgcore?
At that point, put portage into feature-freeze, and only bugfix it.
Impossible with a new EAPI around the corner.
Call a hiatus on new EAPIs for 6 months and put all effort into
making damn sure pkgcore is a drop-in replacement.
This stops the progress of part of the distribution; actually, we've
stopped progress on that for quite some time already as we're already
running a bit behind on the yearly release of a new EAPI.
We could have a technical
discussion on the merits of pkgcore vs. portage, but in the end,
it's about the users.
I don't think anyone who's serious can come down on any side but
pkgcore in that debate, so I agree we skip that discussion.
I could indeed conclude the opposite, evening it out; it's pointless to
discuss that right now. We're currently discussing the PMs' futures;
but we're not discussing the new package manager default yet, neither
are we discussing a well planned proper migration yet at this time.
We can blah blah about performance of resolving package dependencies
all day long, but it's clearly not a game changer feature for users.
(or they just don't know)
They just don't know: and it really is a game-changer, once you've
tried it. We must be able to finally deliver pkgcore speed, 5 years
after the event.
With Portage only spending half a minute* a day on it here, /care; and
as seen in benchmarks, Portage allows quite some improvements to it.
So, if needed, it is certainly possible to bring its time down.
* --backtrack=0 is my new favorite parameter, don't need more. :)
[1] /* Side details */
In a nutshell we make it possible to track the results of make
check or equivalent test coverage for every source package.
There's a little work involved for setting up each package, but it
gives the easy ability to *know* and prove that between xyz
${FLAGS} there's no regressions. For example: How do you *know*
that fftw or ssl is regression free when you enable avx, fma or
some other higher level of optimization (-O3). By knowing I don't
mean just result correctness, but also potentially runtime
performance, code size and potentially compile times. (If those are
things you care about)
OFC they are, or we wouldn't be using Gentoo ;) And that does sound
like an interesting project, of wider interest to the community.
(hint hint;)
Now that I read this again:
Performance, size and compile times mean nothing if you don't get a
correct result; it all gets funny when the scrollbar of your browser
breaks just because one of its many deep dependencies was built with
-Ofast, then that small bit of extra performance can be laughed at.
OT And yeah, searching that dependency is also a lot of fun; which
means you'll want to recompile libraries with -O2 again until you have
found it, unless you want to spend days figuring it out. And for this
last thing, it's handy you can grep flags in the /var/db/pkg/. /OT
OT2 At which point you can be lucky it isn't done by some package
that has bad QA and has overridden CFLAGS or something like that. /OT2
--
With kind regards,
Tom Wijsman (TomWij)
Gentoo Developer
E-mail address : tom...@gentoo.org
GPG Public Key : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D
signature.asc
Description: PGP signature
Re: [gentoo-dev] Re: [OT] pkgcore bikeshed
On Mon, 13 Jan 2014 14:50:44 +0100 Ulrich Mueller u...@gentoo.org wrote: On Mon, 13 Jan 2014, Andreas K Huettel wrote: So far, I dont know of any work on the exact EAPI-6 feature set yet. We should maybe open a new thread on that, *if* there is already something. I've started collecting things already some months ago: http://wiki.gentoo.org/wiki/Future_EAPI/EAPI_6_tentative_features Just for reference: https://bugs.gentoo.org/show_bug.cgi?id=174380 According to [1] there are besides the tracker a 78 bugs open, of which the Gentoo Wiki page has tentatively selected a 16 bugs (till Sep 2013); which means that there are a 52 bugs unconsidered so far. I'd think that looking at that either EAPI 6 will be big, or otherwise an EAPI 7 would follow in the very near future... [1]: https://bugs.gentoo.org/buglist.cgi?quicksearch=future%20eapi -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 13 Jan 2014 09:56:13 -0500 Ian Stakenvicius a...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 13/01/14 09:46 AM, Tom Wijsman wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. ..you mean, define the API and include it in PMS? how much bureaucracy will that need to occur? The API can be separate of the PMS. The bureaucracy needed for an API to provide to tools, that doesn't need as much involvement as PMS. - -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJS1AbYAAoJEJWyH81tNOV90DEIAIk+zQe4vgJ2XWchQLET064Q mij/sQQRcI52PZwXFfRh23G+dKkywsPIg+US5KmbWJTtT4UbbLFnn4KCRB/QZE6E 43wHYsKGMgTovpkXQZEy8O4GbFUOb1k9DrsR05r91+QUUUmxxxi9ODq86kavtPVs UaPyz3me4j3olHLSxzUtxJd5/AgHGwkH+YHtcWny2oaXNUIFpusIPRURqyFyE4xI Leq/w8/nsmeCc25pPN4cbY6bBwOwCxsZZwik3gmz6FzrAVwCjuRvX8n7szl8viNr Kd56odFjX/n7pAL+Rj52spz6TNcSe1H8IMkwI9rMYc/LYQSw+/uCGnAXeIxrzWY= =F2d3 -END PGP SIGNATURE-
Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead
On Mon, 13 Jan 2014 15:46:12 +0100 Peter Stuge pe...@stuge.se wrote: Tom Wijsman wrote: So, yes, we need more people on pkgcore; no, we can't just leave Portage behind, as it still is the beating heart of Gentoo for now. I guess the point is that it might continue beating long enough mostly on its own. How certain are you of this guess? How long is long enough? Mostly? At the moment, it beats mostly on our efforts to it; and in the near future, it'll need to adapt further to newer EAPIs as well as further bugs and other complexities in the Portage trees. Otherwise you'll get a lot of we can't continue because they stopped developing Portage. Hence that's why it comes up every now and then as a subject, hence someone finally made the call to reinforcement, ...; so, it won't have to continue beating on its own at all. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. I guess he means that the (according to the file sizes) extensive caching doesn't seem to be of much use. -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:38:59 +0100 Luis Ressel ara...@aixah.de wrote: On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Can you give an example of that? Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. In any case, I've not been experiencing problems with this; but am interesting to know how this could go wrong, as no backtracking works for me and I hope it will continue to do so. Btw, this also makes me question how proper pkgcore can do rebuilds. Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. I guess he means that the (according to the file sizes) extensive caching doesn't seem to be of much use. For that, _all_ caching would need to be removed before the first run; when there is a mention of the same, I doubt this was done at all. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead
On Mon, Jan 13, 2014 at 7:38 AM, Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 15:46:12 +0100 Peter Stuge pe...@stuge.se wrote: Tom Wijsman wrote: So, yes, we need more people on pkgcore; no, we can't just leave Portage behind, as it still is the beating heart of Gentoo for now. I guess the point is that it might continue beating long enough mostly on its own. How certain are you of this guess? How long is long enough? Mostly? At the moment, it beats mostly on our efforts to it; and in the near future, it'll need to adapt further to newer EAPIs as well as further bugs and other complexities in the Portage trees. Otherwise you'll get a lot of we can't continue because they stopped developing Portage. Hence that's why it comes up every now and then as a subject, hence someone finally made the call to reinforcement, ...; so, it won't have to continue beating on its own at all. As usual, it is nigh impossible to force people to work on a specific project. I would encourage radhermit to ask for assistance if he thinks he needs it. dol-sen asked because he is working on a bunch of projects and knows he cannot maintain portage by himself. Even if the council were to decree that portage be frozen, I doubt that would actually halt contributions. -A -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 7:38 AM, Luis Ressel ara...@aixah.de wrote: On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. I guess he means that the (according to the file sizes) extensive caching doesn't seem to be of much use. The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. That being said, for most users that use rsync, the metadata is pre-generated. As long as they are not changing the cache indicators (not sure if this is mtime or md5sum these days) they should be seeing a benefit. This is not meant to imply that portage is always fast; there are plenty of other modules (such as the aforementioned backtracking) that can take a long time to find solutions. That is partially why you see Tomwij turning off that feature. It is helpful for users cause it can automatically find solutions for users that are otherwise unsolvable (and thus avoids the user having to find a solution to the depgraph manually.) -A -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:46:08 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 16:38:59 +0100 Luis Ressel ara...@aixah.de wrote: On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Can you give an example of that? Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. No, the problem wasn't that rebuilds weren't done (btw: this is not about @preserved-rebuilds, but about subslot dependencies), but that updates which would trigger such rebuilds are silently ignored. This happened to me yesterday while trying --backtrack=0. The available update to dev-haskell/parsec simply didn't show up (haskell ebuilds make heavy use of subslot deps), I only noticed this because I knew there was in fact an update available (thanks to eix-diff). Only after enabling backtracking Portage found the update. This might well be a bug, perhaps I'll examine the situation when I've got more time. -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 5:49 PM, Alec Warner anta...@gentoo.org wrote: [...] This is not meant to imply that portage is always fast; there are plenty of other modules (such as the aforementioned backtracking) that can take a long time to find solutions. That is partially why you see Tomwij turning off that feature. It is helpful for users cause it can automatically find solutions for users that are otherwise unsolvable (and thus avoids the user having to find a solution to the depgraph manually.) Yeah, correct. But it would be nice if Portage backtrack_depgraph() would memoize (asynchronously serializing to disk, for instance) partial results for later use. AFAIR, last time I checked, it wasn't doing that. Also, it would be nice if the aux_db cache of the vdb could be stored in a sqlite3 db rather than in RAM. This dict is consuming a huge cut of memory for little reason (a single vdb.dbapi.match() can eat 100MB of RAM). We know how Python deals with freed memory... Sigh... -A -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD -- Fabio Erculiani
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote: At the end of the day we have one codebase which is engineered and another which has evolved. I'll take an evolved codebase over engineered anyday. You do realize that is exactly why Linux has succeeded, right? The kernel has evolved, and was never engineered. There's lots people should be learning from biology... So you are using the benifits of evolution right now on your system, don't knock it, it's proven to work. greg k-h
Re: [gentoo-dev] [OT] pkgcore bikeshed
On Mon, 13 Jan 2014, Tom Wijsman wrote: I've started collecting things already some months ago: http://wiki.gentoo.org/wiki/Future_EAPI/EAPI_6_tentative_features Just for reference: https://bugs.gentoo.org/show_bug.cgi?id=174380 According to [1] there are besides the tracker a 78 bugs open, of which the Gentoo Wiki page has tentatively selected a 16 bugs (till Sep 2013); which means that there are a 52 bugs unconsidered so far. I'm well aware of this. The features listed at the wiki page are the ones that a) have been discussed, e.g. in the -dev ML, and where (by my subjective judgement) there is consensus that they belong in the next EAPI, b) where we have at least a rough idea how the specification should look like, and c) that can be implemented in finite time. Feel free to lobby for any of the 52 additional features if you believe that it is missing. (Preferably, patches for PMS and for Portage should be provided. ;-) I'd think that looking at that either EAPI 6 will be big, or otherwise an EAPI 7 would follow in the very near future... [1]: https://bugs.gentoo.org/buglist.cgi?quicksearch=future%20eapi Ulrich pgprYfjlFROdg.pgp Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Tue, Jan 14, 2014 at 12:42:00AM +0700, C. Bergström wrote: On 01/14/14 12:37 AM, Greg KH wrote: On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote: At the end of the day we have one codebase which is engineered and another which has evolved. I'll take an evolved codebase over engineered anyday. You do realize that is exactly why Linux has succeeded, right? The kernel has evolved, and was never engineered. There's lots people should be learning from biology... So you are using the benifits of evolution right now on your system, don't knock it, it's proven to work. I'll bite - While I don't think nature stopped to properly design interfaces along the way. I bet you Linus wouldn't agree with your comment very much. I don't think you have been paying attention much, I'm directly quoting Linus: Linux is evolution, not intelligent design - Linus There are many more statements exactly like this from him over the years, do a bit of research to dig them up. 1) I expect quite a bit of time has gone into (Solaris and Linux) kernel interfaces Time doesn't mean they haven't evolved. 2) Any larger or invasive changes require quite a bit of planning, review and testing. (Possibly with tests (public/private) to cover a large amount of the new/existing feature And we always get it wrong, so they evolve into something that later on works. Seriously, this is how the very system you are using has been created, it's a well-documented fact (look at our changelogs for details.) sorry, greg k-h
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/14/14 12:37 AM, Greg KH wrote: On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote: At the end of the day we have one codebase which is engineered and another which has evolved. I'll take an evolved codebase over engineered anyday. You do realize that is exactly why Linux has succeeded, right? The kernel has evolved, and was never engineered. There's lots people should be learning from biology... So you are using the benifits of evolution right now on your system, don't knock it, it's proven to work. I'll bite - While I don't think nature stopped to properly design interfaces along the way. I bet you Linus wouldn't agree with your comment very much. 1) I expect quite a bit of time has gone into (Solaris and Linux) kernel interfaces 2) Any larger or invasive changes require quite a bit of planning, review and testing. (Possibly with tests (public/private) to cover a large amount of the new/existing feature To clarify - by engineering I mean significant forethought into the design and implementation. I'm not trying to take away the achievements of any particular piece of software. In this case it just one of those you know it when you see it. If I wasn't clear or you don't understand what I meant - that's ok because I meant no offense to anyone.
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:46:08 +0100 Tom Wijsman tom...@gentoo.org wrote: Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. There used to be a feature whereby if resolution took too long, you'd get an incomplete answer. The second time you ran the resolution, the time could differ, so you'd get a different incomplete answer. Dunno if this has been fixed yet. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 2014-01-13 at 15:46 +0100, Tom Wijsman wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. I have been working towards that for years, but, things keep getting in the way. By things, I mean other project work that needs to be done as well. I got started in all this working on porthole, which uses portage's api for much of it's information. To get more features, information, I wanted to be able to incorporate some of gentoolkit's info. Namely equery, but most of it's working code was embedded with it's output. I worked hard at separating out it's working code from it's output so it would be usable by other tools. I have also been working on making a pkgcore backend for porthole. In doing that it required making a different backend for portage to get some uniformity for the frontend. I have had to put development of those on hold, due to taking over layman's development. I gave it a new high level api, modified it's mid level code and gave it a nice api that can be used by other apps, guis, etc.. One of the things that came up with layman was to be able to gpg enable it to verify it's repositories.xml list it downloads. I did that. In so doing created dev-python/pyGPG a python lib, which has now brought me in to developing Gentoo-keys (another project that could use help) to manage gentoo's gpg release, developer keys, and verification. Also at the same time a year ago, there was a lot of talk about moving the default location of the gentoo tree, but it could not be done with current catalyst code. So offered to help out as that project was severely lacking people with decent python skills. That code base is like what portage code was 8 years ago. And if you thought todays portage code is bad... you would cringe to see it's code from 8 years ago. Now portage was in trouble, while there were some people stepping in to fix things, I stepped up to help drive out a new release. I am now interim lead till we hold an election. So most of my time now is spent steering projects, more than coding. Hey, it's all work that has to be done. So I'm putting out fires where it needs it most. With that aside. One of the biggest hurdles new developers face with pkgcore is the lack of decent api docs and flow charts. Brian was brilliant and OCD about attaining speed, but at the cost of difficulty in following the code and creating the steep learning curve. I have been trying to get these other projects to a point I could create the docs and charts needed so that it would be easier for new developers to find their way in pkgcores code. That is when pkgcore will make more headway at becoming portage's replacement. But some new fires keep popping up. Long story in a nutshell, gentoo could use more GOOD firefighters. Sorry for the long speech ;) your friendly gentoo python firefighter -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 15:46:59 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. Not an API. APIs are bad. What we should have is a good set of lightweight Unix-friendly command line tools. See, for example, the Scripting Commands section of man cave. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:03:31 +0100 Luis Ressel ara...@aixah.de wrote: No, the problem wasn't that rebuilds weren't done (btw: this is not about @preserved-rebuilds, but about subslot dependencies), but that updates which would trigger such rebuilds are silently ignored. This happened to me yesterday while trying --backtrack=0. The available update to dev-haskell/parsec simply didn't show up (haskell ebuilds make heavy use of subslot deps), I only noticed this because I knew there was in fact an update available (thanks to eix-diff). Only after enabling backtracking Portage found the update. This might well be a bug, perhaps I'll examine the situation when I've got more time. That description is broad, it sounds like you didn't pass -uD to emerge; if it is part of the dependency graph (it should, otherwise this discussion would be useless) and is an update (it is, due to eix-diff), then with -uD it would pull that in. This even doesn't have to do anything with sub-slot dependencies as far as I know; because why would it check the sub-slot dependencies if it were an upgrade, the upgrade takes precedence over the sub-slot dependencies. The other option is that I am confused about this; as you can see, given the possible scenarios that could happen I might have a different one on mind than you do. If backtracking doesn't get to it, it would have to list it as a conflict; if it doesn't do that at the moment, then we certainly have a bug somewhere I think. Thank you in advance for investigating this. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 08:49:17 -0800 Alec Warner anta...@gentoo.org wrote: The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. It sounds crazy to use gentoo-x86 without a cache; to some extent, the earlier mentions on this ML by Ciaran that we should improve input would make more sense if would want to run without a cache. But I feel like this shouldn't be a topic of discussion unless we intent 'works fast on a plain CV checkout' to be an actual feature. I haven't checked, but doesn't Portage in that case build its own cache? It is helpful for users cause it can automatically find solutions for users that are otherwise unsolvable (and thus avoids the user having to find a solution to the depgraph manually.) If we document how to process --depgraph=0 output, I believe it could just as well be used; but yes, it basically involves having to run a few manual upgrades (emerge -1 ...) and/or masks to get going. But you can do those things in seconds rather than to wait a few minutes... Some of these Portage could even do automatically regradless, like for instance the 'no parents' slot conflicts; in which case I think you can just always upgrade to the newer version, instead of outputting them it put them as upgrades at the end of the emerge list I think. But I haven't investigated that further, I'm just getting tired of seeing 'em. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:05:21 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Mon, 13 Jan 2014 16:46:08 +0100 Tom Wijsman tom...@gentoo.org wrote: Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. There used to be a feature whereby if resolution took too long, you'd get an incomplete answer. The second time you ran the resolution, the time could differ, so you'd get a different incomplete answer. Dunno if this has been fixed yet. We should document what Portage does; but from the word 'backtrack' I think it tries with what it is given, and if it doesn't succeed it goes one step back and tries to do it with more. But how exactly that is implemented, I think only people like Zac and Arfrever can tell us; or we read through the relevant code ourselves. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 19:16:45 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 08:49:17 -0800 Alec Warner anta...@gentoo.org wrote: The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. It sounds crazy to use gentoo-x86 without a cache; to some extent, the earlier mentions on this ML by Ciaran that we should improve input would make more sense if would want to run without a cache. But I feel like this shouldn't be a topic of discussion unless we intent 'works fast on a plain CV checkout' to be an actual feature. I think you're misunderstanding that comment. It's not just the file formats that are bad. It's also what data is provided. Two examples are the massive lengths we have to go to to figure out what to do with || dependencies (thanks to the horrific abuses commonly used in the tree...), and the well-documented REQUIRED_USE screwup. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:07:39 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Mon, 13 Jan 2014 15:46:59 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. Not an API. APIs are bad. What we should have is a good set of lightweight Unix-friendly command line tools. See, for example, the Scripting Commands section of man cave. It still is an API that way, just expressed differently; if you would only do this you're introducing forks where you might not need them. Providing shell commands is one possible binding to the API... -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:21:58 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Mon, 13 Jan 2014 19:16:45 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 08:49:17 -0800 Alec Warner anta...@gentoo.org wrote: The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. It sounds crazy to use gentoo-x86 without a cache; to some extent, the earlier mentions on this ML by Ciaran that we should improve input would make more sense if would want to run without a cache. But I feel like this shouldn't be a topic of discussion unless we intent 'works fast on a plain CV checkout' to be an actual feature. I think you're misunderstanding that comment. It's not just the file formats that are bad. It's also what data is provided. Two examples are the massive lengths we have to go to to figure out what to do with || dependencies (thanks to the horrific abuses commonly used in the tree...), and the well-documented REQUIRED_USE screwup. Iotw, 'input'; but thank you for clarifying. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 19:27:36 +0100 Tom Wijsman tom...@gentoo.org wrote: Not an API. APIs are bad. What we should have is a good set of lightweight Unix-friendly command line tools. See, for example, the Scripting Commands section of man cave. It still is an API that way, just expressed differently The term API seems to mean library or communication interface these days (see Git API and the like). So for clarity, it's probably better to make the distinction. if you would only do this you're introducing forks where you might not need them. Providing shell commands is one possible binding to the API... Forks are far less of a big deal than trying to share a process between programming languages. This is a *huge* deal when trying to integrate modern C++ with languages with primitive thread-hostile runtimes like Python and Ruby. Unless you're doing a heavy GUI (in which case portability probably isn't going to cut it anyway, without a far larger API than it's worth providing), most of the cost for most of the use cases for this is in the PM reading in configs and the like. The way around that is to allow communication via pipes, still in a Unix-friendly manner. This can be implemented nearly transparently. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [OT] pkgcore bikeshed
On 11:02 Mon 13 Jan , Steven J. Long wrote: On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote: Realistically, we have to keep updating them both in parallel. pkgcore needs to be brought up to portage-level functionality, Yeah but it already outshines under the hood: all you're talking about is EAPI and radhermit is working on it; I'm sure he and dol-sen would be happy for more help as well, so long as it's supportive. ISTR TomWij is involved too. Working on the UX of the emerge frontend is also of extreme importance if you want people actually using it. Speed is absolutely a usability feature but it's not the only one that matters. Maintaining EAPI parity is table stakes. -- Thanks, Donnie Donnie Berkholz Council Member / Sr. Developer, Gentoo Linux http://dberkholz.com Analyst, RedMonk http://redmonk.com/dberkholz/ pgpQJ5K5w1XJT.pgp Description: PGP signature
Re: [gentoo-dev] Default USE changes for fortran and mudflap?
On 01:53 Sun 12 Jan , Ryan Hill wrote: fortran: Do we want to keep enabling fortran by default? The majority of users will never get the urge to install a fortran package, and the fortran eclass handles those that do. I think it should be treated as all the other optional languages and disabled by default, but I'd like to know if there are other opinions. It's essentially only used by the small minority of people installing sci-* packages so I'd be fine with that. Reasonable defaults FTW. -- Thanks, Donnie Donnie Berkholz Council Member / Sr. Developer, Gentoo Linux http://dberkholz.com Analyst, RedMonk http://redmonk.com/dberkholz/ pgpKSLofMizut.pgp Description: PGP signature
Re: [gentoo-dev] Re: RFC: storing predefined INSTALL_MASK directory lists in repos
On 19:59 Sat 11 Jan , Peter Stuge wrote: Duncan wrote: Michał Górny wrote: INSTALL_MASK=systemd bash-completion What are your thoughts? It seems like this will generally duplicate all -USE flags. Would it make sense to instead have a single setting which changes the meaning of USE to be that everything not USEd is install-masked? No, this would not be a duplicate. I did generalize, but think more about this - certainly for both Michał's examples I have already either set or unset systemd and bash-completion in USE. It would largely be a duplicate, since we've already made a decision about whether the ability to build without a feature is important enough to merit a USE flag. This is essentially rethinking the same decision and adding complexity to it. I think having this as an additional feature in the core PM would be confusing to users. It would probably be a better fit in gentoolkit or a similar tool. -- Thanks, Donnie Donnie Berkholz Council Member / Sr. Developer, Gentoo Linux http://dberkholz.com Analyst, RedMonk http://redmonk.com/dberkholz/ pgpcJ3C4i8c3x.pgp Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/13/2014 10:58 PM, Tom Wijsman wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Or if you don't have a supercharged turbo-CPU ... no joke ... Half an hour. Yeah, on an E-350 it's GREAT FUN to WAIT for portage ;) It took me three days of frustration just to figure out an upgrade path.
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Tue, 14 Jan 2014 07:22:25 +0800 Patrick Lauer patr...@gentoo.org wrote: On 01/13/2014 10:58 PM, Tom Wijsman wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Or if you don't have a supercharged turbo-CPU ... no joke ... Half an hour. Well, in your case disabling it thus leads to a quite huge improvement. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Default USE changes for fortran and mudflap?
On Sunday 12 January 2014 02:53:47 Ryan Hill wrote: While I'm adding USE defaults to toolchain.eclass and moving them out of the profiles, I thought now would be a good time to review a couple default flag settings. mudflap: This is currently enabled by default but I'd like to disable it. It controls libmudflap and the -fmudflap flag. I think the only reason this flag exists is so we can disable it in crossdev. It's not required by anything in the tree, the code is bitrotten and has been removed for GCC 4.9. If you know how to use -fmudflap, you know how to set a USE flag. it was the hot new checker when it was first added, but it never really caught on and asan is the way to go now. i occurred to me a few weeks ago to turn it off by default after watching another cross-compile go by with it on, but looks like you got it covered. -mike signature.asc Description: This is a digitally signed message part.
[gentoo-dev] pkgcore EAPI-6 (Was: OT: pkgcore bikeshed)
On Mon, Jan 13, 2014, Alexander Berntsen wrote: Updating both in parallel isn't hard: once pkgcore is up to EAPI-5, EAPI-6 isn't that much work (mostly bash afair.) If it is trivial: show us the code. Ah that old canard. Tell you what: I hereby undertake to deliver everything currently listed at [1] for pkgcore, once radhermit has declared he's done on his round of work with pkgcore internals and it's up to EAPI-5, as tested against ebuilds, ferringb has approved it, and its portage formatter is good enough for update[2] to use it again, on EAPI-5 ebuilds. Sorry the latter is why I care, and I've always pushed for it, so whether that annoys anyone else is not my problem. I will be involved in testing that aspect, obviously, and may need to patch update for other interactions with pmerge, so that will come first. I will deliver that within a calendar month, or 30 days, of the above happening, for QA and acceptance-testing, unless ferringb, radhermit or dol-sen veto it, preferably in #pkgcore but see disclaimer at end. To avoid confusion, in case more is added, I'll list them, so you can hold me to my word later: bash 4.2 #431340 nonfatal die #451938 -- truly lame to do this via a new func, imo reads like working round the PMS, which /is/ insane. static libs in /usr is nothing on that, when you think about it. get_libdir #463586 Used in econf, but so far not available as separate PM function Doc install function (edocs) #459692 Allow DOCS= #463736 Directory support for DOCS #481980 Query function for IUSE #449862 in addition, query IUSE_EFFECTIVE -- from [3] this is what should be queried? PATCHES support in default src_prepare #463692 Unpack .txz #458102 Case-fold extensions in unpack #476730 unpack() accept absolute paths #483244 epatch_user - a sane implementation as we discussed 2 years ago or so Source eclasses only once #422533 -- amended at [4] EJOBS #273101 - emake integration -- for other languages or build- systems, the eclasses would have to handle it. They are not part of the PM/ebuild.sh, so out of scope. (There is a qm as to whether this is still desired 4 years later.) non-bash, but i'll provide support script as and if req'd: HDEPEND #317337 -- CHOST dependencies req at build-time (yaf way of adding a borked LDEPEND if you ask me, but w/e.) I won't do failglob (#463822) as it will make everyone's life a lot harder in the long run, and turn off anyone who actually has real BASH experience. AFAIC we want them in the same way as we want expert pythonistas. I'll happily add Kent's idea though, at least for this setting; he's a guy you should all listen to a lot more, imnsho. I'd do it like this: failglob will apply to the function it appears in, and any callee functions, and must be paired with reglob (or w/e) to restore, which will also restore a nullglob invocation. (reglob must be called on all return paths before function return, if you're doing something complex. Most functions won't be though, ime.) If ferringb and radhermit are ok with it, I'll implement it. If not, it'll be trivial to import the crappy version from portage. At that point, put portage into feature-freeze, and only bugfix it. Call a hiatus on new EAPIs for 6 months and put all effort into making damn sure pkgcore is a drop-in replacement. I don't see this happening as it stands right now. Let's revisit this when pkgcore is more up to date. Yes, at that point. Now you may think I'm talking before I can implement, what I accused someone else of; feel free to. Those are mostly trivial to a BASH scripter, and I write (and throw away) more complex things practically every day. See [2] if you need verification, though it's not a work-project any more, and was never meant to be when I started it as a training assignment. We do use it at work though, so I get to maintain it. And there is no way we could have done the ABI /etc/warning stuff, if we hadn't had pkgcore to test resolves with. portage was just too slow, when we were paired and needed to flow, to implement quite major stuff in hindsight; ferringb even mentioned a need for exactly that sort of mechanism in Gentoo, on the list last feb/mar i think it was, when I was on hiatus. I remember being annoyed I'd missed the discussion when I got back to FLOSS stuff. Porcelain and plumbing. Feel free to step up and pitch in too. The more the merrier, and the lighter the load. Note there are queries on some of the above, and no clear definition of what EAPI 6 is yet. So once you've made your minds up about that, then you can start asking for your users to implement it, I'd think. After all, devs are users too, and we've all got other things to do. TTFN, steveL. Disclaimer: I don't speak for pkgcore, or any of the people mentioned, nor have I discussed any of this with them, so it's their call, now or when they get to that stage. I'm just
[gentoo-dev] Re: [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014, Ciaran McCreesh wrote: On Mon, 13 Jan 2014 19:27:36 +0100 Tom Wijsman tom...@gentoo.org wrote: Not an API. APIs are bad. What we should have is a good set of lightweight Unix-friendly command line tools. See, for example, the Scripting Commands section of man cave. It still is an API that way, just expressed differently The term API seems to mean library or communication interface these days (see Git API and the like). So for clarity, it's probably better to make the distinction. I think Tom was aware of that: hence expressed differently. You appear to have missed his point, which was not about something so simple and well-known, judging by your inability to perceive it, and the state of modern software. if you would only do this you're introducing forks where you might not need them. Providing shell commands is one possible binding to the API... Forks are far less of a big deal than trying to share a process between programming languages. This is a *huge* deal when trying to integrate modern C++ with languages with primitive thread-hostile runtimes like Python and Ruby. Unless you're doing a heavy GUI (in which case portability probably isn't going to cut it anyway, without a far larger API than it's worth providing), most of the cost for most of the use cases for this is in the PM reading in configs and the like. The way around that is to allow communication via pipes, still in a Unix-friendly manner. This can be implemented nearly transparently. Yes and the traditional Unix way to implement those shell commands is to write a lib and then some apps as thin wrappers on top of it, and possibly more complex ones on top of that as time goes by, and users require. These components can in turn be used by other apps, or their crappy non-C runtimes, as a basis for future development. So Tom's point stands. As for modern C++ being the be-all and end-all that sets the exemplar .. LMAO. Good one. 30 years to reimplement LISP badly. Greenspun^N I won't bother with your assertions about GUIs etc. Just don't presume silence for agreement, as you are so wont to do. The obvious truths about pipes etc, while true have very little to do with the point being made. Still, glad you finally got hold of APUE. *wipes tears from eyes and moves on* -- #friendly-coders -- We're friendly, but we're not /that/ friendly ;-)
Re: [gentoo-portage-dev] [PATCH] Check for and report read-only filesystems
On Friday 10 January 2014 22:07:52 Chris Reffett wrote:
Attached is a patch to test if Portage is going to write to a
read-only filesystem and print out the list of filesystems that need
to be remounted RW. This leaves ${D} intact rather than having some
files moved before hitting the RO filesystem. Fixes bug 378869. Since
git.overlays.gentoo.org is down, I haven't had the chance to rebase
this against latest, but I can resubmit if it doesn't cleanly apply.
This is my first patch to the list, so I apologize if I didn't submit
correctly.
please use `git send-email` to post patches. attaching them makes things
harder to review.
--- /dev/null
+++ b/pym/portage/util/checkwriteable.py
+ with open(/proc/mounts) as procmounts:
this won't work on non-Linux systems
also, as a general style thing, unless there's a real need for the var to have
a full name, just use f or fp
+ roregex = re.compile(r'(\A|,)ro(\Z|,)')
+ for line in procmounts:
+ if roregex.search(line.split( )[3].strip())
is not None:
+ romount = line.split( )[1].strip()
+ ro_filesystems.add(romount)
ad hoc parsing of /proc/mounts isn't a good thing. at the risk of over
engineering, there should be a func that would take care of expanding the
mount paths into a list of namedcollections (one per mount). then you simply
walk it looking at its options.
+ for directory in dir_list:
+ for filesystem in ro_filesystems:
+ if re.match(filesystem, directory):
+ ro_filesystems_written.add(filesystem)
there's no need to use re and in fact you don't want to. you're looking at
wrong strings here, not regexes.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] First draft of a portage pylint.
people have been using pyflakes of late. not that i'm against adding this.
we should have a wrapper script:
$ cat pylint
#!/bin/sh
exec pylint --rcfile ${0%/*}/pylintrc $@
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] [REV] Added support for variable expansion in source command. wrt bug #490896
cnf/make.conf.example.source_test| 7 +++ cnf/make.conf.example.source_test_after | 8 i don't think test files belong in cnf/. keep them in the same dir as the test that uses them. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1
On Monday 13 January 2014 19:08:30 creff...@gentoo.org wrote: From: Chris Reffett creff...@gentoo.org you might want to set ~/.gitconfig to have your name/email so that git sendemail uses the right info in the actual e-mail From: field + undefined_phases_re = re.compile(r'^\s*(src_configure|src_prepare)\s*\(\)') you use re.match, so technically the ^ is redundant i'd suggest merging the regex a little: r'^\ssrc_(configure|prepare)\s*\(\)' could check for pkg_pretend/pkg_info too (it's new to EAPI 4) the regex is naive and can match valid ebuilds. consider ones that handle $EAPI itself and will call the right funcs as necessary. can you check the tree to see if anyone is doing this ? if so, we should drop a note to the dev list and get them fixed up. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1
On Mon, 13 Jan 2014 19:50:44 -0500
Mike Frysinger vap...@gentoo.org wrote:
On Monday 13 January 2014 19:08:30 creff...@gentoo.org wrote:
From: Chris Reffett creff...@gentoo.org
Subject: [gentoo-portage-dev] [PATCH] Add repoman check to warn if
src_prepare/src_configure are used in EAPI 0/1
Well, I was planning to cover these bugs; seems we really need to
organize with regards to avoiding double work, as two others have got
the same happen to them. Let's synchronize it.
So, what you have done here is wrote a partial patch for this bug:
https://bugs.gentoo.org/show_bug.cgi?id=379491
Therefore I suggest you that you mention it in the commit subject.
Now that you're working on phases, I wonder whether you plan to deal
with the other bug as well:
https://bugs.gentoo.org/show_bug.cgi?id=365691
While vapier's case could be considered as valid; I think we should
consider that as a bad practice, as one could just as well put the if
inside a phase which is the much more common practice.
Can you let me know whether you would work on this one as well as to
avoid both of us doing the same work on these?
As a note, my WIP work is at the next url; there you can see the bug
numbers I've been working on (still need to revise them, properly
document the added check errors [eg. in `man repoman`] and so on).
https://github.com/TomWij/gentoo-portage-next/branches
If you do something similar, we could check whether we attempt to work
on the same bug; as to avoid clashing. (In this case I think I told
you in #gentoo-qa that I planned to do this; but well, then I can be
lucky that I've not started on it. ^^)
you might want to set ~/.gitconfig to have your name/email so that
git sendemail uses the right info in the actual e-mail From: field
Then what is the From: field that it puts separately into the e-mail
itself? Seems quite odd for these to differ; or, is the latter an
attempt to mark the author? Then why does the e-mail From: field need
to be correct? Afaik it would only use one of both, so only one of both
needs to be correct. But that would make the actual question: Which one?
+ undefined_phases_re =
re.compile(r'^\s*(src_configure|src_prepare)\s*\(\)')
you use re.match, so technically the ^ is redundant
Indeed, they match on a line basis from the beginning of the string.
i'd suggest merging the regex a little:
r'^\ssrc_(configure|prepare)\s*\(\)'
Note that the * behind the first \s was lost; so, with the redundant ^
removed and the * put back it would be:
r'\s*src_(configure|prepare)\s*\(\)'
You can then proceed further and move the re outside:
r'\s*src_(configu|prepa)re\s*\(\)'
The closing parentheses can also be removed because phase( ) is valid:
r'\s*src_(configu|prepa)re\s*\('
(We don't need to account for phase( invalid ) here; as that results
in invalid Bash syntax, which would bail out harder than our checks)
But if we would do that, it would be like:
r'\s*src_(configu|prepa)re\s*\(\s*\)'
could check for pkg_pretend/pkg_info too (it's new to EAPI 4)
+1
the regex is naive and can match valid ebuilds. consider ones that
handle $EAPI itself and will call the right funcs as necessary.
From a QA point of view it seems more preferable to move away from old
EAPIs, than to conditionally support them. The common case is that
ebuilds move from older to newer EAPI and thus would get these
functions as they get to a newer EAPI.
Is there an actual case for downgrading back to an older EAPI?
If not, conditional code that checks $EAPI has no purpose.
--
With kind regards,
Tom Wijsman (TomWij)
Gentoo Developer
E-mail address : tom...@gentoo.org
GPG Public Key : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v3] Added support for variable expansion in source command argument in make.conf
seems reasonable, although i haven't used the shlex module myself before -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1
On Mon, 2014-01-13 at 23:59 -0500, Mike Frysinger wrote: On Monday 13 January 2014 22:23:01 Tom Wijsman wrote: On Mon, 13 Jan 2014 19:50:44 -0500 Mike Frysinger wrote: On Monday 13 January 2014 19:08:30 creff...@gentoo.org wrote: From: Chris Reffett creff...@gentoo.org Subject: [gentoo-portage-dev] [PATCH] Add repoman check to warn if src_prepare/src_configure are used in EAPI 0/1 Well, I was planning to cover these bugs; seems we really need to organize with regards to avoiding double work, as two others have got the same happen to them. Let's synchronize it. we have a bugzilla workflow doc posted which we'll merge once git is back up. please do not reassign portage bugs to yourself. instead, set the status to INPROGRESS. Sorry, that was me that told him to. I didn't know about a bugzilla workflow doc. Plus I didn't see it at www.gentoo.org. So it wasn't converted to the wiki. -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
