Re: [gentoo-dev] [EAPI 7] Cross-compile improvements (BDEPEND, BROOT, sysroot)
On Tue, 2015-12-01 at 22:58 +, James Le Cuirot wrote: > Sorry [snip] > gentle! :) Not sure I have anything meaningful to contribute to the discussion from a technical perspective at the moment, but I just wanted to voice my support for what you're doing with cross: I think Gentoo is setting some kind of standard -- or at least exerting a huge influence -- on the worldwide group-think about what is possible and/or reasonable to expect in cross-buildling frameworks and environments. Unfortunately, the "standard" we are setting could clearly be higher. Part of that comes down to a lack of manpower and testing at the package level, which is not an easy problem to solve, as it arises from the fundamentals of the market for open-source hacking labor. But what could be solved more readily is that the framework we have today conflates some entities and concepts which are unified when building locally, but divergent when cross-building. Unravelling those conflations should help everyone to experience less cognitive dissonance and/or head-scratching-befuddlement, which, in turn, could lead to a more productive, fun and easy-to-use cross- building experience for users and developers alike, without expecting any mysterious motivational forces to materialize from nowhere to rescue us from the desire to watch TV or go on dates. Anyhow, I say it's easier, but I know it's still pretty damn hard. So, thanks for working on this! -gmt
Re: [gentoo-dev] impending c++11 clusterfuck?
On Wed, 2015-12-02 at 08:06 +0100, Michał Górny wrote:
> On Tue, 01 Dec 2015 18:12:17 -0800
> "Gregory M. Turner" wrote:
>
> > On Tue, 2015-12-01 at 07:18 -0500, Anthony G. Basile wrote:
> > > So on that bug we're talking about selectively
> > > adding -std=c++11 (or possibly gnu++11) to packages.
> >
> > Yes, this is the biggest real-world problem we face. It requires
> > an
> > immediate solution in the ~* branches; the affected ebuilds just
> > dump a
> > bunch of gcc gobbldeygook and crash.
> >
> > If I understand the generalized problem we are facing there, as a
> > package gets "c++11-ized", all of its reverse-BDEPs find themselves
> > in
> > the following situation:
> >
> > Imagine cat/foo is a library. cat/foo-3.0 is the newest non-c++11-
> > ized
> > version (in its slot, perhaps). Now cat/foo-3.1 comes along and
> > from
> > then on, anything BDEPENDing on it must be built with -std=c++11
> > (I'm
> > ignoring the c++11 vs. gnu++11 but presumably we'll eventually need
> > to
> > figure out some kind of game plan about that).
>
> Just to be clear, this only happened because upstream started to use
> C++11 features in API. Normally libraries don't go that far, or use
> #ifs to support C++03.
>
> > So, now, "everything" pulling in headers from cat/foo finds itself
> > in
> > this situation:
> >
> > o if it pulls in headers from > o if it pulls in headers from >=cat/foo-3.1, we must add a CFLAG
> >
> > I guess the reasonable way to achieve that sort of behavior is
> > pkgconfig (which is not really a rock-solid solution. First, some
> > packages might fail to put it in there (but, OK we just add it
> > ourselves, let's say). Second, how many of cat/foo's reverse BDEPs
> > side-step pkgconfig? Sometimes this is pretty common, I'm afraid.
> >
>
> pkg-config is not a solution. -std= flags don't belong there.
> Furthermore, if you put -std= flags there, it is likely to override
> ${CXXFLAGS} and in particular make it really painful to force other
> standard on ebuild's end.
I see what you mean. Maybe pkgconfig needs a new feature? I wonder if
they are already thinking about it in pkgconfig-land. I'll look around
on their ml/bugdb and see if I can find anything.
> > So, let's say lots of packages depending on cat/foo have this
> > pkgconfig
> > side-stepping problem, we could theoretically write some eclass to
> > inject it when appropriate, and expect those side-stepping ebuilds
> > to
> > consume it, no? That seems pretty easy and hopefully wouldn't
> > require
> > any of the scary ideas that have been discussed in this thread.
>
> So you're saying we should write a whole eclass to do:
>
> append-cppflags -std=c++11
>
> ?
If we could just do "append-cppflags" somewhere, it'd be a non-bug as
we would have just done it. That doesn't seem to be the case.
If I'm not mixed up, the difficulty is determining /when/ that's
appropriate. Specifically, this c++11 requirement can bubble up
indirectly through deep dependencies or even non-DEPENDencies. Source
file "X" just needs to #include header "Y" that #include's header "Z"
with the c++11 requirement, and "kablooey!".
So, yeah, an eclass that figures out when to append the flag (and then,
as you say, appends it) doesn't seem so crazy to me.
For example: if we can identify the dependencies, deep, hidden, or not,
that cause these failures (i.e., by waiting for bugzilla reports and
acting on them), we could pretty trivially fix them with such an
eclass.
So, just imagining one possible implementation of that, here's a
possible user-story -- which I mean, an eclass user (and therefore and
ebuild or eclass developer, but developer-story isn't a word afaik):
Bug wranglers observe that www-client/frobzilla ebuilds are crashing
due to headers, which are coming, ultimately, from net-libs/tldr.
Assignees of the bug would:
o figure out which versions of net-libs/tldr trigger the problem
and ensure they are selected by the hand-crafted list of
DEPENDency specifiers that live in cxx11reqs.eclass.
o add the following code to all affected www-client/frobzilla ebuilds:
CXX11REQ_CULPRITS=( net-libs/tldr )
inherit cxx11reqs
This would cause the right thing to happen (CXXFLAGS augmentation) when
and only when the affected versions of tldr are installed.
Only thing left to do is to check that frobzilla doesn't strip the flag
back out, for some reason; if it did, perhaps it could add it back in,
by calling a xxx11reqs eclass API.
Note that frobzilla might not even DEPEND on tldr (not even indirectly
through several layers of DEPENDency); this would still do the trick;
the eclass could just use has_version to find out if the flag is
needed, according to a hand-crafted dependency specifier table.
I guess as far as bright ideas go, that's the best I've got, for now.
If a fully automagical solution is practical to implement, I haven't
yet imagined it.
-gmt
Re: [gentoo-dev] rfc: native multilib support in portage for eapi 7
On Tue, 2015-12-01 at 18:45 +0100, Michał Górny wrote: > On Tue, 1 Dec 2015 11:38:08 -0600 > William Hubbs wrote: > > > I find the multilib eclasses and their separate multilib phase > > functions > > to be confusing, so I was wondering if we could discuss making > > multilib > > support native to portage in eapi 7 so that we can use the normal > > phase > > functions again? > > That won't help you since the in-spec support would require special > phase functions anyway. It might help with other things, though. I think it might be hard to build sufficient consensus around this, but, if we could do so, it could really be a boon. As I recently pointed out in another thread, if we had a really easy- to-use, high-performing dedicated vdb feature behind "multi-ness," we might realize all kinds of benefits. For example -- of course the following are hoplessly speculative but just humor me for a moment -- in a sufficiently flexible implementation, concievably things like prefix and crossdev could be normalized as, respectively, user-configurable "EPREFIX" and "target" "multi-" dimensions. Not saying they should be, though, just making the point that the concievable applications of a feature tend to correspond to the feature's limitations and strengths from a practical perspective, and our current multi- eclass frameworks are fairly limited in precisely that way, despite being tremendously clever solutions to a very difficult problem. -gmt
Re: [gentoo-dev] impending c++11 clusterfuck?
On Tue, 2015-12-01 at 07:18 -0500, Anthony G. Basile wrote: > So on that bug we're talking about selectively > adding -std=c++11 (or possibly gnu++11) to packages. Yes, this is the biggest real-world problem we face. It requires an immediate solution in the ~* branches; the affected ebuilds just dump a bunch of gcc gobbldeygook and crash. If I understand the generalized problem we are facing there, as a package gets "c++11-ized", all of its reverse-BDEPs find themselves in the following situation: Imagine cat/foo is a library. cat/foo-3.0 is the newest non-c++11-ized version (in its slot, perhaps). Now cat/foo-3.1 comes along and from then on, anything BDEPENDing on it must be built with -std=c++11 (I'm ignoring the c++11 vs. gnu++11 but presumably we'll eventually need to figure out some kind of game plan about that). So, now, "everything" pulling in headers from cat/foo finds itself in this situation: o if it pulls in headers from =cat/foo-3.1, we must add a CFLAG I guess the reasonable way to achieve that sort of behavior is pkgconfig (which is not really a rock-solid solution. First, some packages might fail to put it in there (but, OK we just add it ourselves, let's say). Second, how many of cat/foo's reverse BDEPs side-step pkgconfig? Sometimes this is pretty common, I'm afraid. So, let's say lots of packages depending on cat/foo have this pkgconfig side-stepping problem, we could theoretically write some eclass to inject it when appropriate, and expect those side-stepping ebuilds to consume it, no? That seems pretty easy and hopefully wouldn't require any of the scary ideas that have been discussed in this thread. -gmt
Re: [gentoo-dev] impending c++11 clusterfuck?
On Sun, Nov 29, 2015 at 10:42 PM, Michał Górny wrote: > On Sun, 29 Nov 2015 19:56:04 -0800 > "Gregory M. Turner" wrote: > > the mess gets magically cleaned up by robots somehow. > > Sadly := can't help here since gcc switches occur independently of > package installs. And AFAIK revdep-rebuild doesn't help either. Indeed... magical hand-wavey part is the only teeny-tiny flaw with my proposal that the problem be magically, hand-wavically solved -- otherwise, it's rock-solid :) OTOH when you're spending ten hours a week waiting for revdep-rebuild, preserved-rebuild stops seeming like an intractable fantasy and more like an interesting puzzle you'd might as well take a crack at solving. I dunno about y'all but my not-really-that-computationally-powerful workstation has 2000 packages installed, about three-quarters of which seem to have forked webkit (just kidding, but it sure feels that way). emerge -e @world literally takes me two days if I'm lucky. If I'm not, more like a week, with my cores left mostly idling while my productivity is thwarted by various non-webkit-compiling distractions like using the toilet or working. Point being, it's damn slow; I'm starting to feel like the guy waiting for revdep-rebuild all week. There's gotta be a way to do it. If it requires some encapsulation-breaking hacks to get it done (as preserved-rebuild kinda did) it's probably well worth it. -gmt Greg Turner g...@be-evil.net
[gentoo-dev] impending c++11 clusterfuck?
I'm quoting myself from bug #566328 here. These were off-the-cuff remarks that got away from me and became a call-to-arms... (In reply to Michał Górny from comment #7) > This is never this simple. C++11 can change the ABI. So the point kinda is, > we need to ensure that all C++ libraries in a depgraph use the same C++ > version. This is pretty awful when you really think about it. I feel like I'm watching a train-wreck in super slow motion. I'm not sure we're taking this seriously enough -- sooner or later it seems destined to become a major clusterfuck if we don't do something proactive about it now while the drawing-board is relatively uncluttered. The only thing I can think of that has this kind of two-way depgraph magic property are the major "abi" USE_EXPAND values (multilib-build and python-r1, in other words). But those rely on fancy framework-generated USE-flag deps, which seem like overkill and likely to incur unjustifiable user-experience-costs. Perhaps a solution to this cxx11 clusterfuck can be found that works more like perl? By that I mean, pick your poison (respectively, your cxx11 ABI of preference or your major perl version of choice), rely on inbuilt portage features do the trick most of the time, and, when it breaks, run "magically-fix-everything.sh," grab a caffeinated beverage or three and fire up your favorite VOD client while the mess gets magically cleaned up by robots somehow. -gmt Greg Turner g...@be-evil.net
Re: [gentoo-dev] Re: [RFC] Cleaning up PMS to have ${D} not end with a slash
On 4/15/2013 10:31 AM, Steven J. Long wrote:
On Mon, Apr 15, 2013 at 12:01:24PM +0100, Ciaran McCreesh wrote:
On Mon, 15 Apr 2013 10:56:45 +
"Aaron W. Swenson" wrote:
ROOT being a user set variable, having ROOT be an empty string by
default still does not guarantee that ROOT won't end with a
slash. Even if we change it so that it defaults to an empty string, it
won't negate the need to do ${ROOT%/}/some/path.
The spec guarantees that ROOT will be non-empty and end in a slash. If
Portage isn't enforcing this, file a bug.
Yes, but his point was this:
The only thing that would help is if PMS defined that ROOT must not
end with a slash.
ie, let the mangler enforce empty or a valid directory not ending in a
slash, which is hardly difficult, and makes ROOT easier to work with,
along the lines of how any shellscripter would do it, similarly to the
motivation for the change to D. Only with ROOT it's much more important
that the path is correctly-formed for cross-platform compatibility.
That's how EPREFIX works; indeed, you don't need to do ${EPREFIX%/},
ever (unless, perhaps, we are manually scrubbing user inputs that
haven't been scrubbed by portage). Try it yourself: EPREFIX="/"
portageq envvar EPREFIX.
If we "wanted to" do the same thing to {E,}ROOT we could, at least in
future EAPIs.
However, given that ROOT is named "ROOT", "/" makes some kind of
reasonable, intuitive sense. I see double-slash mistakes involving ROOT
here and there, but they are not hopelessly ubiquitous as they are with
D, where, as Michał demonstrates in #465772, approximately 10 ebuilds in
portage use the variable incorrectly for every 1 getting it right!
-gmt
[gentoo-dev] OT: was Re: bash-3.1 stable
On 4/2/2013 10:54 AM, Duncan wrote: Tho in practice, if you're here long you do develop a rather thick hide. Nothing NEAR what it was a few years ago, tho. Before it gets anywhere near that, there's warnings, these days. Kinda just what happens, when you put a bunch of unpaid, sleep-deprived, anal-retentive engineers in a virtual project room and set them all about trying to create stuff by tearing apart each others' work products and rebuilding them to be "better". Open source seems to arrive at consensus by means of a process something like: whiny trolling -> unfettered troll-feeding -> hyperbolic butt-hurt -> (optional) finger-pointing and self-pity -> reluctant compromise -> write code -> (repeat) Think of the above as the Open-Source equivalent of the Rational Unified Process :) -gmt
Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash
On 4/13/2013 2:23 PM, Michał Górny wrote: Your thoughts? +1, this causes untold agonies for my pet platform. See #465772 for more thorough griping :) -gmt
Re: [gentoo-dev] Re: rfc: toolchain-funcs: target-has-split-usr API?
On 4/13/2013 9:56 PM, Ryan Hill wrote: On Fri, 12 Apr 2013 13:04:57 -0700 "Gregory M. Turner" wrote: What do people think of something like this? Obviously the equivalent patch to prefix would need to include a test for PREFIX_DISABLE_GEN_USR_LDSCRIPT: Author: Gregory M. Turner Date: Fri Apr 12 11:13:21 2013 -0700 eclass/toolchain-funcs: Add target-has-split-usr API Move the platform-filtering code from gen_usr_ldscript into its own API so that ebuilds can reliably test whether gen_usr_ldscript does something or not in the currently applicable environment. See in-source comments for more details. Signed-off-by: Gregory M. Turner File a bug with the maintainers of that eclass. That's my plan, but I first wanted to give the wider audience on this list the opportunity to voice any opinions on the matter. -gmt
[gentoo-dev] rfc: toolchain-funcs: target-has-split-usr API?
What do people think of something like this? Obviously the equivalent
patch to prefix would need to include a test for
PREFIX_DISABLE_GEN_USR_LDSCRIPT:
Author: Gregory M. Turner
Date: Fri Apr 12 11:13:21 2013 -0700
eclass/toolchain-funcs: Add target-has-split-usr API
Move the platform-filtering code from gen_usr_ldscript into its own
API so that ebuilds can reliably test whether gen_usr_ldscript does
something or not in the currently applicable environment. See
in-source comments for more details.
Signed-off-by: Gregory M. Turner
diff --git a/eclass/toolchain-funcs.eclass b/eclass/toolchain-funcs.eclass
index 9b94512..7600aaf 100644
--- a/eclass/toolchain-funcs.eclass
+++ b/eclass/toolchain-funcs.eclass
@@ -604,6 +604,36 @@ gcc-specs-nostrict() {
return $([[ "${directive/\{!fstrict-overflow:}" != "${directive}" ]])
}
+# @FUNCTION: target-has-split-usr
+# @DESCRIPTION:
+# This function returns 0 (true) if, for the currently
+# targeted platform, portage is presuming that a "split-usr"
+# configuration must be maintained. When target-has-split-usr
+# returns a nonzero (false) value, gen_usr_ldscript is a noop. This
+# may be handy in cases where an ebuild needs to decide, i.e., whether
+# it makes sense to move certain files from /usr/bin to /bin
+# in an ebuild. In the future, the assumption that "critical"
+# stuff ends up in "/bin" and "/$(libdir)" whereas "noncritical"
+# stuff ends up in "/usr/bin" and "/usr/$(libdir)" may be removed
+# from portage entirely, or relegated to a legacy-support role.
+# As of April 2013, discussion was ongoing in the Gentoo developer
+# community as to what exactly to change and how (see bug #417451).
+# By predicating ebuild code on target-has-split-usr, which exists solely
+# to maintain a split-usr layout, ebuilds can future-proof themselves
+# against changes to the precise criteria that determine whether or not
+# split-usr is in effect for a given target and configuration.
+target-has-split-usr() {
+tc-is-static-only && return 1
+
+case ${CTARGET:-${CHOST}} in
+*-darwin*) return 0;;
+*linux*|*-freebsd*|*-openbsd*|*-netbsd*)
+use prefix && return 1
+return 0
+;;
+*) return 1 ;;
+esac
+}
# @FUNCTION: gen_usr_ldscript
# @USAGE: [-a]
@@ -620,19 +650,10 @@ gcc-specs-nostrict() {
# the library (libfoo.so), as ldconfig should usually update it
# correctly to point to the latest version of the library present.
gen_usr_ldscript() {
+target-has-split-usr || return 0
local lib libdir=$(get_libdir) output_format="" auto=false
suffix=$(get_libname)
[[ -z ${ED+set} ]] && local ED=${D%/}${EPREFIX}/
-tc-is-static-only && return
-
-# Eventually we'd like to get rid of this func completely #417451
-case ${CTARGET:-${CHOST}} in
-*-darwin*) ;;
-*linux*|*-freebsd*|*-openbsd*|*-netbsd*)
-use prefix && return 0 ;;
-*) return 0 ;;
-esac
-
# Just make sure it exists
dodir /usr/${libdir}
--
gmt
Re: [gentoo-dev] eclass error-handling post-EAPI4
On 10/20/2012 4:07 PM, Zac Medico wrote: On 10/20/2012 03:51 PM, Gregory M. Turner wrote: Anyhow, given that eclasses can probably avoid auto-die() if they need to, this leaves the issue of nonfatal allowing die to return to callers... is there really any reason that die() needs to work this way? If helpers always used __helpers_die() instead of die(), then couldn't die() be changed to ignore PORTAGE_NONFATAL? Yes. It's a bug, as Ciaran said. For anyone following this discussion, just a note that I've created a bug for this issue: https://bugs.gentoo.org/show_bug.cgi?id=439356 A proposed patch is attached to the bug. -gmt
Re: [gentoo-dev] eclass error-handling post-EAPI4
On 10/20/2012 11:16 AM, Zac Medico wrote: On 10/20/2012 03:52 AM, Gregory M. Turner wrote: EAPI[0-3] conventions encourage the habit of writing ebuild code like invoke_fn || ... # handle error, probably by die()ing but maybe not But eclasses that auto-die violate this expectation and create a situation where ebuild authors must be mindful of whether a particular function is a helper-function or an eclass utility function to correctly anticipate whether auto-death will occur. should we really write new eclass code to behave this way as well? Yeah, I think we should. If the eclass function is doing something that might result in an undesirable die, then it can try to avoid the die by first doing a sanity check. For example, it can check that a given file path exists before passing it to the helper, and skip calling the helper if the file path doesn't exist. That seems reasonable -- in practice, I don't find myself needing to invoke helpers very much at all in eclasses, nor does it seem too terribly frequent in the existing eclasses (dodir is pretty common, but unlikely to fail unless something is seriously wrong anyhow). Would it be feasible to lift the >=EAPI4 restriction on nonfatal()? I guess the problem would be that old portage's wouldn't have it. Anyhow, given that eclasses can probably avoid auto-die() if they need to, this leaves the issue of nonfatal allowing die to return to callers... is there really any reason that die() needs to work this way? If helpers always used __helpers_die() instead of die(), then couldn't die() be changed to ignore PORTAGE_NONFATAL? It almost seems like the portage code was designed for nonfatal only to apply to helper invocations, and then the change to die() was added as an afterthought, to support the broader range of usages suggested (rather vaguely, I'd say) by the language in the PMS, ... yet somehow, the PMS ends up strongly suggesting -- I think most people would read the language as an explicit specification -- that ebuilds can rely on die() to terminate regardless of any usage of nonfatal. -gmt -gmt
Re: [gentoo-dev] eclass error-handling post-EAPI4
On 10/20/2012 4:05 AM, Ciaran McCreesh wrote:
On Sat, 20 Oct 2012 03:52:49 -0700
"Gregory M. Turner" wrote:
Took me a while, but I think I see why this is correct, now (mostly
-- see below). The source of my confusion was a mistaken assumption
that die() would not respect PORTAGE_NONFATAL.
The source of your confusion is more the impression that there is such
a thing as PORTAGE_NONFATAL. You should be reading the spec, not code.
Seriously? OK, let's try:
http://dev.gentoo.org/~ulm/pms/4/pms.html#x1-1312.3.3.1
and
http://dev.gentoo.org/~ulm/pms/4/pms.html#x1-13500012.3.3.6
do not comport with the actual behavior of portage.
Specifically, they very strongly suggest that nonfatal does not
influence the behavior of die(). So depending on one's reading of the
situation, either the PMS is misspecified or portage is broken. Meanwhile,
http://dev.gentoo.org/~zmedico/portage/doc/ch05s03s05.html#package-ebuild-eapi-4-helpers-die-nonfatal
is dusting so much under the carpet that I can't possibly do my job as
an eclass author without reverse-engineering, dry-run code analysis, or
asking the SME's here on the mailing list how this is supposed to work
in practice.
I've already asked the mailing list and so far the closest thing I have
to an answer is Zac's response. Although I appreciate his help, I've
explained why his response doesn't fully address my concerns. If you
don't trust my dry-run analysis then we can try the reverse-engineering
approach. Give this ebuild a try:
>8
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
EAPI=4
DESCRIPTION="Foo"
SLOT="0"
KEYWORDS="~x86 ~amd64 ~x86-linux ~amd64-linux"
IUSE=""
DEPEND=""
RDEPEND="${DEPEND}"
utility_fn() { die "uhoh"; }
src_unpack() { mkdir -p "${S}"; }
src_prepare() { nonfatal utility_fn; }
src_compile() { echo "yaaay" > foo.txt; }
src_install() { dodoc foo.txt; }
8<
As you can see, my dry-run analysis appears to have been correct.
There is no documentation concerning best practices for eclass authors
post-EAPI4, so I can't just RTFM -- there is no FM. Furthermore, it
appears that EAPI4 has broken a great many eclasses when invoked with
the new nonfatal operator. I guess I can just file a bug for that since
so far nobody seems too concerned by it, but that still doesn't help me
to write my eclass.
-gmt
Re: [gentoo-dev] eclass error-handling post-EAPI4
Thanks for your reply, but I still have some concerns about this.
On 10/19/2012 8:11 AM, Zac Medico wrote:
Regardless of EAPI, don't call the helpers that die in EAPI 4 unless you
want the function to die when the helpers fail, and use helper || die so
it behaves the same regardless of EAPI.
Took me a while, but I think I see why this is correct, now (mostly --
see below). The source of my confusion was a mistaken assumption that
die() would not respect PORTAGE_NONFATAL. But isolated-functions.sh has
roughly:
die()
{
[[ ${PORTAGE_NONFATAL} == 1 ]] && {
echo "${FUNCNAME[1]}: WARNING: $*" >&2
return 1
}
.
.
So -- correct me if I'm wrong -- this all works as follows:
o helpers always handle errors by remembering the exit code of some
error-prone operation, and, if that exit code is nonzero, invoking
__helpers_die with some description of the problem, and finally
returning the remembered exit code.
In EAPI[0-3], this will dump the error text to stderr and return
the remembered exit code.
In >=EAPI4, this will call isolated-functions.sh's die(), which will
swoop in and terminate everything before it has the chance to
go any further, effectively ensuring that __helpers_die never
returns -- /unless/ PORTAGE_NONFATAL is 1, in which case, it will
prepend "__helpers_die:" to the error message, but otherwise
behave as in EAPI[0-3].
o die incantations in an ebuild or eclass will either directly or
indirectly call die() in isolated-functions.sh, which still works
as described above when used in these contexts -- which is to say,
that die is nonfatal when PORTAGE_NONFATAL is set.
o PORTAGE_NONFATAL is only supposed to be manipulated by means of
the nonfatal API, which is only callable in >=EAPI4, which means
that unless somebody "cheated" by directly messing with the
variable, in EAPI[0-3], die() never returns (like "exit")
It's okay to die in older EAPIs, as long as you're not changing the
> behavior of a previously existing eclass function.
ACK
Previously existing eclass functions that only start to
die in EAPI 4 are okay, since they only start to die when the ebuild
developer bumps the EAPI.
ACK
If I indeed understand the facts correctly, I'm still a bit
uncomfortable with your advice to just use "helper || die" in eclass
code. It seems to me that if I follow this recipe, >=EAPI4 kinda works
OK, but EAPI[0-3] doesn't.
Specifically, an EAPI[0-3] ebuild author will not have any means at his
or her disposal to handle errors that occur in eclass utility functions.
The error-handling semantics of eclass utility functions would be like
EAPI4, except without nonfatal around to provide relief from auto-die.
EAPI[0-3] conventions encourage the habit of writing ebuild code like
normal bash script code, which is to say, that authors expect to be able
to write code like
invoke_fn || ... # handle error, probably by die()ing but maybe not
But eclasses that auto-die violate this expectation and create a
situation where ebuild authors must be mindful of whether a particular
function is a helper-function or an eclass utility function to correctly
anticipate whether auto-death will occur.
This seems to be by design and may not be so horrible as I make it sound
-- for example, it would be pretty weird to expect eclass code written
before EAPI4 to suddenly start working differently in EAPI[0-3].
But should we really write new eclass code to behave this way as well?
Worse, eclasses authored in the pre-EAPI4 era ubiquitously assume that
die() never returns, and do things like:
efoo() {
[[ ${EBUILD_PHASE} != "prepare" ]] &&
die "efoo can only be used during src_prepare"
.
.
# stuff that isn't safe if ${EBUILD_PHASE} != "prepare"
.
.
}
Since, in EAPI4, no means is provided to specify that a particular
invocation of die() is unconditionally terminal, (except directly
manipulating PORTAGE_NONFATAL, which doesn't seem to be encouraged), the
only non-encapsulation-breaking non-EAPI-specific solution would be to
re-code every presumed-terminal use of die() to look something like:
failure_prone_action
ret=$?
if [[ ${ret} != 0 ]] ; then
die "Error message"
return ${ret}
fi
I can imagine no reasonable way to avoid this without using aliases...
or perhaps some construct like:
do_or_die() {
diemsg="$1"
shift
"$@" || { ret=$?; die "${diemsg}"; return ${ret}; }
}
efoo() {
do_or_die "error message 1" \
failure_prone_action1 arg1 arg2 arg3 && \
safe_action1 arg1 && \
safe_action2 arg1 arg2 && \
do_or_die "error message 2" \
failure_prone_action2 arg1 arg2 && \
.
.
Hopefully I'm missing something
[gentoo-dev] eclass error-handling post-EAPI4
I'm cooking up some eclass functions and it occurred to me that perhaps
I had some responsibility regarding EAPI4's new error handling
semantics. After looking into it, it seems that, superficially, the
answer to my question is "no, the EAPI4 changes only apply to ebuild
helpers."
Even so, this got me thinking: if ebuild helpers are going to warn(*) in
EAPI<4 and die in EAPI>=4 (unless somebody used "nonfatal"), then, those
error-handling semantics are going to bubble up through my eclass to its
heirs (assuming my eclass uses helpers).
The problem I see with that is that a consumer of my eclass function is
effectively required to anticipate these EAPI helper-function outcomes,
which is fair-enough if s/he's consuming them directly, but pretty
unreasonable if they're just consuming my eclass In effect, this
seems to create a situation where they can't know what to expect,
without wading through my code, unless I go in and document "in EAPI4+
it does X, but in EAPI3, it does Y". Which, let's face it, most eclass
authors probably won't get around to doing.
For example, suppose poop.eclass has a function "epoop" which performs
the following steps:
o Downloads the latest "post" from the
http://www.unnecessaryquotes.com/ "website" using "wget"
o Strips out the top 3 posts from piped wget output using
a bunch of sed and awk magic. Saves each post-processed post as
"${S}/$(mktemp -u $( printf %100s |tr " " "X" )).xml" and
remembers these filenames in three hard-coded variables.
o Installs the posts as documentation using dodoc statements
o Returns "$(( $(date +'%s') % 3 ))"
Let's say I started writing this code before I knew anything about EAPI
4, and the current epoop documentation states that:
o errors in the wget and postprocessing steps will result in a return
code of '3'
o errors from dodoc will result in a return code of '3' + the
return-code of dodoc
o epoop() succeeds every third second and otherwise fails, returning
the number of seconds ago that the API could have been invoked
successfully as the error code (WTF: this is tounge-in-cheek:
$(( $(date +'%s') % 3 )) is the current UNIX time in
seconds, modulo 3, an extremely pointless and arbitrary success
criterion for my hypothetical function that will cause it to
randomly fail 66% of the times it's invoked and runs to completion.
If you don't find this amusing, that's OK; it's not terribly
important to the matter at hand.)
Well, now I have an awkward situation. In EAPI's 0-3, the function
works as documented. In EAPI 4+, however, epoop() will die if any error
occurs in dodoc. I can fix this by replacing the dodoc code -- let's
say it originally was coded like:
dodoc ${docfile1} || { r=$?; eerror dodoc1; return $((r+3)); }
dodoc ${docfile2} || { r=$?; eerror dodoc2; return $((r+3)); }
dodoc ${docfile3} || { r=$?; eerror dodoc3; return $((r+3)); }
-- with code like:
.
.
if has "${EAPI:-0}" 0 1 2 3 ; then
nonfatal dodoc ${docfile2}
else
dodoc ${docfile2}
fi || { r=$?; eerror dodoc2; return $((r+3)); }
.
.
but I will have to do something like this for each of the three dodoc
statements. Seems like a real PITA.
Obviously, I've chosen a very silly example, and perhaps I've
cherry-picked a solution that illustrates my point -- but it's not hard
to imagine having to make fairly significant changes to an eclass in
order to preserve its pre-EAPI4 semantics in EAPI4+.
The conclusion I'm tempted to draw is that it makes more sense to bring
my eclass error-handling behavior into line with the EAPI-specific
behavior of helper-functions than it does to gum up helper usage in my
eclass with a bunch of EAPI-specific error-handling spaghetti-code.
i.e., returning to epoop: instead of the above solution, I might leave
the dodoc code alone, and instead implement something vaguely like
__helpers_die, i.e.:
# note: stupidly clever -- not how i'd implement this irl
_poop_die_maybe() {
local ret="$?" eapi="${EAPI:-0}"
eapi=${eapi::1}
(( (3-eapi)*(PORTAGE_NONFATAL-1) > 0 )) && die "$@"
eerror "$@"
return ${ret} # pass through pre-invocation $? unchanged
}
With this in place, it's not terribly difficult to bring the
error-handling/EAPI semantics of epoop() into line with those of the
helper functions across all EAPIs. For example, where we had
return $(( $(date +'%s') % 3 ))
we might now put
ret="$(( $(date +'%s') % 3 ))"
(( ret == 0 )) || \
_poop_die_maybe "epoop: invoked ${ret} seconds too late"
return "${ret}"
Or, if it was the last statement in the epoop function, just:
( exit $(( $(date +'%s') % 3 )) ) || \
_poop_die_maybe "epoop: invoked $? seconds too late"
I hope my gist isn't getting lost in the particulars of this silly
example. My point is: EAPI4 changes the behavior of eclasses if those
eclasses consume helper f
Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
On 10/14/2012 9:29 PM, Mike Frysinger wrote: On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote: "Thirdly" has been addressed ad nauseam in this thread and will be solved by prepending the LDFLAG rather than appending, or, preferably, by patching autotools (but only if I can find a simple, low-maintenance approach that is likely to work without building any new per-platform matrices or case-statements). i'm fairly certain this isn't autotools. i've poked around the python build system before in the past and while it uses autoconf to do platform tests, it doesn't use automake/libtool. make is used to bootstrap python, and then they descend into a horrible hack of a custom build system written in python. i suspect much of the pain you're seeing is coming from that last part. -mike I guess I should say "in src_prepare" rather than "in autotools." Specifically, I was thinking some kind of configure.in patch might be good, since configure.in seems to churn less than Makefile.in, a good thing if I want to produce a one-size-fits-all patch. And, yeah, setup.py is definitely behind all this yucky. Python clearly has an amazing community, so I hate to say anything negative... but I sometimes wish they would "build" less and "buy" more. Anyhow, as everyone knows, bitching and moaning about FOSS is pointless so I'll stop there and spend my time fixing stuff instead :) -gmt
Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
On 10/12/2012 4:03 AM, Gregory M. Turner wrote:
First, something puts in all kinds of inappropriate amd64 multilib paths
(this ends up being harmless as wrong-arch libraries get rejected at
link-time and treated as non-matches for -lclauses... still, WTF?).
Secondly, something puts the built-in ld search paths into the
command-line ld search path (a practice which has been roundly
disparaged in this thread, and correctly so, IMO).
Thirdly, prefix x86-linux also clearly suffers from the original problem
I was seeing in cygwin, and which inspired this thread (putting -L.
after $(libdir) paths and therefore linking against the wrong
libpython.so). I never suspected this applied to every prefix python
ebuild, but that's what the above seems to suggest... not cool.
Just wanted to update this thread with latest current triage and
prognosis data for this patient:
Regarding etiology, for "Firstly" and "Secondly" above: it seems that
these are by design (which is not to say that they should be left: in
each case, the "design" is pretty bad and ought to be fixed, I think).
My hope is that "Firstly" and "Secondly" are needed only to accommodate
prefix bootstrap, in which case we can fix them by detecting whether a
prefix bootstrap is ongoing or not, and conditioning these hacks on that
(improving "Firstly" to not be braindead would be a nice touch, although
doing so in a way that doesn't break anything may be prohibitively
difficult or beyond my abilities).
"Thirdly" has been addressed ad nauseam in this thread and will be
solved by prepending the LDFLAG rather than appending, or, preferably,
by patching autotools (but only if I can find a simple, low-maintenance
approach that is likely to work without building any new per-platform
matrices or case-statements).
Note that, even without this third fix, absent a PEBKAC placement of
"-L${EPREFIX}/usr/$(libdir)" into LDFLAGS, either:
o it's a prefix bootstrap, in which case there probably is no
${EPREFIX}/usr/$(libdir)/libpython${PV}.so, or
o it's /not/ a prefix bootstrap, in which case the above-mentioned
fixes would prevent the "-L${EPREFIX}/usr/$(libdir)" from
getting into LDFLAGS, thus preventing the problem from arising.
So by fixing "Firstly" and "Secondly," we have at least brought things
sufficiently in line with expectations that we are only fixing
user-level configuration mistakes instead of our own, as originally
expected. **
I'm feeling ready, at this point, to file bugs for these issues; once I
do, I'll post bug#'s into this thread and plan to take all off this
chatter off-list, unless something more comes up that seems to require
community review.
Thanks for everyone's help figuring this stuff out,
-gmt
** I suppose a third possibility is that someone is doing an in-place
"re-bootstrap" or restarting a bootstrap build that failed mid-merge,
but ... doesn't matter, it'll be academic once we fix the bug.
Re: [gentoo-dev] Proposal: removing "server" profile variants from profiles.desc
On 10/11/2012 3:31 PM, Ben Kohler wrote: There are other ways to achieve a "lighter" system, but that's not really what this is about. The server profiles are not any lighter than the base profiles. To those in favor of keeping some kind of "server" profile around, how would it differ from the base profile? What would you enable or disable on top of the base? I am pretty sure that the current USE="-perl -python snmp truetype xml" is not what any of you would suggest. In my opinion, removing /usr/portage/profiles/targets/server/make.defaults and having the "server" target apply nothing over the base profiles, and then dropping the warning from the server profiles, would be a better situation than where we are now. -Ben That seems pretty reasonable. Another possibility, I suppose, would be to combine Rich's idea with yours and get: base -> minimal -> server (noop) Also super-easy if we just build the above cascade, take what's in server, move it into minimal, git rid of the awful warning, and call it fixed. At least by clarifying what is meant by "server", we might inspire someone to contribute more enhancements later. Kind of the opposite of your original proposal, Ben :P At least we've established broad consensus that the warning should be silenced. -gmt
Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
On 10/11/2012 2:40 PM, Marien Zwart wrote:
I'm going to do something potentially rude and comment on this without
having read the entire thread.
On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner wrote:
Anyhow one thing I have figured out is how things can work correctly on
Linux wihtout -L.: on Linux, the python plugins aren't actually linked
against libpython.so!
Python can be built with and without a shared library
(libpythonx.y.so). If the shared library is built the interpreter
executable is linked against it (of course), and normally so are the
extension modules. This is useful because it makes the linker do the
right thing for applications that embed python (and end up loading
libpythonx.y via dlopen). If the shared library is not built extension
modules cannot link against it, and the python executable itself
exports the relevant symbols. So on a normal shared python build
extensions should end up linked against libpythonx.y.so.
However, at least one popular distro and presumably most of its
derivatives (Debian) has a system python built without the shared
library, plus the shared library built separately, for use by
applications that embed python. I am not entirely sure why they still
do this (if I understand correctly it was originally done for
performance reasons, but I'm not sure if they've checked those are
still valid). Their extension modules cannot link to libpythonx.y:
their python interpreter executable provides the same symbols
libpythonx.y provides, so loading such an extension module into the
regular interpreter would result in collisions.
The reason I mention this is Gentoo's python is built with the shared
library, so on Gentoo your extensions *should* end up linked to
libpythonx.y (if they use a more or less standard build system, at
least). If yours are not I'm curious what happened, and would
appreciate it if you gave me some more information off-list (marienz
on freenode or email).
Thanks a ton for this info, that explains a lot. Actually, got this
notion testing outside of Gentoo, building from a vanilla python 3.3.0
tarball on a Fedora box. I wanted to see what happens in a completely
"Gentoo-free" environment so as to get some kind of baseline
understanding of how upstream is propagating stuff through the Makefiles
to setup.py.
Probably, I screwed up ./configure (I tried to approximate Gentoo's
arguments), and requested a non-shlib build. Guess it would have been
clearer if I'd mentioned that in my post -- sorry if I worried you :)
I do, of course, intend to try it on my main Gentoo ~amd64 -- if I ever
see it fail to do as you describe, I'll let you know, Marien.
My i686 prefix is up to date and I can confirm that it is linking
against libpython there.
But... wow. In my almost-vanilla i686 cross-prefix (hosted by my
only-moderately-rice multilib ~amd64 workstation), in
=dev-lang/python-2.7.3-r2, the modules are linked like so:
i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-fno-strict-aliasing \
-O2 -march=i686 -pipe -m32 \
-fwrapv -DNDEBUG \
-I. -IInclude -I./Include \
-I${EPREFIX}/usr/include \
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/mathmodule.o
\
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/_math.o
\
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-lm -lpython2.7 \
-o build/lib.linux-i686-2.7/math.so
Which is just shameful. In this case, I'm pretty sure all those
-L{${EPREFIX},}{/usr,}/$(libdir) clauses are not coming from me:
[PREFIX] $ emerge --info --verbose | grep /lib
sys-devel/libtool:2.4.2::gentoo_prefix
COLLISION_IGNORE="/lib/modules/* *.py[co]"
PKG_CONFIG_PATH="${EPREFIX}/usr/lib/pkgconfig:$( :;
)${EPREFIX}/usr/share/pkgconfig"
PORTAGE_BIN_PATH="${EPREFIX}/usr/lib/portage/bin"
PORTAGE_PYM_PATH="${EPREFIX}/usr/lib/portage/pym"
UNINSTALL_IGNORE="/lib/modules/*"
(snippets edited for readability obv).
Regarding the gcc-as-linker invocation above:
First, something puts in all kinds of inappropriate amd64 multilib paths
(this ends up being harmless as wrong-arch libraries get rejected at
link-time and treated as non-matches for -lclauses... still, WTF?).
Secondly, something puts the built-in ld search paths into the
command-line ld search path (a practice which has been roundly
disparaged in this thread, and correctly so, IMO).
Thirdly, prefix x86-linux also
Re: [gentoo-dev] Proposal: removing "server" profile variants from profiles.desc
On 10/11/2012 1:04 PM, Walter Dnes wrote: On Thu, Oct 11, 2012 at 03:22:17PM -0400, Mike Frysinger wrote sounds like something to fix rather than punt. i don't know why you think having server profiles is "undesirable", but i certainly desire it on many systems. like servers. the desktop and developer profiles are not appropriate. If you want a light profile, I suggest doing what I do... start your USE variable in make.conf with "-*", and add any flags you need, either in package.use or in make.conf. -gmt
Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
On 10/11/2012 8:50 AM, Mike Frysinger wrote:
On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:
On 10/10/2012 9:14 PM, Mike Frysinger wrote:
it's not particularly important, but on one hand, the LDFLAGS parsing logic
should not be in the tree ever.
I've no major attachment to it. Took all of five minutes to code up
reading the ld manpage, and as you pointed out, it's probably solving a
non-problem.
on the other, i can stomach much smaller one-
off hacks like prepending -L. to LDFLAGS if the maintainers of the python
ebuild really really want to add it.
If fixing the python builds proves too onerous then this is what I'll
end up filing a bug for.
my [limited] understanding of the prefix compiler is that they wrap things with
a custom shell script to inject -L flags behind the back of the compiler at the
very last possible minute and point to the right place.
Pretty fuzzy on this myself. Whatever binutils-config does works so
well I've never had to look into it very deeply :)
Above, however, I'm referring to the prefix-specific cpython patch-sets,
where I was "sure" I'd seen patches to project LDFLAGS into setup.py's
compiler invocation tables. But it seems I was mistaken. Perhaps it
was in the main gentoo-patches after all... or maybe I just need more
coffee...
Anyhow one thing I have figured out is how things can work correctly on
Linux wihtout -L.: on Linux, the python plugins aren't actually linked
against libpython.so!
With any luck, this explains what's going on and suggests that platforms
linking python modules against libpython.so just need to wedge -L${S}
somewhere in configure.in, possibly do-able via sed script (unless we
end up having to mess with setup.py, in which case, things may get a bit
more complicated).
-gmt
Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
On 10/10/2012 9:14 PM, Mike Frysinger wrote:
On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
(1) is worse than (2), but it does have some quasi-legitimate usages.
For example, prefix bootstrap does this (or used to), as do many of the
crossdev-wrapper scripts. I've also resorted to such usage, myself,
when repairing a prefix after I've broken its compiler. These cases
don't really seem completely "correct" or "incorrect" -- about the best
I can say for them it is that they are mostly efficacious, but prone to
problems.
and no, crossdev doesn't do this. it is properly sysrooted.
You are right. I guess I was thinking of this:
http://tinderbox.x86.dev.gentoo.org/misc/cmerge,
Which does use -L but looks pretty ancient. I see now that crossdev's
equivalent hacks up the linker scripts -- which seems as "correct" as
one could reasonably hope for... pretty sexy.
further, you cannot do multilib with users putting -L into
their LDFLAGS. they just forced a single ABI and any other will simply fail.
the toolchain's compiler driver knows exactly where to find its own libraries.
if it doesn't, it's broken, and it should be fixed.
ACK
So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
complete abomination. Still... I don't like it.
it's significantly better than the proposed code which tries to parse the
meaning of various flags and insert it at the "right" point. i can't see it
being useful at all. it's a recipe for fragility and being unmaintainable.
I think I meant the above differently than you interpreted it: I meant
that solving library-search-path selection in autotools steps and/or
Makefile.in would make the question of how to manipulate LDFLAGS
academic, and seems more elegant. Apparently you agree:
If the Makefiles are building against libraries expected to be in
${PWD}, it seems to me that the Makefiles should know to look there
automatically.
yes, so why is the ebuild specifying -L. for it ? the package should already
have a -L to the right place, and if it wants to make sure it gets used before
the user's LDFLAGS, it should show up in the linking before it (or have the
build system prepend the value to LDFLAGS if it's appending currently).
If we decide that manipulating LDFLAGS is correct, whether we do this
using my "parser" or LDFLAGS="-L. ${LDFLAGS}" isn't particularly important.
My point above was that coming to that decision in the first place
suggests some underlying issue exists that arguably ought to be fixed
instead.
i'm not sure why this applies to the larger build system. i can understand
that the python build itself might not be putting the -L. in a place where a
user's misconfigured LDFLAGS will cause a problem.
This only pertains to dev-lang/python AFAIK... there may be others
lurking out there somewhere, but I'm not too worried about them. The
use-cases I have in mind are prefix bootstrap and system-recovery when
gcc breaks.
In general, if python breaks, folks can end up in pretty bad shape due
to chicken-and-egg issues with portage. That's the only reason I've
spent so much effort trying to think this fairly obscure corner-case
through.
-gmt
P.S.: After poking around, the first commit where this appeared is this
one:
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-lang/python/python-2.5.1.ebuild?hideattic=0&r1=1.4&r2=1.5&view=patch,
a mere 5.5 years ago by uberlord.
The cvs ChangeLog references bug 177426 which doesn't seem particularly
illuminating. It also says:
export LDFLAGS=-L. so we link modules correctly on FreeBSD
and possibly other systems where python2.5 isn't installed yet.
Which at least seems to confirm that this was originally a problem not
of search path ordering, but of complete failure to search ${S} in the
first place.
Meanwhile, I've looked into the patches a bit. In regular Gentoo
nothing special happens to LDFLAGS, but in prefix, there is special
handling that suggests some avenues of attack. Before I do anything
further, I want to figure out if it works right without LDPATH
manipulation on linux, and, if so, whether I can't generalize whatever
mechanism allows that to happen.
Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
On 10/9/2012 2:26 PM, Mike Frysinger wrote:
On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:
My god, I am a horrible self-editor. Sorry. Please ignore the magnum
opus above and allow me to try again.
In dev-lang/python*, we use
append-ldflags '-L.'
to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir). But, this doesn't work if
LDFLAGS contains "-L$(libdir)".
well, some notes here ...
for users: putting -L is wrong. it changes the meaning of
"system paths" when it comes to searching for linking by elevating it to "user
specified path". imo, you should be fixing the source rather than the symptom.
for packagers: using -L$(libdir) is almost always wrong and pointless. the
toolchain itself knows the proper system path to search for libraries to link
against, and when you cross-compile, the --libdir you specify to configure is
not going to be valid when dereferenced on the build system.
Agreed on these points. Not until we combine the doubly broken
circumstances of (1) User specified -L/usr/$(libdir) in LDPATH and (2)
an ebuild that puts -L. into LDPATH to fix Makefile recipes that appear
more-or-less broken, do we get this conflict. This leads to a certain
odor of "unsupportable circumstances" which, I must admit, makes me
slightly uncomfortable trying to fix it.
Breaking this down:
(1) is worse than (2), but it does have some quasi-legitimate usages.
For example, prefix bootstrap does this (or used to), as do many of the
crossdev-wrapper scripts. I've also resorted to such usage, myself,
when repairing a prefix after I've broken its compiler. These cases
don't really seem completely "correct" or "incorrect" -- about the best
I can say for them it is that they are mostly efficacious, but prone to
problems.
As for (2)... meh. LDFLAGS, to my thinking, means something like: "Use
these guidelines and parameters for linking, in preference to the
default guidelines that the build process would normally use." LDFLAGS
coming from make.conf have a slightly different meaning, I guess, but
the gist of it is the same.
So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
complete abomination. Still... I don't like it.
If the Makefiles are building against libraries expected to be in
${PWD}, it seems to me that the Makefiles should know to look there
automatically.
In the particular case of Gentoo's cpython, I have to admit I'm
concerned that if I start mucking around with the Makefiles/autotool
scaffolding, I'm going to break stuff. But maybe it's worth taking that
risk if it allows us to do things more correctly going forward... I
could at least take a crack at it and see what people think of the results.
-gmt
Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
On 10/6/2012 1:31 AM, Fabian Groffen wrote:
On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:
In dev-lang/python*, we use
append-ldflags '-L.'
to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir). But, this doesn't work if
LDFLAGS contains "-L$(libdir)".
We could try to fix this like:
export LDFLAGS="-L. ${LDFLAGS}"
or so. That would cover 99.9% of the cases out there. But very rarely,
indiscriminately placing our '-L.' before every other clause in LDFLAGS
might cause an unanticipated side-effect.
I think it would make more sense in this case to just add one more patch
to Python, such that the build-system just inserts it. I recall it's
necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
more why it works/worked on Linux fine.
I was thinking along these lines as well. As it turned out, however, I
was not able to find an automagical, PMS-compliant, non-crazy way to do
it that resolved my test-case (more on that below), without effectively
hooking econf().
Literally hooking a core API seems like a Pandora's box better left
unopened, so this is what I came up with (note: the fpectl stuff was
also in my overlay and the patches got entangled -- rather than editing
the hunk by hand, I just left it. It's orthogonal to this issue,
however, and for present purposes can be ignored):
-->8->
--- PORTAGE/dev-lang/python/python-2.7.3-r2.ebuild
+++ OVERLAY/dev-lang/python/python-2.7.3-r2.ebuild
@@ -218,13 +255,6 @@
# http://bugs.python.org/issue15506
export ac_cv_path_PKG_CONFIG=$(tc-getPKG_CONFIG)
- # Set LDFLAGS so we link modules with -lpython2.7 correctly.
- # Needed on FreeBSD unless Python 2.7 is already installed.
- # Please query BSD team before removing this!
- # On AIX this is not needed, but would record '.' as runpath.
- [[ ${CHOST} == *-aix* ]] ||
- append-ldflags "-L."
-
local dbmliborder
if use gdbm; then
dbmliborder+="${dbmliborder:+:}gdbm"
@@ -248,11 +278,18 @@
&& myconf="${myconf} --enable-framework=${EPREFIX}/usr/lib" \
|| myconf="${myconf} --enable-shared"
+ if [[ ${CHOST} == *-cygwin* ]] ; then
+ fpeconfig="--without-fpectl"
+ myconf="${myconf} ac_cv_func_bind_textdomain_codeset=yes"
+ else
+ fpeconfig="--with-fpectl"
+ fi
+
# note: for a framework build we need to use ucs2 because OSX
# uses that internally too:
# http://bugs.python.org/issue763708
- OPT="" econf \
- --with-fpectl \
+ OPT="" cpython_econf \
+ ${fpeconfig} \
$(use_enable ipv6) \
$(use_with threads) \
$( (use wide-unicode && use !aqua) && echo "--enable-unicode=ucs4"
|| echo "--enable-unicode=ucs2") \
--- PORTAGE/eclass/python.eclass
+++ OVERLAY/eclass/python.eclass
@@ -401,6 +401,64 @@
fi
}
+# see http://thread.gmane.org/gmane.linux.gentoo.devel/80633/focus=80635
+_python_prepend_cwd_ldpath() {
+ local new=()
+ local f
+ local done=no
+ for f in ${LDFLAGS} ; do
+ case "${f}" in
+ -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+ new+=( "${f}" )
+ ;;
+ -L*|-T*|--library-path*|--script*)
+ if [[ ${done} == yes ]] ; then
+ new+=( "${f}" )
+ elif [[ "${f}" == "-L." ]] ; then
+ # if it's somehow already there, don't
duplicate it
+ new+=( "-L." )
+ done=yes
+ else
+ new+=( "-L." "${f}" )
+ done=yes
+ fi
+ ;;
+ *)
+ new+=( "${f}" )
+ ;;
+ esac
+ done
+ [[ ${done} == no ]] && new+=( "-L." )
+ export LDFLAGS="${new[*]}"
+}
+
+# @FUNCTION: cpython_econf
+# @DESCRIPTION:
+# econf() substitute for use in dev-lang/python ebuilds
+#
+# On some platforms, it is neccesary to prepend "-L." to ldpath before
+# proceeding with python's configure process. Using cpython_econf()
+# instead of econf() will ensure that this is taken care of correctly
+# before python's
[gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
My god, I am a horrible self-editor. Sorry. Please ignore the magnum
opus above and allow me to try again.
In dev-lang/python*, we use
append-ldflags '-L.'
to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir). But, this doesn't work if
LDFLAGS contains "-L$(libdir)".
We could try to fix this like:
export LDFLAGS="-L. ${LDFLAGS}"
or so. That would cover 99.9% of the cases out there. But very rarely,
indiscriminately placing our '-L.' before every other clause in LDFLAGS
might cause an unanticipated side-effect.
The flag-o-matic patch in my previous post analyses LDPATH and finds the
first statement that might possibly cause the ld command-line
library-search-path to be impacted, and inserts the "-L." just before
that statement, hopefully achieving the desired result with the least
possible side-effects.
I wonder: do people feel this distinction is meaningful enough to
justify actually making it? Or am I just being an anal-retentive nut-job?
If you tried to read my first post, thanks for not offing yourself like
the guys who get stuck sitting next to Ted Striker in _Airplane!_
-gmt
[gentoo-dev] [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
[ "eclass/flag-o-matic.eclass" ]->8->
--- PORTAGE/eclass/flag-o-matic.eclass
+++ OVERLAY/eclass/flag-o-matic.eclass
@@ -117,6 +117,42 @@
return 0
}
+# @FUNCTION: prepend-ldpath
+# @USAGE:
+# @DESCRIPTION:
+# Place the specified ldpath into LDFLAGS before any options which could
+# add additional paths to ld's search path. Specifically, it will place
+# the new "/foo" into LDFLAGS as "-L/foo" just before the first
+# occurance matching any of the globs: '-L*', '-T', '--library-path*',
+# and '--script*', but not matching any of the globs: '-Tbss=*',
+# '-Tdata=*', '-Ttext=*', and '-Ttext-segment=*'. If no such match is
+# found, then this is equivalent to "append-ldflags -L".
+prepend-ldpath() {
+ local new=()
+ local f
+ local done=no
+ for f in ${LDFLAGS} ; do
+ case "${f}" in
+ -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+ new+=( "${f}" )
+ ;;
+ -L*|-T*|--library-path*|--script*)
+ if [[ ${done} == yes ]] ; then
+ new+=( "${f}" )
+ else
+ new+=( "-L${1}" "${f}" )
+ done=yes
+ fi
+ ;;
+ *)
+ new+=( "${f}" )
+ ;;
+ esac
+ done
+ [[ ${done} == no ]] && new+=( "-L${1}" )
+ export LDFLAGS="${new[*]}"
+}
+
# @FUNCTION: filter-lfs-flags
# @DESCRIPTION:
# Remove flags that enable Large File Support.
<-8<-
I think my code is probably fine, or if it's buggy, so be it. A prior
question is: does this have sufficient utility?
I "need" (not exactly... keep reading) something like this for python.
Currently python*.ebuild contains:
append-ldflags "-L."
which is required in order to pull in the newly-rebuilt
libpython${VERSION}.so at ebuild-time, instead of the system version
(perhaps it's only so on obscure platforms?).
The problem is, LDFLAGS may already have some library paths coming in
from the environment. In this case, we end up with something like:
LDFLAGS="-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L."
and python goes ahead links everything against, i.e.:
/random/prefix/usr/lib/libpython3.2.so,
or whatever is the version in question, so we achieve precisely nothing,
effecting the outcome we are trying to prevent (the problem manifests in
subtle, hard-to-diagnose ways, unfortunately).
In general, when we say append-ldflags -Lsomething, we may or may not
mean "always use something/libfoo, not $(libdir)/libfoo to link." In
the case where we do mean "always", append-ldflags is a broken approach.
An alternative would be to just replace
append-ldflags "-L."
with
export LDFLAGS="-L. ${LDFLAGS}".
however, there are problems with this:
First, it's aesthetically annoying that this spreads out the "-L*"
arguments into two different "zones" in LDFLAGS. No big deal, but it
definitely doesn't make visually scanning the logs any easier.
Second, although I'm not aware of any ld arguments which act as
"modifiers" to "-L" arguments (except the other -L arguments and
ldscript arguments, which the above patch scans for), ld does use
order-dependent patterns for other arguments.
For example: "-lfoo -( -lbar -lbaz -)" is not the same thing as "-(
-lfoo -lbar -) -lbaz". So order-dependencies might emerge in some
future binutils... in that case we'd still probably need to extend the
case statement above to include the new order-affecting arguments, but
at least we'd have a place in which to do so.
Third, although the meaning of -L* options may not be affected by other
arguments in an order-dependant manner, the reverse is not so. For
example, see --mri-script, --script (which both can affect and be
affected by the command-line library search path in an order-dependent
manner!), -rpath-link (SunOS only), etc...
One could certainly argue that, in practice, points two and three amount
to hopelessly obscure nitpicking and that nobody in their right mind
should be relying on this type of stuff in their LDFLAGS.
I'm not sure I'd state it quite so strongly as that, (after all these
might come just from profile.bashrc and be targeted to a particular
ebuild) but justification is clearly on the "thin" side.
Indeed, if we're going to worry about side effects, it's not entirely
clear that what what my patch does is safe. For example, if the
environment supplied "-L/foo/bar --script baz", and python for some
reason had a "baz" file in "${S}", then some kind of breakage would
surely happen when we did
prepend-ldpath "."
Also, we might also legitimately worry that the presence of this
function in flag-o-matic will somehow "encourage" peop
Re: [gentoo-dev] CVS -> git, list of where non-infra folk can contribute
Brian Harring wrote: 1) We need a thin manifest -> thick manifest converter. Thin manifests are used for git- they store just DIST entries. Thick (also known as 'full'), are what cvs/rsync users are familiar with- it holds checksums for all content. carebear is the current person volunteering to sort this (help may be appreciated, talk to him/her/it). heh :) I'll read up, spend some time on IRC, and see what I can do to help here. replay it into git via tailor; Never knew about that tool... not sure about the wisdom of adding an extra moving part just to keep the lights on for those few hours... Given the "2G of history" issue Diego mentioned, which if I understand correctly, effectively means that the future gentoo git can never rebase its commit history, why chance it? In my last experience with cvs->git (at the time I was building a rsync (binutils cvs)->git mirror for a client), the most difficult thing about cvs->git was trying to scrub the identity data. I don't remember the exact issue, but somehow, git had identity uniqueness constraints that cvs happily ignored, or something like that. I never thought to try using svn as an intermediate -- but I like that idea a lot and wish I had thought of it when I needed to. Anyhow, wrong ml for this, I'll subscribe to -scm. -gmt
Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss
If you're going to paint me and the other folks expressing opinions as entitled mouth-breathers, certainly you can't expect not to hear any reply because it's "off-topic"! On 10/1/2012 2:00 PM, Diego Elio Pettenò wrote: On 01/10/2012 13:54, Rich Freeman wrote: I don't think we can keep the discussion off -dev forever though. It seems like we're close to being able to implement, which means lots of changes that impact all devs. I can't imagine that we'd want to implement that without some kind of council vote. Perhaps the appropriate approach is to propose a GLEP? No, the appropriate approach is first to _talk with Infra_. Guys I said that before, but unless you actually factor in Infra when you want infrastructure stuff done, you have to do the work yourself. And it might not be nice and fun. Well I definitely can't argue with the above, and I didn't know about -scm but, hey, you learn something new every day; anyhow I'm not sure it's makes it inappropriate to discuss. So to close this in a few words: You walk the walk, you talk the talk. And I have no intention to read another mail with "what other awesome thing we can do if we migrate to git and we don't even have to worry about what it might happen on the serverside because git is just magical and will sort itself out", okay? OK, also agreed that git lacks entirely in magical properties -- I wasn't aware of having said anything suggesting otherwise. I have some limited experience as a git admin and am aware of git's considerable limitations with respect to provisioning, security configuration, scalability and other non-optional features for a public-facing deployment. There is more to be said on the matter but I'll take that to -scm. Anyhow, I get it: administering the vcs for a huge project such as Gentoo is very hard work. If I somehow gave some other impression, I'm sorry. Perhaps Rich and I insensitively voiced our shared assumption that Gentoo's continued reliance on cvs stems from a lack of motivation and consensus, rather than a shortage of labor and resources. Then again, if the folks in the trenches doing the work think I've slighted them, surely they are perfectly capable of chewing me out on their own behalfs and don't need you to do it for them? I'm not looking for a fight, but after reading the above and some other remarks in this thread, by you and others, I did want to at least clarify my position: No, of course I couldn't possibly know all the repercussions of a change like this, but I also find it difficult to believe that whatever hurdles exist are so intractable that we'd might as well just throw in the towel. Although I regret any bad feelings I may have caused, I stand by my statements. To be clear, the "magic" of the Gentoo community -- not git -- is what I believe will make this doable, and yes, I appreciate that that "magic" is actually just a lot of people doing a lot of nasty, thankless chores. As for the whole put-up/shut-up business, I'd be happy to help out any way I can (although tbh I'd still say whatever was on my mind even if I was too busy). I'll contact Infrastructure to make sure they're aware of me. -gmt
Re: [gentoo-dev] CIA replacement
On 10/1/2012 10:29 AM, Rich Freeman wrote: Looking at the tracker [1], we need a pre-upload hook (I'm not quite sure why), an rsync conversion script, the ability to validate the converted tree, and documentation. There is still an open bug for commit signing, and I'm not quite sure why as this was implemented. a job for gitolite? what does ogo use? Am I missing some big concern that just isn't obvious in these bugs? Things will break. There will be much wringing of virtual hands. But in 1 weeks' time, everyone will get over it and find ways to fix whatever broke. There will be one less thing on the big to-do list, and most people will be happier overall. It's one of those "Social Security" type of things it's hard to build a clean consensus around /when/ to make a break like this. I think we basically need a headstrong asshole in a position of sufficient authority to say, "fuck everyone, we are doing this, here is the changeover date." Too much democracy, tactical planning and consensus-building will almost certainly perpetuate the status quo. Look, Gentoo will probably fall to pieces for a few days. The alternative, IMO, is a huge, elaborate multi-phase plan of some kind, like how Microsoft or NASA would do it: armies of 40-somethings toil for months or years in cubicles... Elaborate Gantt charts and process diagrams are drawn up and approved by layer after layer of middle management. Gentoo can tolerate a hiccup; it's survived way scarier challenges. We have a huge community of very talented people highly motivated to keep this machinery working... there's literally zero chance that Gentoo will somehow have it's spirit broken by cvs->git. In other words, devs -- imo -- just go for it. Throw caution to the wind. Break some bylaws if need be. Ignore anyone saying "we're not ready." It'll be easier to get forgiveness than permission, and exactly what process issues needed to be smoothed over will frankly only truly be clear in the aftermath. Of course, that's easy for me to say since I'm not the one whose door everyone will be beating down when TSHTF. There are worse things than CVS (Visual SourceSafe comes to mind). But if we don't bite the bullet now, we all know we'll be having this discussion again later. -gmt
Re: [gentoo-dev] Re: [PATCH] eutils: Warn on built_with_use usage
On 9/17/2012 1:00 AM, Ciaran McCreesh wrote: On Mon, 17 Sep 2012 00:58:02 -0700 "Gregory M. Turner" wrote: Unless I'm missing something, it seems that once we deprive the ebuild developer of this feature, there is no simple, supported way to retrieve the information except to depend on it. has_version. On 9/17/2012 1:10 AM, Ralph Sennhauser wrote:> On Mon, 17 Sep 2012 00:58:02 -0700 > > has_version foo/bar[baz] can be used in EAPI 2 and later. > oh, duh... I guess I was just conflating the two, sorry for the noise :) -gmt
Re: [gentoo-dev] Re: [PATCH] eutils: Warn on built_with_use usage
My main focus here is switching built_with_use to actively nagging
> people to stop using it; this includes nagging EAPI0/1 users of it.
Sans the implementation details, anyone got complaints with the
intent?
I have a concern about it, yes. But, maybe there's a good answer to my
concern, so please consider this a friendly ebuild development question
disguised as a complaint :)
Unless I'm missing something, it seems that once we deprive the ebuild
developer of this feature, there is no simple, supported way to retrieve
the information except to depend on it.
The issue is that calculating dependencies is not the only reason we
might want to know if a package was built with a particular USE-flag,
and if we get rid of built_with_use, we literally cut ourselves off from
retrieving this information in any officially sanctioned way (except to
DEPEND on it, which may not be semantically correct).
I can think of all kinds of legitimate reasons we might want to know if
the installed such-and-such package was built with so-and-so use-flags
without depending on it. i.e.:
o if the current gcc falls within a certain range of version
numbers and was built with graphite, we are going to trigger
a compiler bug. Suppose that there is no graphite support
or dependency in ${P}, and that we can apply a patch which will work
around the bug, but at a performance cost in ${P} we'd rather not
pay unless we have to.
o We need to modify a Makefile based on how a package we
BDEPEND on was built -- but suppose there is no BDEPEND
/limitation/ to enforce -- in other words, either way, our package
will build, and there is no correlating reverse dependency to worry
about at runtime.
Such needs are fairly unusual, but they do come up in real life.
My concern is that this will lead to people doing things like:
o cut-pasting the old implementation of built_with_use into ebuilds,
-- but that implementation will break if the portage database layout
changes
o creating bogus one-off use-flags as a way of performing these queries
(and, thanks to the upcoming requirement that USE flags
always appear in IUSE, exposing those flags to the end-user, perhaps
with some confusing description like "whether such-and-such was
built with so-and-so").
o creating BDEPENDs of -- and sketchy parsers for -- portage-utils or
similar suites, just to ask this question.
Admittedly, it's hard to prevent people from doing
built-with-use foo/bar baz || die "${P} needs foo/bar with baz"
since, once upon a time, that was SOP, and we'd have to parse the bash
code or something to qa warn for it automatically.
But any number of similar prohibitions are simply documented in the
developer handbook, including this one.
Am I missing something, here? I kinda think we should go the opposite
direction and un-deprecate the API. It seems like we are cutting off
our nose to spite our face here.
-gmt
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 9/12/2012 5:58 AM, Ian Stakenvicius wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 12/09/12 05:55 AM, Gregory M. Turner wrote:
Note that, effectively, we have this already, and it's called
"portage". But one could certainly make a case for modularizing it
better, since, in truth, we are talking about a very common, very
abstract problem here which portage shares with any number of
batch-build systems.
Such an engine could very well do exactly the right thing if it
were faced with a constraint that a certain part of a certain build
needed to proceed without parallelism due to limitations coming
from the build.
Also, there are very large parts of most builds -- configure comes
to mind -- that don't parallelize even if, perhaps, they should.
In such cases, a really smart global parallelism arbiter could
easily respond by spawning more jobs from other builds.
So essentially what you're saying here is that it might be worthwhile
to look into parallelism as a whole and possibly come up with a
solution that combines 'emerge --jobs' and build-system parallelism
together to maximum benefit?
Yeah, couldn't have said it better myself ... apparently :)
Advanced HPC systems (sys-cluster/torque along with an appropriate
scheduler, for instance) can do such things with their jobs when the
jobs are properly built; I could see portage being able to handle this
as well given most of what is necessary is already known (ebuild
phases, build system type (via eclass), etc). However, given the
limitations already put on parallelism in terms of emerge order, etc,
I could see this solution needing to be -very- complex and integration
needing to occur on multiple levels. We'd also need to consider
distcc (and other cluster-shared compilation methods if there are
any??).. It would be an interesting project, though.
ACK all of the above.
Tempting to think more deeply about this but probably the last thing I
need to do right now is to talk myself into another speculative project.
I've hurt my wrist a bit -- probably an RSI -- so should help deter me :S
Only a few major sources of parallelism exist in portage: --jobs /
--load-average in emerge opts, multiprocessing eclass & equiv. ebuild
helper, distcc, and make... Infrastructure is already in place for all
of those, so perhaps a good holistic solution exists that isn't /too/
complicated.
...OK another f!#!%$^ brainstorm incoming :)
For "JOBS" syntax... what really seems missing in portage are:
o a clean way to say "dont parallelize this particular make
invocation" in ebuilds
o a clean way to globally say "try to use this parallelization
strategy when emerging."
So what about something like:
o EMERGE_JOBS and EMERGE_LOAD_AVERAGE make.conf vars equiv. to
--jobs and --load-average emerge options
o EBUILD_JOBS and EBUILD_LOAD_AVERAGE make.conf vars
o If the latter are not specified, they are copied respectively from
the former (debatable for *_JOBS, since now we get 16 processes when
we asked for four).
o MAKEOPTS is auto-extended to reflect EBUILD_JOBS/EBUILD_LOAD_AVERAGE
if & only if -j|--jobs|-l|--load-average options aren't provided in
make.conf/profile/envvar MAKEOPTS
o however, if MAKEOPTS "override" EBUILD_JOBS or EBUILD_LOAD_AVERAGE,
issue a conspicuous yellow-stars warning
o extend "emake" to accept a "--non-parallel" option which will
strip all -j|--jobs|-l|--load-average options from MAKEOPTS;
perhaps support an equivalent EBUILD_NON_PARALLEL envvar as well,
with support for override in profile.bashrc. Don't warn about this
overriding EBUILD_JOBS -- treat as SOP.
o debatable: respect EBUILD_NON_PARALLEL in multiprocessing, etc?
or, perhaps, something like:
EMAKE_NON_PARALLEL=${EMAKE_NON_PARALLEL:-${EBUILD_NON_PARALLEL:-no}}
could be used to distinguish between "don't use any parallelism"
and "don't use GNU's make parallelism in emake". Also maybe a
better name exists that doesn't use double-negatives.
?
Seems to me something vaguely like the above would provide
o backward compatibility for ebuilds and make.conf
o not so vastly different than what we have
o a decent way to specify what "we really want" globally;
insofar as portage doesn't do the best job effecting the requested
parallelization strategy, more ambitious tactics could be
implemented later, hopefully without huge interface revisions.
-gmt
P.S.:
(Kind-of-crazy additional idea: put ceil(sqrt(EMERGE_JOBS)) into
EBUILD_JOBS when only the former is specified, and then let
effective_emerge_jobs equal floor(EMERGE_JOBS/EBUILD_JOBS) but maybe
too much automagic for this to be a good idea.)
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 9/11/2012 9:54 AM, Ian Stakenvicius wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/09/12 12:43 PM, Zac Medico wrote: On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. I personally wonder about why this would be necessary from the perspective of the user; if the user's system at emerge time can handle X concurrent processes per emerge-job , i don't see why it would matter what phase these jobs would be launched from. At the ebuild level, certainly, but that's one of the reasons for EJOBS in the first place, so that it can be overridden consistently within a phase, if necessary for the ebuild (regardless of build system type), right? I would hope so. There are definitely real-world reasons to want to restrict jobs -- usually to just one -- in a particular phase... several ebuilds, notably several in @system, do this already by injecting -j1 somewhere. To say it's all about performance seems to be forgetting a major reason somebody might want to do this as a user or an ebuild author. Hmm... I should preface this by saying that it really pains me to say this, because now this just starts to seem so fucking complicated -- but now that I think about it, this really seems to highlight a kind of semantic discrepancy that this thread has mostly dusted under the rug. There are really TWO things we seem to care about -- one is some kind of global build-parallelism frob-set, and the other is the ability to turn off all parallelism for certain parts of an ebuild and those are actually kind-of orthogonal. In other words, imagine if we had some kind of "global parallellism arbiter" thingy that looked at EJOBS or ELOADAVG or maybe some other things we haven't thought of yet, and decided things like: o should portage start a new parallel emerge process? o What is the appropriate "MAKEOPTS" for the "emake" that foo-1.0.ebuild just issued? or even, one could imagine, such things as: o should portage decide that parallelism has gotten out of control somehow and suspend or kill a running ebuild for subsequent resuming or restarting? o should portage re-nice some of its processes? Note that, effectively, we have this already, and it's called "portage". But one could certainly make a case for modularizing it better, since, in truth, we are talking about a very common, very abstract problem here which portage shares with any number of batch-build systems. Such an engine could very well do exactly the right thing if it were faced with a constraint that a certain part of a certain build needed to proceed without parallelism due to limitations coming from the build. Also, there are very large parts of most builds -- configure comes to mind -- that don't parallelize even if, perhaps, they should. In such cases, a really smart global parallelism arbiter could easily respond by spawning more jobs from other builds. Not sure what I'm suggesting we do about it, exactly, but just pointing out that maybe a completely "correct" solution requires a much more elaborate implementation than just a bunch of syntactic sugar around what we currently call MAKEOPTS. Whether or not Gentoo wants to take that all on, right now, as part of the next EAPI, is certainly debatable -- in fact, I'm inclined to say maybe it's not the best idea. Perhaps we should really be asking: is the status-quo really so problematic/inelegant that it needs fixing? -- before we decide how best to fix it. -gmt
Re: [gentoo-dev] Re: News item 1: changes to stages (make.conf and make.profile)
On 9/10/2012 10:39 PM, Duncan wrote: Gregory M. Turner posted on Mon, 10 Sep 2012 20:29:53 -0700 as excerpted: However, IIRC, /etc/make.conf is just ignored by portage if /etc/portage/make.conf is present, so symlinking, or even better, if possible, hardlinking those files would probably "do the right thing" for legacy tools that don't know about the new location... unless I'm mistaken, which is always plausible :) Thanks. Reasonable approach and good to know. Well, I did warn about the likelihood I was wrong :) Pretty clever 'cause that way even when I'm wrong (as turns out to have been the case), I'm still right :P In all seriousness, if both of them are sourced, then could one get away with something like this? /etc/make.conf: source /etc/portage/make.conf /etc/portage/make.conf: if [[ __GENTOO_MAKE_CONF_ONCE == gotit ]] ; then __GENTOO_MAKE_CONF_ONCE=gotit . . . endif or are conditionals disallowed? As Zac mentions, hopefully it's harmless to duplicate things, but, personally, I would worry about the effect of duplicates on performance, and also in PORTDIR_OVERLAY. Plus, it just seems dirty. -gmt
Re: [gentoo-dev] Re: News item 1: changes to stages (make.conf and make.profile)
On 9/9/2012 6:34 PM, Zac Medico wrote:
On 09/09/2012 05:59 PM, Duncan wrote:
To your knowlege (IOW have you tested) having /etc/make.conf either a
symlink to /etc/portage/make.conf or a simple one-line
"source /etc/portage/make.conf"?
I've tested them both just now, and they work for me. Why wouldn't they?
If both /etc/portage/make.conf and /etc/make.conf were evaluated, stuff like
FOO="${FOO} bar"
could cause, i.e., duplications... not sure what all the rules are
limiting what one can and can't put in make.conf, but one could imagine
all kinds of wacky stuff.
However, IIRC, /etc/make.conf is just ignored by portage if
/etc/portage/make.conf is present, so symlinking, or even better, if
possible, hardlinking those files would probably "do the right thing"
for legacy tools that don't know about the new location... unless I'm
mistaken, which is always plausible :)
-gmt
Re: [gentoo-dev] On flags being in IUSE (and the prefix USE-flag in particular)
On 9/7/2012 10:32 AM, Fabian Groffen wrote: With the introduction of IMPLICIT_IUSE (scheduled for EAPI 5), a phrase has been added to PMS, that finally makes a statement on what's supposed to be in IUSE, and what not[2]. To me, this patch means that things like userland_BSD, elibc_glibc, etc. do *NOT* belong in IUSE of an ebuild/eclass (and hence should b removed). 'prefix', on the other hand, should be added to IUSE of those ebuilds/eclasses that use them. What, exactly, is the difference -- the principle behind the "should"s above? USE_EXPAND? Probably more a problem of me being lazy than anything being wrong with it, but [2] reads like Greek to me. For EAPI 5 (assuming it contains IMPLICIT_IUSE) the base profile can be enriched with IMPLICIT_IUSE="prefix". For all currently Council approved EAPIs this means 'prefix' has to be added to IUSE. I haven't looked into IMPLICIT_IUSE too carefully, but ... shouldn't this be... implicit? Sorry, I'm being super lazy and not reading anything here. In case you wonder why this is a problem now, Portage/repoman has a rule that USE-flags that are masked in the profiles implicitly are defined. Probably making a total ass of myself at this point but... could you define "defined"? I'm guessing I'd understand how to get flags masked implicitly if I read the IMPLICIT_IUSE stuff? Or do you mean "are defined implicitly" (in which case, again, I don't see why we'd need to make them explicit). [2] http://git.overlays.gentoo.org/gitweb/?p=proj/pms.git;a=commitdiff;h=d9040ab3482af5f790368bac5d053bf1cd760ba8;hp=f9f7729c047300e1924ad768a49c660e12c2f906 Apologies for these questions -- in my defense, being both lazy and ignorant puts me at a real disadvantage here :) -gmt
Re: [gentoo-dev] ROOT, EROOT, and EPREFIX in scripts (was: relative ROOT: correct behavior when ROOT=)
On 9/6/2012 5:56 AM, Ian Stakenvicius wrote:
On 06/09/12 03:55 AM, Ulrich Mueller wrote:
On Thu, 06 Sep 2012, Gregory M Turner wrote:
${ROOT:=/} EPREFIX="@GENTOO_PORTAGE_EPREFIX@"
EROOT="${ROOT}${EPREFIX}"
When ROOT is undefined or empty, this script will assign "//foo"
to EROOT and bad things will happen in Cygwin.
Several correct-ish solutions exist, i.e., in the above we could
change the concatenation statement to read:
EROOT="${ROOT}${EPREFIX#/}"
I'd rather do it the other way around: EROOT=${ROOT%/}${EPREFIX}
Reason: EPREFIX is guaranteed to start with a slash, whereas for
ROOT I wouldn't be so sure that it always ends with one.
Not to mention that `man 5 ebuild` specifically defines EROOT to be
"${ROOT%/}${EPREFIX}/" already , so all that's needed is the removal
of the final '/' if I'm reading this correctly..
Yeah, that trailing slash is kinda sneaky because it saves the day in
the decidedly non-pathological "normal" case where ROOT is undefined and
EPREFIX is empty.
That's another kind-of gross inconsistency that crops up with some
regularity. In portage python code, EROOT does, indeed, always end in a
slash, just as man 5 ebuild specifies.
But in some places, i.e., in eselect, EROOT does not to end in a slash,
or even has trailing slashes stripped from it. In the case of eselect
(which I'm just using as an example), this "quirk" has a bunch of
consumers: the eselect plugins use code like:
foo="${EROOT}"/usr/bar
which, if it were in an ebuild, would instead be properly written as:
foo="${EROOT}"usr/bar
Indeed, most ebuilds do this correctly although occasionally I've seen
exceptions. My overlay is for Prefix so it "lucks out" if ebuild
authors forget this and add the slash; but if/when non-prefix cygwin
support is resuscitated in Gentoo, all of those extra slashes need to be
fixed "or else."
Anyhow, my point is that this tendency to effectively standardize subtly
inconsistent usage of EROOT is pretty !@#$&*ing gross.
I'd meant to save this "argument" for a separate thread, since a) now my
original question isn't likely to get answered and b) I might seem to be
bitching and moaning, as I don't, yet, have any patches to share -- but
IMO if a bunch of scripts are going to call a variable "EROOT" in
Gentoo, it seems like a no-brainer that the variable should have the
same meaning in all of them.
This isn't the case right now, and has led to some real-world problems,
so my feeling is that there ought to be some official documentation
explaining in clear, simple terms:
o what is the idea/point behind ROOT, EPREFIX and EROOT
o where these variables (legitimately) come from, in other words,
when can a user or developer inject them, and when should he
or she be retrieving them from portage (i.e.: ebuilds).
o the circumstances under which these variables must be changed
in-flight, especially, for relative-ROOT activation
o how to properly cook these variables when writing a script
that will interact with portage (specified in a way that does
the right thing for invokers and invokees, alike).
o any EAPI interactions with the above (don't think we have any
yet(?) but some recent proposals seem poised to change this).
Once we have that right, it's just a matter of changing all the scripts
to do what the official documentation says -- if we can't, then
obviously it's back to the drawing board to fix the specification,
lather, rinse, repeat.
I'd be happy to do this myself but I'm not sure I know all the correct
answers to the above questions (hence my original thread topic, for
example), nor am I 100% sure what the correct place would be to document
this (probably the dev manual or the wiki?)
An alternative to documenting and synchronizing code would be to build a
source-able library script to automatically handle this type of thing.
Maybe that's overkill, or maybe it's brilliant, I'd be interested to
hear people's opinions. I'm pretty sure we are just talking about a
very few lines of simple code so my gut instinct is that it's overkill
-- but if we expect things to get more complicated in the future, then
centralizing the code, now, could save us some hassles, down the line.
tl;dr: we should standardize EROOT, but how?
-gmt
Re: [gentoo-dev] relative ROOT: correct behavior when ROOT=
On 9/6/2012 12:55 AM, Ulrich Mueller wrote:
On Thu, 06 Sep 2012, Gregory M Turner wrote:
Several correct-ish solutions exist, i.e., in the above we could change
the concatenation statement to read:
EROOT="${ROOT}${EPREFIX#/}"
I'd rather do it the other way around:
EROOT=${ROOT%/}${EPREFIX}
Reason: EPREFIX is guaranteed to start with a slash, whereas for ROOT
I wouldn't be so sure that it always ends with one.
Good point, but EPREFIX is empty for non-prefix gentoo. So, I guess:
EROOT="${ROOT%/}/${EPREFIX#/}"
It's a contrived example anyhow, so it's mostly academic.
My main concern is the correct behavior when ROOT is defined, but empty
-- I mostly included this explanation to preempt anyone inclined to ask
"what are you trying to achieve?" :P
-gmt
[gentoo-dev] relative ROOT: correct behavior when ROOT=
Hello, in my overlay I need to fix a bunch of issues that crop up when
implementing EPREFIX construction in scripts due to Cygwin's
idiosyncratic, but POSIX-compliant, handling of paths beginning with
"//" (Cygwin does some arguably pathological stuff when such paths are
used).
Almost all of these stem from the careless tacking of "ROOT" onto
"EPREFIX" in bash-scripts, i.e., as in this (made-up) bash-script snippet:
${ROOT:=/}
EPREFIX="@GENTOO_PORTAGE_EPREFIX@"
EROOT="${ROOT}${EPREFIX}"
When ROOT is undefined or empty, this script will assign "//foo" to
EROOT and bad things will happen in Cygwin.
Several correct-ish solutions exist, i.e., in the above we could change
the concatenation statement to read:
EROOT="${ROOT}${EPREFIX#/}"
Thats a matter for another thread, however.
As I've been reading code like the above, I've stumbled across some bugs
and inconsistencies pertaining to relative-ROOT and I've been trying to
file bugs to get these fixed.
One such issue pertains to the case where the ROOT environment variable
is defined, but empty. Portage and most python code seem to treat this
as equivalent to ROOT="${PWD}", which is to say, that relative-root
functionality is triggered by this circumstance.
However, several app-admin and sys-devel scripts "disagree" and treat
this as equivalent to ROOT="/" (as does my example above).
So... which behavior is correct? Since I'm aware of the issue, I'd
might as well file bugs against the incorrect ones and get everything
consistent.
-gmt
Re: [gentoo-dev] Re: prune_libtool_files() and pkg-config dependency
On 8/31/2012 4:48 AM, Rich Freeman wrote: On Fri, Aug 31, 2012 at 6:42 AM, Michał Górny wrote: So please introduce virtual/compiler, virtual/linker, virtual/posix-system, virtual/sratatata and add them to DEPEND of every single ebuild. Every ebuild doesn't need all of those - that is the whole point. As Duncan already pointed out, reducing @system is a goal, but it doesn't mean that we need to get there overnight. However, we'll never get there if we keep going backwards. My 2c on this: I'm reluctant to make "sweeping statements" like this, for any number of reasons, but -- well, I'm gonna. IMO, getting there by slow evolution is not the right way. At some point, @system becomes so primitive that bootstrapping must come to depend on more than @system, or we have to add add more "phases" to the bootstrap process, or split @system up into smaller sets or something. The point is, we can't gradually reach a goal that's incompatible with the fundamental premise. It's all well and good to say "let's not put more stuff into @system because we want it to shrink," but, as it stands, there's a de-facto policy that @system includes everything needed to have a reasonably functional Gentoo, including all of the compilers, development tools and interpreters portage, gcc, and your rc system of choice rely on. Until that fundamentally changes, IMO what belongs in @system is whatever best suits its current purpose. For the record, I'm not saying we need to put pkgconfig in - I'm totally agnostic about that, as I am about whether it should be brought in as a dependency. I just mean, probably the best way to fix the fat-@system problem is to create some kind of vision for a more-modular "Gentoo of the Future" first, and create a roadmap for getting there, second. Its possible, perhaps even likely, that if we try to go the incremental route towards @system reduction, we will find, along the way, creative solutions to the various issues that have kept it fat-ish so far. But that's likely to lead to a fairly ad-hoc patchwork of hacks, which imo would most likely be inferior to what could be achieved with some kind of destination in mind (even if that destination is subject to major revision as Gentoo progress toward it). -gmt
[gentoo-dev] cygwin: gmp/mpc: package.use.mask static-libs? (was: supporting static-libs)
On 8/28/2012 4:05 PM, Diego Elio Pettenò wrote: On 28/08/2012 15:36, Mart Raudsepp wrote: static-libs is for installing static libraries IN ADDITION to shared libraries, not instead. USE=static is for what you have in mind there. PE is not the same as ELF so on Windows you either build one or the other for a number of reasons. In this case, the issue is purportedly that the headers get generated differently, depending on whether you build static or dynamic. I don't think too much, unless I have to, about what kind of libtool/gcc/gentoo magic makes it all possible, but by and large, static and dynamic libraries "just work" in my overlay. So if gentoo-prefix would create both static and dynamic for a given library on elf, the same thing happens in my overlay (I mean that this typically happens without any hand-holding, on my part, to make it work). Now on a different note, this is not even what USE=static is for — but that's way behind what we were discussing before. euse -i says: static - !!do not set this during bootstrap!! Causes binaries to be statically linked instead of dynamically Pretty sure I've done the right thing in my overlay by masking, but it can't hurt to mention that I'm always happy to hear any suggestions or ideas (although of course I reserve the right to politely decide that your ideas are crap and ignore them :P). My reasoning for the flag-mask is that Gentoo always builds the dynamic libs for these packages and therefore so should my overlay. Unless I fix the underlying conflict or implement some kind of side-by-side scheme (a la libncursesw), I really have no choice but to forbid building statically (note that cygports adopts the same solution). Masking the use-flag for the affected atoms accomplishes this prohibition exactly, exclusively, and without any kludge (except unfortunately I still have to patch the e-build, as, for some goofy reason, configure doesn't interpret the static/dynamic options the same way after applying the cygports patches -- hopefully I can fix that just by leaving some patches out; I just haven't gotten to it yet). -gmt
Re: [gentoo-dev] supporting static-libs
On 8/28/2012 1:09 AM, Michał Górny wrote:
On Tue, 28 Aug 2012 02:15:40 +0200
hasufell wrote:
static-libs
pointless
I have to mask this flag for dev-libs/{gmp,mpc} in my cygwin overlay,
where one can have static or dynamic, but not both, as per. upstream
requirements (no idea why). So FTR, this is not always a matter of
personal taste.
-gmt
Re: [gentoo-dev] CWD-relative ROOT support in portage: misfeature?
On 8/18/2012 5:50 PM, Ian Stakenvicius wrote: On 2012-08-17, at 11:00 PM, "Gregory M. Turner" wrote: greg@fedora64vmw /tmp $ mkdir foo greg@fedora64vmw /tmp $ ROOT=foo portageq envvar ROOT /tmp/foo/ Does /anybody/ use this feature? Sorry for the HTML response... am on the road. I don't use the feature but I would fully expect said behavior. ie, going with the example above I would expect that I'd need the / in front for the path to not be relative. A user and maintainer of this (vapier) has emerged. I pooh-poohed the relative-ROOT idea when I discovered it a few days ago, but I've flip-flopped. I was concerned it would be exploitable by Bad People(tm), but I think it's no more exploitable than absolute-only ROOT, so long as its implemented correctly. So far, nobody's turned up to advocate against the status quo (except me, but I'm fine with it now), so I think the matter can be considered resolved. -gmt
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On 8/16/2012 6:26 PM, Rich Freeman wrote:
On Thu, Aug 16, 2012 at 4:05 PM, Michael Mol wrote:
The limited-visibility build feature discussed a week or so ago would
go a long way in detecting unexpressed build dependencies.
[snip]
If portage has the
dependency tree in RAM then you just need to dump all the edb listings
for those packages plus @system and feed those into sandbox.
That just requires reading a bunch of text files and no searching, so it
should be pretty quick.
Portage could hypothetically compile such a list while it crawls the
package dependency tree, but I suspect the cost will not be small as you
predict.
As far as I can tell the relevant calls to
check for read access are already being made in sandbox already, and
obviously they aren't taking forever. We just have to see if the
search gets slow if the access list has tens of thousands of entries
(if it does, that is just a simple matter of optimization, but being
in-RAM I can't see how tens of thousands of entries is going to slow
down a modern CPU even if it is just an unsorted list).
I appreciate your optimism but I think you're underestimating the cost.
Can't speak for others, but my portage db's churn too much for comfort
as is. Once we start multiplying per-package-dependency iteration by
the files-per-package iteration, that's going to be O(a-shit-load).
Of course, where there's a will there's a way. I'd be surprised if some
kind of delayed-evaluation + caching scheme wouldn't suffice, or,
barring that, perhaps it's time to create an indexed-database-based
drop-in replacement for the current portage db code.
I've enclosed some scripts you may find helpful in looking at the
numbers. They are kind-of kludgey (originally intended for
in-house-only use and modified for present purposes) but may help shed
some light, if they aren't too buggy, that is...
"dumpworld" slices and dices "emerge -ep" output to provide a list of
atoms in the complete dependency tree of a given list of atoms (add
'@system' to get the complete tree, dumpworld won't do so).
"dumpfiles" operates only on packages installed in the local system
(non-installed atoms are silently dropped), and requires/assumes that
'emerge -ep world' would not change anything if it is to give accurate
information. It takes a list of atoms, transforms them into the
complete lists of atoms in their dependency tree via dumpworld, merges
the lists together, and finds the number of files associated with each
atom in portage. Any collisions will be counted twice, since it doesn't
keep track. It also doesn't add '@system' unless you do. By default it
emits:
o A list of package atoms and the files owned by each atom (stderr)
o total atoms and files
o average filename length
What is, perhaps, more discouraging than the numbers it reports is how
long it takes to run (note: although I suspect an optimized python
implementation could be made to do this faster by a moderate constant
factor, I'm not sure if the big-oh performance characteristics can be
significantly improved without database structure changes like the ones
mentioned above).
My disturbingly bloated and slow workstation gives these answers (note:
here it's even slower because it's running in an emulator):
greg@fedora64vmw ~ $ time bash -c 'dumpfiles @system 2>/dev/null'
TOTAL: 402967 files (in 816 ebuilds, average path length: 66)
real15m33.719s
user13m18.909s
sys 2m8.436s
greg@fedora64vmw ~ $ time bash -c 'dumpfiles chromium 2>/dev/null'
TOTAL: 401300 files (in 807 ebuilds, average path length: 66)
real15m28.900s
user13m15.126s
sys 2m8.088s
My workstation is surely an "outlier" as I have a lot of dependencies
and files due to multilib, split-debug, and USE+=$( a lot ). It's also
got slow hardware Raid6 and the emulator only gives it 2G of ram to work
with. But I'm a real portage user; I'm sure there's other ones out
there, if not many, with similar constraints.
-gmt
#!/bin/bash
if [[ x$(qlist -IC app-portage/portage-utils)x == xx || \
x$(qlist -IC app-portage/gentoolkit)x == xx ]] ; then
echo "This utility requires both app-portage/portage-utils" >&2
echo "and app-portage/gentoolkit. Emerge them both and try again." >&2
exit 1
fi
declare -a arguments atoms
arguments=( )
atoms=( )
verbose=yes
redic=no
for arg in "$@" ; do
case $arg in
-q|--quiet) verbose=no ;;
-r|--redic) redic=yes ;;
*) arguments=( "${arguments[@]}" "$arg" ) ;;
esac
done
[[ ${#arguments[*]} == 0 ]] && arguments=( '@world' )
for arg in "${arguments[@]}" ; do
if [[ ${arg} == @* ]] ; then
newatoms=( "${arg}" )
else
newatoms=( "$( qlist -eICv "${arg}" | sed 's/^/=/' )" )
fi
newatoms=( $( dumpworld "${newatoms[@]}" ) )
result=$?
[[ ${result} != 0 ]] && { echo "dumpworld failed, giving
[gentoo-dev] CWD-relative ROOT support in portage: misfeature?
It has come to my attention that gentoo supports "relative" ROOT, which
is to say that, by design, portage will act as though (in bash terms):
ROOT
equals
"${PWD}/${ROOT}"
when (again in bash terms):
[[ $ROOT != /* ]]
at the moment execution crosses the boundary between a non-portage
program and a portage program. For example, I ran the following from a
bash-prompt with PWD=/tmp in a portage-2.2 ~amd64 environment:
greg@fedora64vmw /tmp $ mkdir foo
greg@fedora64vmw /tmp $ ROOT=foo portageq envvar ROOT
/tmp/foo/
Question: do we really want this behavior?
I have reason to believe that almost nobody uses this feature (namely,
gcc-config and binutils-config are both broken under it for ages and
nobody filed a bug or fixed it: see bugzilla #431104).
Does /anybody/ use this feature? If not, I'd suggest that the portage
team might ask itself whether the benefits of continuing to maintain it
are greater than the hassle and potential for error it facilitates.
Just my 2c,
-gmt
Re: [gentoo-dev] remove system set?
On 8/16/2012 8:26 PM, Michael Mol wrote: Ideally, you'd want as narrow a bootstrapping channel as possible. I guess I tend to think that, too, and I'm pretty sure it's correct. But I don't normally think about why, and since you've prompted me to do so, perhaps it's a good moment to interject the fairly obvious, but amusingly contrary notion that there is too much of a good thing to be had in this dimension. I suppose the main reason to want a minimal pre-bootstrap tool-chain is to maximize repeatability, minimize platform quirks, maximize time-stability in the face of changing code bases, and so forth. Plus... you know... it's a lot more likely to make somebody say "Wow!" the "narrow" way. Which maybe sounds like I'm poking fun, but I do actually think there's some intrinsic value in that. However, there /are/ also reasons to make the bootstrap-er machine more fat and complex. Most of them boiling down to there only being so much time in a day. Otherwise we would boot-strap from stage -10, consisting only of a sed script and some architecture files to generate crude asm lexers :) -gmt
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On 8/16/2012 4:59 AM, Rich Freeman wrote: On Wed, Aug 15, 2012 at 3:18 PM, Michael Mol wrote: It also sounds like something like that could be a benefit to shrinking @system. I think the solution to the circular dependency issue isn't to make Portage able to completely bootstrap the whole system, but rather just to make it capable of coping with the issues and knowing when to raise an alarm. Gentoo will always involve extracting a tarball/etc for the initial installation since you always need SOMETHING to start with. You can't even chroot into your install directory without a shell being there, and typing "emerge" won't go so well if portage isn't actually installed. So, continue to build stages like we do right now - no doubt with hard-coding and such to get around the dependencies. As far as objections to listing gcc and such in every ebuild go, why not? We list all kinds of routine stuff in hundreds of ebuilds so that we can install systems without them. Why not just have a toolchain virtual or something? And since ssh was brought up - this is what happens with hacks like this. When you combine the "default install" with the "minimum deps for everything" list you end up with an ssh you can't get rid of without the package.provided hack (which really should be used for stuff that is, well, provided). It would be nice if people who want to build a server with Gentoo but then reduce it to only true RDEPENDS could do so. Obviously they'd have to use binary packages to continue to maintain it (and even then they'd need to keep portage on it), or they'd have to build another one. Actually, the trend in general is towards disposable servers anyway so generating an entire new server every time one thing changes is probably a desirable thing, since you probably want to be able to do it every time you add a server anyway. tldr: I like, approve and otherwise +1 the idea of somehow paring down or eliminating @system but I think it's going to be fairly challenging, so more discussion on this topic is warranted in my humble non-developer opinion :) -- I really like everything you have to say here. Unfortunately, assumptions of toolchain availability have gotten into the DNA of Gentoo in ways that make it nontrivial -- although probably not rocket science, either -- to implement these ideas. I'd say it's the kind of thing where somebody needs to do the work. I think there is demand for this, but when it comes down to brass tacks, people who really need features like this can just write a script to push some tarballs or files around in a way that's "good enough" for their purposes. What is the cost/bene for a single sys-admin to do all the work and politics of making this change? However, staying with the cost/bene theme, we have here a kind of externality, as they say in economics, (which is a fancy way, I guess, of saying a bad decision or a raw deal), because, in the aggregate, I think it's pretty clear that the cost/bene favors doing that work. To be clear, I don't have religion about getting rid of @system, per-se, but I do have religion about the stuff Larry the Cow told me when I first visited the Gentoo homepage in 2001, or whenever, which was, basically, that the software I was using had a bunch of frobs that I couldn't touch, because I was running an rpm- or .deb-based system, and that Gentoo was going to let me frob them. It's not a total disaster, even now -- a determined sysadmin can absolutely do this right now with features like prefix, ROOT, binpkg and so forth but /really/ fixing this, so that non-standard/minimal setups "just work", would allow Gentoo to effectively address a whole bunch of really practical, real-world use-cases -- use-cases Gentoo is in many aspects uniquely suited to address, due to Larry the Cow's brilliant insights -- yet, by-and-large, due to precisely this @system thing and the package-management decisions that have stemmed from it, for which Gentoo has become unsuitable or impractical. Specifically, I'm talking, here, about managed LAMP servers, big-data clusters, and embedded. I suppose I'm not doing much to fix it by ranting and raving like this however. So see first paragraph :) -gmt
[gentoo-dev] a libtool + multilib gentoo host + 64->32 cross-prefix problem: a request for eyeballs
I've stumbled onto an interesting little problem trying to build my 64->32 cross-prefix, which I bootstrapped with a no-multilib 32-bit crossdev toolchain. Please see https://bugs.gentoo.org/show_bug.cgi?id=430722, especially the final two posts. I'd appreciate input on the correctness/feasibility of the various solutions I propose there. Apologies in advance for the verbose, stream-of-consciousness tone, it's more of a brainstorm than a bug at this point, and in retrospect, bugzilla probably wasn't the right place for it. -gmt
Re: [gentoo-dev] Portage FEATURE suggestion - limited-visibility builds
On 7/26/2012 11:26 AM, Rich Freeman wrote: I've been messing around with namespaces and some of what systemd has been doing with them, and I have an idea for a portage feature. But before doing a brain dump of ideas, how useful would it be to have a FEATURE for portage to do a limited-visibility build? That is, the build would be run in an environment where the root filesystem appears to contain everything in a DEPEND (including @system currently) and nothing else? It might be useful both in development/testing, and also in production use (not sure how performance would work in the real world - I was able in a script to get it to build an enviornment in a few seconds for a few packages). In practice I think it's going to be very hard to make this work in a platform-independent way; however in principle this is a ridiculously sexy idea that has crossed my mind more than once. The challenge is that it requires either o Building very large sandboxes on a per-package basis or o Python-level access to unionfs/aufs-style COW features. Imagine the tree of dependencies which would need to be thrown together for, i.e.: kmail or firefox! This makes the former approach seem damn nearly infeasible. The latter approach holds more promise, I think, but represents a pretty big development effort. Still very sexy idea, if a python-fs-layering API could be coded up. One thing to consider: even if it does work, continuing to support the "old" way without fancy COW features is going to be required if portage is still going to support Gentoo/Alt in all of its flavors (either that, or unionfs/aufs features would need to be coded up for all those platforms that lack them). -gmt
Re: [gentoo-dev] x32 release candidate
- Original Message - > i'm pleased to announce the initial x32 release candidate: > http://dev.gentoo.org/~vapier/x32/stage3-amd64-x32-20120605.tar.xz Also pleased to hear this! Thanks! Can't wait to find the time to play with it. Did you do all that work yourself? Is there a wiki or forum thread somewhere where folks can gloat and/or commiserate? I'm cautiously hopeful/enthusiastic about x32's prospects and a prefix seems like a fantastic way to bootstrap a community which could blueprint "cheat sheets" for prospective binary package maintainers (potentially driving adoption -- at least in my fantasies -- in the Grandma Zone). -gmt
Re: [gentoo-dev] cygwin prefix patch: request for eyeballs
- Original Message - > > 'if not os.environ["PORTAGE_PYTONPATH"]:' > If PORTAGE_PYTHONPATH is not in os.environ then it will raise a > KeyError, that is why we are doing a contains to begin with. I somehow got the idea that the python gods had sprinkled magical syntax-sugar on bool(x[y])? But maybe that doesn't make sense, it could cause gross surprises. Anyhow I should rtfm rather than post my OT un-educated guesses... -gmt
Re: [gentoo-dev] cygwin prefix patch: request for eyeballs
- Original Message -
> On Mon, Mar 26, 2012 at 4:12 PM, Gregory M. Turner
> wrote:
> > https://github.com/gmt/gmt-cygwin-overlay/blob/master/sys-apps/portage/files/portage-2.2.01.20271-cygdll_protect.patch
> Consistency in the style would be nice.
>
> For instance in cygdll-update:
>
> # Here you have spaces after the $( and before the )
> extra_protection="$( portageq cygdll_protect_list ${EROOT} )"
>
> # Later on you don't put spaces...
> eval $(portageq envvar -v CYGDLL_PROTECT PORTAGE_HOSTNAME \
> + PORTAGE_CONFIGROOT PORTAGE_INST_GID PORTAGE_INST_UID \
> + PORTAGE_TMPDIR EROOT USERLAND)
yes, good point, (the latter is cut-and-paste from etc-update, the former my
own; I do that habitually as a way to avoid learning how tokenization works in
bash :)
> Assuming $PORTAGE_BASH is always bash, you should use [[ consistently
> in your bash.
vs. "["? That sounds like cut-paste from etc-update again -- it seems to have
been coded with sh support in mind. Don't think that matters for cygwin (in
fact it'd better not, the way I coded it), so, well spotted, I'll fix it.
> IMHO it is a common mistake to write your own argument processing, is
> there a reason getops is not appropriate (portability?)
Got me there -- I'm a getopts ignoramus.
> Onto the python:
>
> if os.environ.__contains__("PORTAGE_PYTHONPATH"):
>
> Any reason why
> if "PORTAGE_PYTHONPATH" not in os.environ:
> was not used?
indeed, couldn't I just use 'if not os.environ["PORTAGE_PYTONPATH"]:' (if the
python executable were "False" or "0" would something stupid happen?)? Also
cut-paste from portage, but I did notice it, and had the exact same question.
After my initial, unprintable reaction :)
Maybe the answer is buried somewhere in the docs ... afaik in 2.7, "x in foo"
<=> "foo.__contains__(x)" (?). Guess I'd need to double check 2.7 and 3.2 docs
as well to be sure.
One idea: maybe meant as future-proofing against security bugs that could open
up in future pythons. I'll try to find blame (or whatever they call that in
cvs) on this line of code, maybe we can ask the culprit himself.
> Python functions are not perl, don't pass 'argv' to the function.
>
> If the function takes to arguments, it should be:
>
> def func1(arg1, arg2)
> If some of the args have defaults:
> def func1(arg1, arg2=5)
Hm, are you looking at bin/portage_master_lock when you mention this? It's
cut-and-paste from portageq (starting to see a pattern yet? :) )
> I've never seen anyone use <> before; I didn't even know it existed.
> Most folks use !=.
Sounds right, I'll seek and destroy.
> I can't really imagine why people do stuff like share
> /var/lib/portage
> across machines, so adding a bunch of complexity just to make it work
> seems nuts to me. That being said I have not worked on portage in
> years and Zac seems happier to support weird use cases.
Yikes, that's a pretty good critique, I'm afraid. Almost painful to read :)
I decided to go ahead and implement this first, because I've seen some
real-world file-sharing setups serving cygwin to clusters of development
workstations, and secondly, because the consensus on #gentoo-portage was that
this was a supported use-case.
Anyhow, seeing as how I already implemented it, it's too late for me to decide
that it's too much work :)
In my defense, I thought it would be way less involved and it just kinda
feature-creep'ed up on me. By the time I figured out what a monster I'd
created I was kinda pot-stuck.
-gmt
[gentoo-dev] cygwin prefix patch: request for eyeballs
Hello, I would appreciate if those of you with portage development experience and a moment to spare could please take a look at: https://github.com/gmt/gmt-cygwin-overlay/blob/master/sys-apps/portage/files/portage-2.2.01.20271-cygdll_protect.patch Not gunning for upstream or anything, but I would eventually like to start moving towards "not a complete pile of crap" status :) Notes: o based against a-few-weeks-old rsync prefix-portage o may contain a very few hunks that should arguably go upstream--I will make bugs for them and they are not what (I feel) needs eyeballs. o trying to actually run this may be difficult. let me know where you're stuck and I'll try to help. o I must admit, I pretty much learned python as I went, so... if you are thinking "that looks like a total rookie mistake" it probably is, and I'd probably appreciate your constructive critique. o I'll probably keep hacking on this so... moving target. But it's mostly working, I think. o This doesn't make any big changes to upstream code. So, If it looks like it might be a big indentation change, it probably is. o this is to solve the problem that, in cygwin, a running python crashes as soon as you change any filesystem .dll's on which it depends and fork() -- this is semi-orthogonal to the typical cygwin "rebasing" problem and distinct from it. The problem has nothing to do with locked files (except that my solution does use some standard portage file- locking features). -gmt
