Re: [gentoo-dev] RFC: BLAS and LAPACK runtime switching
Hi David, David Seifert writes: >> > An actual ABI compliance test, e.g. done using abi-compliance- >> > checker would be more interesting. >> >> As said above, the symbols don't need to be 1-1 copy of each other. >> Any library which is a superset of the reference one will work. > > Again, I'm willing to accept this under a USE="lapack_targets_virtual" > configuration, I hear you, and I was with the same opinion, too. Nevertheless, if a runtime switch works, it is simpler than USE_EXPAND. We already have a couple of them, PYTHON_TARGET, RUBY_TARGET, etc. I don't want to add more to this list unless absolutely necessary. Alternatively, it could be exclusive USE flags instead of USE_EXPAND. That's possible and we need to add a lot of USE flags, OpenBLAS, blis, reference, altas, nvBLAS, etc. I don't think it a clean solution. > but wholesale editing of DT_NEEDED entries is definitely too scary and > too invasive for most non-sci/hpc users of Gentoo. Sorry if I confused you. There is no hack of DT_NEEDED involved. An application is compiled against the reference, then it is pointed by LDPATH and ld.so.conf to a drop-in replacement library, and it profits. That's not more than upgrading a dynamic library. The scheme was shown to work with gcc runtime, libstdc++, and opengl, in Gentoo. > Again, for 99% of users, OpenBLAS will be the right trade-off between > performance and customizability. Every recompilation of libreoffice or > chromium will devour more CPU cycles than switching between USE-flag > implementations. I understand your point. Let me elaborate: 1. OpenBLAS has quite a bit of hand-tuned assembly, it is not very portable. Special care is need to manage the default BLAS on profiles, like ppc64, arm64, arm, riscv, etc. 2. All the Gentoo users care about optimization. And there are non-science applications that call linear algebra routines. 3. Maintaining different BLAS frameworks in repo/gentoo.git and proj/sci.git wastes everybody's time, and causes endless confusion to users. That offsets the time saved to make OpenBLAS the global default. Therefore, from the users' point of view, I still think runtime switching makes more sense. Yours, Benda
[gentoo-dev] Re: [PATCH-r1] darcs.eclass: use BDEPEND with EAPI >= 7
On Wed, 29 May 2019 12:01:20 +0200
Michael Haubenwallner wrote:
> ---
> eclass/darcs.eclass | 12 ++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/eclass/darcs.eclass b/eclass/darcs.eclass
> index 489008a87f1..09b71882367 100644
> --- a/eclass/darcs.eclass
> +++ b/eclass/darcs.eclass
> @@ -85,8 +85,16 @@ SRC_URI=""
>
> # --- end ebuild-configurable settings ---
>
> -DEPEND="dev-vcs/darcs
> - net-misc/rsync"
> +case ${EAPI:-0} in
> + [0-6]) # no need to care about 5-HDEPEND and similar
> + DEPEND="dev-vcs/darcs
> + net-misc/rsync"
> + ;;
> + *)
> + BDEPEND="dev-vcs/darcs
> + net-misc/rsync"
> + ;;
> +esac
>
> # @FUNCTION: darcs_patchcount
> # @DESCRIPTION:
> --
> 2.19.2
>
Looks good!
--
Sergei
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
On Wed, 2019-05-29 at 09:28 +0200, Michał Górny wrote: > Hi, > > Here's a reiteration (or well, rewrite) of mjo's earlier work [1]. I've > made it into more GLEP-ish form, removed inline code samples which > belong in implementation part and did some changes. > > The main change from the earlier proposal is that we are extremely > careful not to break stuff. Packages only create users/groups if they > don't exist, they do not modify existing accounts. Users and groups are > not removed at all. The text included below. > > > [1] https://wiki.gentoo.org/wiki/User:Mjo/GLEP:User_packages > FTR, I've placed my initial code in [1]. It's neither complete, nor final design. I have to address a number of old bugs for user.eclass first. [1] https://github.com/gentoo/gentoo/pull/12129 -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
On 5/29/19 3:28 AM, Michał Górny wrote: > > Home directory ownership > > > If the user in question uses a regular home directory (i.e. not > ``/dev/null``), the user package should maintain the directory > via ``keepdir`` command. This allows for clean removal of the home > directory if it is no longer needed. The package manager will also > apply correct permissions if the directory does not exist yet. > This made complete sense when removal of the user package implied removal of the system user. It's still not a big deal, but this could delete the empty home directory of a valid (but locked) user account. I don't feel strongly either way, just pointing it out.
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
On 5/29/19 5:50 AM, Jaco Kroon wrote: >> >> This GLEP follows the best practice of leaving obsolete user/groups >> accounts intact. This guarantees that no files with stale ownership are >> left (e.g. on unmounted filesystems) and that the same UID/GID is not >> reused for another user/group. > > The type of checks for both this and certain updates contemplated above > are similar. And expensive (resources) as you rightly say. I would > provide the tools to perform these checks but in the ebuild I'd rather > the checks be done asap (pkg_pretend?). If we can fail there and > stating what the admin should do to rectify the issue that would be the > best solution in my personal opinion. Ie, from the package manager I'd > state how, but not actually action these changes. > My original goal here was to have "emerge -C " actually uninstall the user. That turns out to be highly unwise, because you need to audit the system for things owned and used by said user, and to clean them all up beforehand. That can take forever, and can't be automated. Plan (b) was to do exactly as you asked: have the uninstall process bail out, and tell you what to do. That's... also technically infeasible, because we don't have a way to make an ebuild bail out of an uninstall. This could change in the future with an EAPI/PMS update, but simply isn't an option right now.
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
On 5/29/19 4:01 AM, Ulrich Mueller wrote: > > I wonder why that would be needed. It won't catch collisions with users > created by the system administrator. The reference implementation did its best not to annoy you here. Ultimately, no, it can't prevent the system administrator from clobbering a UID that some package needs. But, most packages won't insist on a fixed UID. If they do, and if the system administrator created a "compatible" user (one that doesn't conflict with any of your useradd arguments), then the existing user account would be reused. > Collisions between packages could > certainly be prevented without littering the file system, e.g., by > scanning ebuilds in that category (by repoman, or CI)? > Memory hazy, but I think I ventured down this road for a few reasons: * Using file collisions is a PMS behavior that will work everywhere. * Parsing bash variables in repoman/CI sucks and doesn't work. * We want to be warned about the collision during development, before we push to ::gentoo. * We should prevent collisions in overlays, too. * The UID that actually gets created may not be the one in your ebuild, if it wasn't available. The real UID is only known at install-time. (This one is important IIRC.)
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
On Wed, May 29, 2019 at 12:25:59PM +0200, Michał Górny wrote: > On Wed, 2019-05-29 at 11:50 +0200, Jaco Kroon wrote: > > Hi Michal, > > > > This sounds sensible and is an interesting approach. I kinda like it. > > > > There is only one technical comment I have based on the earlier > > discussion, not addressed. > > > > What if users needs to be created into a centralized UID/GID system to > > be pulled in via nss? > > > > So calling system tools by default is fine, but what if the sysadmin > > would prefer to have users and groups pushed into ldap? Can we at least > > accomodate a hook mechanism to allow system administrators not relying > > on local users to deal with this? > We kinda have hooks already. Just drop your 'useradd' etc. replacements > into /usr/local/bin, and tadaam! KISS all the way. Having written one of those replacements (diradm), I would like a little more flexibility: - permit the sysadmin to configure paths to the useradd(etc) tools/wrappers to be actually used. - include a manual mode that just has the package bail out and wait for the sysadmin to do it (e.g. they have to actually create the user on another host). > > My personal rule of thumb is that system users are (and should be) > > local. But there are definite use cases where shared "system uids" are > > a definite legitimate requirement. Created in a central system AND mirrored locally is my preference, using nsscache. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: BLAS and LAPACK runtime switching
On Wed, 2019-05-29 at 22:33 +0800, Benda Xu wrote:
> Hi Michał,
>
> Michał Górny writes:
>
> > On Tue, 2019-05-28 at 01:37 -0700, Mo Zhou wrote:
> > > Different BLAS/LAPACK implementations are expected to be
> > > compatible
> > > to each other in both the API and ABI level. They can be used as
> > > drop-in replacement to the others. This sounds nice, but the
> > > difference
> > > in SONAME hampered the gentoo integration of well-optimized ones.
> >
> > If SONAMEs are different, then they are not compatible by
> > definition.
>
> This blas/lapack SONAME difference is a special case. They are
> parially
> compatible in the sense that every alternative implementation of blas
> is
> a superset of the reference one.
>
> Therefore linking to the reference at build time will make sure the
> compatibility with the alternative implementations, even with
> different
> SONAME.
>
> > > [...]
> > >
> > > Similar to Debian's update-alternatives mechanism, Gentoo's
> > > eselect
> > > is good at dealing with drop-in replacements as well. My
> > > preliminary
> > > investigation suggests that eselect is enough for enabling
> > > BLAS/LAPACK
> > > runtime switching. Hence, the proposed solution is eselect-based:
> > >
> > > * Every BLAS/LAPACK implementation should provide generic
> > > library
> > > and eselect candidate libraries at the same time. Taking
> > > netlib,
> > > BLIS and OpenBLAS as examples:
> > >
> > > reference:
> > >
> > > usr/lib64/blas/reference/libblas.so.3 (SONAME=libblas.so.3)
> > > -- default BLAS provider
> > > -- candidate of the eselect "blas" unit
> > > -- will be symlinked to usr/lib64/libblas.so.3 by eselect
> >
> > /usr/lib64 is not supposed to be modified by eselect, it's package
> > manager area. Yes, I know a lot of modules still do that but
> > that's no
> > reason to make things worse when people are putting significant
> > effort
> > to actually improve things.
>
> Sorry, I didn't see your reply before mine.
>
> We are going to use the LDPATH and ld.so.conf mechanism suggested by
> you.
>
> > > usr/lib64/lapack/reference/liblapack.so.3
> > > (SONAME=liblapack.so.3)
> > > -- default LAPACK provider
> > > -- candidate of the eselect "lapack" unit
> > > -- will be symlinked to usr/lib64/liblapack.so.3 by
> > > eselect
> > >
> > > blis (doesn't provide LAPACK):
> > >
> > > usr/lib64/libblis.so.2 (SONAME=libblis.so.2)
> > > -- general purpose
> > >
> > > usr/lib64/blas/blis/libblas.so.3 (SONAME=libblas.so.3)
> > > -- candidate of the eselect "blas" unit
> > > -- will be symlinked to usr/lib64/libblas.so.3 by eselect
> > > -- compiled from the same set of object files as
> > > libblis.so.2
> > >
> > > openblas:
> > >
> > > usr/lib64/libopenblas.so.0 (SONAME=libopenblas.so.0)
> > > -- general purpose
> > >
> > > usr/lib64/blas/openblas/libblas.so.3 (SONAME=libblas.so.3)
> > > -- candidate of the eselect "blas" unit
> > > -- will be symlinked to usr/lib64/libblas.so.3 by eselect
> > > -- compiled from the same set of object files as
> > > libopenblas.so.0
> > >
> > > usr/lib64/lapack/openblas/liblapack.so.3
> > > (SONAME=liblapack.so.3)
> > > -- candidate of the eselect "lapack" unit
> > > -- will be symlinked to usr/lib64/liblapack.so.3 by
> > > eselect
> > > -- compiled from the same set of object files as
> > > libopenblas.so.0
> > >
> > > This solution is similar to Debian's[3]. This solution achieves
> > > our
> > > goal,
> > > and it requires us to patch upstream build systems (same to
> > > Debian).
> > > Preliminary demonstration for this solution is available, see
> > > below.
> >
> > So basically the three walls of text say in round-about way that
> > you're
> > going to introduce custom hacks to recompile libraries with
> > different
> > SONAME. Ok.
> > > Is this solution reliable?
> > > --
> > >
> > > * A similar solution has been used by Debian for many years.
> > > * Many projects call BLAS/LAPACK libraries through FFI, including
> > > Julia.
> > > (See Julia's standard library: LinearAlgebra)
> > >
> > > Proposed Changes
> > >
> > >
> > > 1. Deprecate sci-libs/{blas,cblas,lapack,lapacke}-reference from
> > > gentoo
> > >main repo. They use exactly the same source tarball. It's not
> > > quite
> > >helpful to package these components in a fine-grained manner.
> > > A
> > > single
> > >sci-libs/lapack package is enough.
> >
> > Where's the gain in that?
> > > 2. Merge the "cblas" eselect unit into "blas" unit. It is
> > > potentially
> > >harmful when "blas" and "cblas" point to different
> > > implementations.
> > >That means "app-eselect/eselect-cblas" should be deprecated.
> > >
> > > 3. Update virtual/{blas,cblas,lapack,lapacke}. BLAS/LAPACK
> > > providers
> > >
Re: [gentoo-dev] RFC: BLAS and LAPACK runtime switching
Hi Michał,
Michał Górny writes:
> On Tue, 2019-05-28 at 01:37 -0700, Mo Zhou wrote:
>> Different BLAS/LAPACK implementations are expected to be compatible
>> to each other in both the API and ABI level. They can be used as
>> drop-in replacement to the others. This sounds nice, but the difference
>> in SONAME hampered the gentoo integration of well-optimized ones.
>
> If SONAMEs are different, then they are not compatible by definition.
This blas/lapack SONAME difference is a special case. They are parially
compatible in the sense that every alternative implementation of blas is
a superset of the reference one.
Therefore linking to the reference at build time will make sure the
compatibility with the alternative implementations, even with different
SONAME.
>> [...]
>>
>> Similar to Debian's update-alternatives mechanism, Gentoo's eselect
>> is good at dealing with drop-in replacements as well. My preliminary
>> investigation suggests that eselect is enough for enabling BLAS/LAPACK
>> runtime switching. Hence, the proposed solution is eselect-based:
>>
>> * Every BLAS/LAPACK implementation should provide generic library
>> and eselect candidate libraries at the same time. Taking netlib,
>> BLIS and OpenBLAS as examples:
>>
>> reference:
>>
>> usr/lib64/blas/reference/libblas.so.3 (SONAME=libblas.so.3)
>> -- default BLAS provider
>> -- candidate of the eselect "blas" unit
>> -- will be symlinked to usr/lib64/libblas.so.3 by eselect
>
> /usr/lib64 is not supposed to be modified by eselect, it's package
> manager area. Yes, I know a lot of modules still do that but that's no
> reason to make things worse when people are putting significant effort
> to actually improve things.
Sorry, I didn't see your reply before mine.
We are going to use the LDPATH and ld.so.conf mechanism suggested by
you.
>> usr/lib64/lapack/reference/liblapack.so.3 (SONAME=liblapack.so.3)
>> -- default LAPACK provider
>> -- candidate of the eselect "lapack" unit
>> -- will be symlinked to usr/lib64/liblapack.so.3 by eselect
>>
>> blis (doesn't provide LAPACK):
>>
>> usr/lib64/libblis.so.2 (SONAME=libblis.so.2)
>> -- general purpose
>>
>> usr/lib64/blas/blis/libblas.so.3 (SONAME=libblas.so.3)
>> -- candidate of the eselect "blas" unit
>> -- will be symlinked to usr/lib64/libblas.so.3 by eselect
>> -- compiled from the same set of object files as libblis.so.2
>>
>> openblas:
>>
>> usr/lib64/libopenblas.so.0 (SONAME=libopenblas.so.0)
>> -- general purpose
>>
>> usr/lib64/blas/openblas/libblas.so.3 (SONAME=libblas.so.3)
>> -- candidate of the eselect "blas" unit
>> -- will be symlinked to usr/lib64/libblas.so.3 by eselect
>> -- compiled from the same set of object files as
>> libopenblas.so.0
>>
>> usr/lib64/lapack/openblas/liblapack.so.3 (SONAME=liblapack.so.3)
>> -- candidate of the eselect "lapack" unit
>> -- will be symlinked to usr/lib64/liblapack.so.3 by eselect
>> -- compiled from the same set of object files as
>> libopenblas.so.0
>>
>> This solution is similar to Debian's[3]. This solution achieves our
>> goal,
>> and it requires us to patch upstream build systems (same to Debian).
>> Preliminary demonstration for this solution is available, see below.
>
> So basically the three walls of text say in round-about way that you're
> going to introduce custom hacks to recompile libraries with different
> SONAME. Ok.
>> Is this solution reliable?
>> --
>>
>> * A similar solution has been used by Debian for many years.
>> * Many projects call BLAS/LAPACK libraries through FFI, including Julia.
>> (See Julia's standard library: LinearAlgebra)
>>
>> Proposed Changes
>>
>>
>> 1. Deprecate sci-libs/{blas,cblas,lapack,lapacke}-reference from gentoo
>>main repo. They use exactly the same source tarball. It's not quite
>>helpful to package these components in a fine-grained manner. A
>> single
>>sci-libs/lapack package is enough.
>
> Where's the gain in that?
>> 2. Merge the "cblas" eselect unit into "blas" unit. It is potentially
>>harmful when "blas" and "cblas" point to different implementations.
>>That means "app-eselect/eselect-cblas" should be deprecated.
>>
>> 3. Update virtual/{blas,cblas,lapack,lapacke}. BLAS/LAPACK providers
>>will be registered in their dependency information.
>>
>> Note, ebuilds for BLAS/LAPACK reverse dependencies are expected to work
>> with these changes correctly without change. For example, my local
>> numpy-1.16.1 compilation was successful without change.
>>
>> Preliminary Demonstration
>> -
>>
>> The preliminary implementation is available in my personal overlay[4].
>> A simple sanity test script `check-cpp.sh` is provided to illustrate
>> the effectiveness of the proposed
Re: [gentoo-dev] RFC: BLAS and LAPACK runtime switching
Dear David,
David Seifert writes:
> We already have such a solution in the sci-overlay. It has proven
> extremely brittle and shaky.
What's more, using eselect set which library to link to was regarded
harmful.
> The plan is to do this via USE flags similar to python-single-r1
> flags.
Yes, that was the conclusion in 2017.
But now there is something to be renewed, see below.
> Optionally, we can leave a "virtual" USE flag, where users can switch
> implementation at runtime, but this will not be a supported
> configuration.
[...]
> While I understand that runtime-switching sounds like a great feature,
> it has proven too difficult to get right and too hard to enforce
> invariants on correct symlinks.
I thought so, too. Back in 2017, I did an investigate of Debian runtime
switching, and concluded the patches were too heavy for us. That was the
main reason we did not consider the Debian runtime switching way.[0, 1]
However, I have made a deeper study this year, and concluded it is quite
doable in a simple way in Gentoo.
Historically, in alternative{,-2}.eclass of the science overlay,
"alternative" is a name from Debian[2]. We have made it too complex over
the years, and now there is a good chance to make it simple again.
> People that want this can go the virtual+eselect approach in the
> overlay,
Which is brittle and shaky as we agree.
> but 99% of Gentoo users will be happy with just linking against
> OpenBLAS/reference-lapack and not having to fix weird stale symlinks
> that eselect-alternatives somehow lost track of.
With the simplified approach Mo is after, there will no longer be weird
stale symlinks issues. The solution has been successfully proved by the
update-alternative mechanism in Debian. Actually, by adopting mgorny's
LDPATH way of handing gcc runtime, we don't need symlinks at all![3]
For more details, please refer to
https://people.debian.org/~lumin/gsoc19-gentoo.pdf
> See also:
> https://bugs.gentoo.org/632624
> https://bugs.gentoo.org/348843#c30
Thanks. That reminded me of all the discussions we all along.
Yours,
Benda
0. https://github.com/gentoo/sci/issues/805#issuecomment-345690034
1. https://github.com/gentoo/sci/issues/805#issuecomment-345701064
2. https://github.com/gentoo/sci/issues/805#issuecomment-332887691
3. https://bugs.gentoo.org/632618
Re: [gentoo-dev] RFC: BLAS and LAPACK runtime switching
On Tue, 2019-05-28 at 01:37 -0700, Mo Zhou wrote:
> Different BLAS/LAPACK implementations are expected to be compatible
> to each other in both the API and ABI level. They can be used as
> drop-in replacement to the others. This sounds nice, but the difference
> in SONAME hampered the gentoo integration of well-optimized ones.
If SONAMEs are different, then they are not compatible by definition.
> Assume a Gentoo user compiled a pile of packages on top of the reference
> BLAS and LAPACK, namely these reverse dependencies are linked against
> libblas.so.3 and liblapack.so.3 . When the user discovered that
> OpenBLAS provides much better performance, they'll have to recompile
> the whole reverse dependency tree in order to take advantage from
> OpenBLAS,
> because the SONAME of OpenBLAS is libopenblas.so.0 . When the user
> wants to try MKL (libmkl_rt.so), they'll have to recompile the whole
> reverse dependency tree again.
>
> This is not friendly to our earth.
>
> Goal
>
>
> * When a program is linked against libblas.so or liblapack.so
> provided by any BLAS/LAPACK provider, the eselect-based solution
> will allow user to switch the underlying library without recompiling
> anything.
>
> * When a program is linked against a specific implementation, e.g.
> libmkl_rt.so, the solution doesn't break anything.
>
> Solution
>
>
> Similar to Debian's update-alternatives mechanism, Gentoo's eselect
> is good at dealing with drop-in replacements as well. My preliminary
> investigation suggests that eselect is enough for enabling BLAS/LAPACK
> runtime switching. Hence, the proposed solution is eselect-based:
>
> * Every BLAS/LAPACK implementation should provide generic library
> and eselect candidate libraries at the same time. Taking netlib,
> BLIS and OpenBLAS as examples:
>
> reference:
>
> usr/lib64/blas/reference/libblas.so.3 (SONAME=libblas.so.3)
> -- default BLAS provider
> -- candidate of the eselect "blas" unit
> -- will be symlinked to usr/lib64/libblas.so.3 by eselect
/usr/lib64 is not supposed to be modified by eselect, it's package
manager area. Yes, I know a lot of modules still do that but that's no
reason to make things worse when people are putting significant effort
to actually improve things.
> usr/lib64/lapack/reference/liblapack.so.3 (SONAME=liblapack.so.3)
> -- default LAPACK provider
> -- candidate of the eselect "lapack" unit
> -- will be symlinked to usr/lib64/liblapack.so.3 by eselect
>
> blis (doesn't provide LAPACK):
>
> usr/lib64/libblis.so.2 (SONAME=libblis.so.2)
> -- general purpose
>
> usr/lib64/blas/blis/libblas.so.3 (SONAME=libblas.so.3)
> -- candidate of the eselect "blas" unit
> -- will be symlinked to usr/lib64/libblas.so.3 by eselect
> -- compiled from the same set of object files as libblis.so.2
>
> openblas:
>
> usr/lib64/libopenblas.so.0 (SONAME=libopenblas.so.0)
> -- general purpose
>
> usr/lib64/blas/openblas/libblas.so.3 (SONAME=libblas.so.3)
> -- candidate of the eselect "blas" unit
> -- will be symlinked to usr/lib64/libblas.so.3 by eselect
> -- compiled from the same set of object files as
> libopenblas.so.0
>
> usr/lib64/lapack/openblas/liblapack.so.3 (SONAME=liblapack.so.3)
> -- candidate of the eselect "lapack" unit
> -- will be symlinked to usr/lib64/liblapack.so.3 by eselect
> -- compiled from the same set of object files as
> libopenblas.so.0
>
> This solution is similar to Debian's[3]. This solution achieves our
> goal,
> and it requires us to patch upstream build systems (same to Debian).
> Preliminary demonstration for this solution is available, see below.
So basically the three walls of text say in round-about way that you're
going to introduce custom hacks to recompile libraries with different
SONAME. Ok.
>
> Is this solution reliable?
> --
>
> * A similar solution has been used by Debian for many years.
> * Many projects call BLAS/LAPACK libraries through FFI, including Julia.
> (See Julia's standard library: LinearAlgebra)
>
> Proposed Changes
>
>
> 1. Deprecate sci-libs/{blas,cblas,lapack,lapacke}-reference from gentoo
>main repo. They use exactly the same source tarball. It's not quite
>helpful to package these components in a fine-grained manner. A
> single
>sci-libs/lapack package is enough.
Where's the gain in that?
> 2. Merge the "cblas" eselect unit into "blas" unit. It is potentially
>harmful when "blas" and "cblas" point to different implementations.
>That means "app-eselect/eselect-cblas" should be deprecated.
>
> 3. Update virtual/{blas,cblas,lapack,lapacke}. BLAS/LAPACK providers
>will be registered in their dependency information.
>
> Note, ebuilds for BLAS/LAPACK reverse dependencies are expected
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
On Wed, 2019-05-29 at 11:50 +0200, Jaco Kroon wrote: > Hi Michal, > > This sounds sensible and is an interesting approach. I kinda like it. > > There is only one technical comment I have based on the earlier > discussion, not addressed. > > What if users needs to be created into a centralized UID/GID system to > be pulled in via nss? > > So calling system tools by default is fine, but what if the sysadmin > would prefer to have users and groups pushed into ldap? Can we at least > accomodate a hook mechanism to allow system administrators not relying > on local users to deal with this? We kinda have hooks already. Just drop your 'useradd' etc. replacements into /usr/local/bin, and tadaam! KISS all the way. > My personal rule of thumb is that system users are (and should be) > local. But there are definite use cases where shared "system uids" are > a definite legitimate requirement. > > The one complaint I do have is repo bloat. > > Further comments below. > > Kind Regards, > Jaco > > On 2019/05/29 09:28, Michał Górny wrote: > > Motivation > > == > > > > User management in Gentoo is currently ad-hoc. Users and groups are > > created through calling system tools directly in packages needing them. > > There is no systematic way of tracking which packages need specific > > users or groups, and determining which ones are obsolete. Coordinating > > properties of users and groups used by multiple packages must be done > > manually by developers. > > > > GLEP 27 originally attempted to address the problem. Posted in 2004, > > it never had reached the reference implementation state, and became > > obsolete. [#GLEP27]_ > > > > A good system user and group management proposal should address: > > > > 1. Tracking usage of users and groups, and determining which ones > > are obsolete. > > > > 2. Sharing users and groups reliably between different packages. > > > > 3. Maintaining fixed UIDs/GIDs that are consistent between different > > systems. > > > > 4. Providing local overrides for user/group properties. > > > > 5. Ensuring that users and groups are not created unnecessarily > > at build time. > > > > At the same time, the proposal should avoid unnecessary complexity > > to avoid sharing the fate of GLEP 27. This proposal aims to address > > those points without requiring a new EAPI or any changes in the package > > manager. > > This looks good. I'd add "6. Allow for centralized management of users > and/or groups" (in order to cover management via for example ldap). I thought this is implied by other points but I guess saying it explicitly doesn't hurt. > > > Maintaining users/groups > > > > > > The primary technical function of user and group packages is to create > > the users and groups. This is done via invoking the respective system > > tools at ``pkg_preinst`` phase. This is done only if the user/group > > does not exist on the system already. > > I would recommend that this be handled via an eclass that deals with the > technicalities. The package then does something like: This is the plan. There is already an old version in mjo's overlay, and I'm planning to put a new one today for review. > > inherit userpackage > > USER_NAME=mail > USER_UID=8 > USER_PRIMARY_GROUP=mail > USER_FORCE_UID=yes > > DEPEND=group/mail > RDEPEND=group/mail > > And that ends up being the whole package. Similar for groups. Even the > R?DEPEND can probably go into the eclass. > > The eclass can then potentially support external hooks or something to > accomodate 6. above. > > > User/group name/identifier collision detection > > -- > > > > The user/group packages can install additional files in subdirectories > > of ``/var/lib`` indicating their respective names and identifiers. > > This ensures that if two packages ever happen to collide, the package > > manager's collision detection mechanism will trigger. > > Ulrich also commented on this. How about enforcing a mapping from > $USER_NAME to package name? (ie, just replace all characters not > allowed in package naming with _ or -. > > Same as Ulrich I struggle to see any use-case where this would be an > issue, but artificial or not it's probably a good idea to deal with it > somehow. This is not so simple. The main concern are names like 'foo-1'. But I guess we can leave it out for now, and address it when we actually have a need for such user/group. > > > Naming rules > > > > > > It has been pointed out that the package naming rules are more > > restrictive than user/group naming rules. This is why the proposal > > allows for package names to be different from user/group names. > > However, this is strongly discouraged and should only be used when > > really necessary. > > I would personally disallow it and enforce a mapping from username to > package name. The question then however is if two names map
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
On Wed, 2019-05-29 at 10:01 +0200, Ulrich Mueller wrote: > > > > > > On Wed, 29 May 2019, Michał Górny wrote: > > User/group name/identifier collision detection > > -- > > The user/group packages can install additional files in subdirectories > > of ``/var/lib`` indicating their respective names and identifiers. > > This ensures that if two packages ever happen to collide, the package > > manager's collision detection mechanism will trigger. > > I wonder why that would be needed. It won't catch collisions with users > created by the system administrator. Collisions between packages could > certainly be prevented without littering the file system, e.g., by > scanning ebuilds in that category (by repoman, or CI)? To be honest, I had mixed feelings about this. I guess I can remove it, and expect that we'll CI it properly before we phase out the old method. > > http://creativecommons.org/licenses/by-sa/3.0/. > > s/http// > Sounds like vim templates need to be updated. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [PATCH] darcs.eclass: use BDEPEND with EAPI >= 7
On Wed, 2019-05-29 at 11:33 +0200, David Seifert wrote:
> On Wed, 2019-05-29 at 11:31 +0200, Ulrich Mueller wrote:
> > > > > > > On Wed, 29 May 2019, David Seifert wrote:
> > > On Wed, 2019-05-29 at 11:00 +0200, Michael Haubenwallner wrote:
> > > > +case ${EAPI:-0} in
> > > > + [0-6]*)
> > > Why the *? Do we really care about EAPI="5-HDEPEND" and others?
> >
> > Worse, this will match EAPI 10. :)
>
> To be honest, I already have nightmares about the EAPI specifier
> becoming a two-digit number, so many globs will fail...
>
We'll go for '9a' instead, and then '9za' when we run out of letters.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
[gentoo-dev] [PATCH-r1] darcs.eclass: use BDEPEND with EAPI >= 7
---
eclass/darcs.eclass | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/eclass/darcs.eclass b/eclass/darcs.eclass
index 489008a87f1..09b71882367 100644
--- a/eclass/darcs.eclass
+++ b/eclass/darcs.eclass
@@ -85,8 +85,16 @@ SRC_URI=""
# --- end ebuild-configurable settings ---
-DEPEND="dev-vcs/darcs
- net-misc/rsync"
+case ${EAPI:-0} in
+ [0-6]) # no need to care about 5-HDEPEND and similar
+ DEPEND="dev-vcs/darcs
+ net-misc/rsync"
+ ;;
+ *)
+ BDEPEND="dev-vcs/darcs
+ net-misc/rsync"
+ ;;
+esac
# @FUNCTION: darcs_patchcount
# @DESCRIPTION:
--
2.19.2
Re: [gentoo-dev] [pre-GLEP] User and group management via dedicated packages
Hi Michal, This sounds sensible and is an interesting approach. I kinda like it. There is only one technical comment I have based on the earlier discussion, not addressed. What if users needs to be created into a centralized UID/GID system to be pulled in via nss? So calling system tools by default is fine, but what if the sysadmin would prefer to have users and groups pushed into ldap? Can we at least accomodate a hook mechanism to allow system administrators not relying on local users to deal with this? My personal rule of thumb is that system users are (and should be) local. But there are definite use cases where shared "system uids" are a definite legitimate requirement. The one complaint I do have is repo bloat. Further comments below. Kind Regards, Jaco On 2019/05/29 09:28, Michał Górny wrote: Motivation == User management in Gentoo is currently ad-hoc. Users and groups are created through calling system tools directly in packages needing them. There is no systematic way of tracking which packages need specific users or groups, and determining which ones are obsolete. Coordinating properties of users and groups used by multiple packages must be done manually by developers. GLEP 27 originally attempted to address the problem. Posted in 2004, it never had reached the reference implementation state, and became obsolete. [#GLEP27]_ A good system user and group management proposal should address: 1. Tracking usage of users and groups, and determining which ones are obsolete. 2. Sharing users and groups reliably between different packages. 3. Maintaining fixed UIDs/GIDs that are consistent between different systems. 4. Providing local overrides for user/group properties. 5. Ensuring that users and groups are not created unnecessarily at build time. At the same time, the proposal should avoid unnecessary complexity to avoid sharing the fate of GLEP 27. This proposal aims to address those points without requiring a new EAPI or any changes in the package manager. This looks good. I'd add "6. Allow for centralized management of users and/or groups" (in order to cover management via for example ldap). Maintaining users/groups The primary technical function of user and group packages is to create the users and groups. This is done via invoking the respective system tools at ``pkg_preinst`` phase. This is done only if the user/group does not exist on the system already. I would recommend that this be handled via an eclass that deals with the technicalities. The package then does something like: inherit userpackage USER_NAME=mail USER_UID=8 USER_PRIMARY_GROUP=mail USER_FORCE_UID=yes DEPEND=group/mail RDEPEND=group/mail And that ends up being the whole package. Similar for groups. Even the R?DEPEND can probably go into the eclass. The eclass can then potentially support external hooks or something to accomodate 6. above. User/group name/identifier collision detection -- The user/group packages can install additional files in subdirectories of ``/var/lib`` indicating their respective names and identifiers. This ensures that if two packages ever happen to collide, the package manager's collision detection mechanism will trigger. Ulrich also commented on this. How about enforcing a mapping from $USER_NAME to package name? (ie, just replace all characters not allowed in package naming with _ or -. Same as Ulrich I struggle to see any use-case where this would be an issue, but artificial or not it's probably a good idea to deal with it somehow. Naming rules It has been pointed out that the package naming rules are more restrictive than user/group naming rules. This is why the proposal allows for package names to be different from user/group names. However, this is strongly discouraged and should only be used when really necessary. I would personally disallow it and enforce a mapping from username to package name. The question then however is if two names map to the same package (which would be confusing to begin with) how to deal with that. User/group updates -- If sysadmin needs to change the properties (e.g. home directory) of some user/group, the obvious course of action is to modify the system databases directly. The GLEP aims to respect that and disallow altering existing user/groups unconditionally. If any updates need to be done, the packages need to verify previous values first. "If any updates need to be done, the package need to verify previous values first." conflicts with "disallow altering existing user/groups unconditionally". I'd rephrase as "If any updates need to be done, refer those to the system administrator." User/group removal -- The original proposal attempted to remove user/groups automatically when the respective package was unmerged. This required verifying
Re: [gentoo-dev] [PATCH] darcs.eclass: use BDEPEND with EAPI >= 7
On Wed, 2019-05-29 at 11:31 +0200, Ulrich Mueller wrote:
> > > > > > On Wed, 29 May 2019, David Seifert wrote:
> > On Wed, 2019-05-29 at 11:00 +0200, Michael Haubenwallner wrote:
> > > +case ${EAPI:-0} in
> > > + [0-6]*)
> > Why the *? Do we really care about EAPI="5-HDEPEND" and others?
>
> Worse, this will match EAPI 10. :)
To be honest, I already have nightmares about the EAPI specifier
becoming a two-digit number, so many globs will fail...
Re: [gentoo-dev] [PATCH] darcs.eclass: use BDEPEND with EAPI >= 7
> On Wed, 29 May 2019, David Seifert wrote:
> On Wed, 2019-05-29 at 11:00 +0200, Michael Haubenwallner wrote:
>> +case ${EAPI:-0} in
>> +[0-6]*)
> Why the *? Do we really care about EAPI="5-HDEPEND" and others?
Worse, this will match EAPI 10. :)
signature.asc
Description: PGP signature
Re: [gentoo-dev] [PATCH] user.eclass: die if hard coded UID or GID is already in use
On 2019-05-27 16:45, William Hubbs wrote: > If a package hard codes the UID or GID when adding a user or group to > the system and that UID/GID already exists, we should abort rather than > changing the UID/GID. +1. It is of course my personal opinion but years of having been working with various scientific software pipelines have made me rabidly allergic to tools which do a different thing than what they were asked to do unless explicitly allowed to do it this way. And what is the point of specifying a "preferred" xID anyway? We cannot rely on these values anywhere so IMHO all they do is add noise. And no, in my book "it has always been like this" is not by itself a good reason to maintain the status quo. -- MS
Re: [gentoo-dev] [PATCH] darcs.eclass: use BDEPEND with EAPI >= 7
On Wed, 2019-05-29 at 11:00 +0200, Michael Haubenwallner wrote:
> ---
> eclass/darcs.eclass | 12 ++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/eclass/darcs.eclass b/eclass/darcs.eclass
> index 489008a87f1..81003651680 100644
> --- a/eclass/darcs.eclass
> +++ b/eclass/darcs.eclass
> @@ -85,8 +85,16 @@ SRC_URI=""
>
> # --- end ebuild-configurable settings ---
>
> -DEPEND="dev-vcs/darcs
> - net-misc/rsync"
> +case ${EAPI:-0} in
> + [0-6]*)
Why the *? Do we really care about EAPI="5-HDEPEND" and others?
> + DEPEND="dev-vcs/darcs
> + net-misc/rsync"
> + ;;
> + *)
> + BDEPEND="dev-vcs/darcs
> + net-misc/rsync"
> + ;;
> +esac
>
> # @FUNCTION: darcs_patchcount
> # @DESCRIPTION:
Might be worth thinking about adding an EAPI whitelist like we do for
other eclasses. Just an idea, not required.
[gentoo-dev] [PATCH] darcs.eclass: use BDEPEND with EAPI >= 7
---
eclass/darcs.eclass | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/eclass/darcs.eclass b/eclass/darcs.eclass
index 489008a87f1..81003651680 100644
--- a/eclass/darcs.eclass
+++ b/eclass/darcs.eclass
@@ -85,8 +85,16 @@ SRC_URI=""
# --- end ebuild-configurable settings ---
-DEPEND="dev-vcs/darcs
- net-misc/rsync"
+case ${EAPI:-0} in
+ [0-6]*)
+ DEPEND="dev-vcs/darcs
+ net-misc/rsync"
+ ;;
+ *)
+ BDEPEND="dev-vcs/darcs
+ net-misc/rsync"
+ ;;
+esac
# @FUNCTION: darcs_patchcount
# @DESCRIPTION:
--
2.19.2
[gentoo-dev] [pre-GLEP] User and group management via dedicated packages
Hi, Here's a reiteration (or well, rewrite) of mjo's earlier work [1]. I've made it into more GLEP-ish form, removed inline code samples which belong in implementation part and did some changes. The main change from the earlier proposal is that we are extremely careful not to break stuff. Packages only create users/groups if they don't exist, they do not modify existing accounts. Users and groups are not removed at all. The text included below. [1] https://wiki.gentoo.org/wiki/User:Mjo/GLEP:User_packages --- GLEP: Title: User and group management via dedicated packages Author: Michał Górny , Michael Orlitzky Type: Standards Track Status: Draft Version: 1 Created: 2019-05-29 Last-Modified: 2019-05-29 Post-History: Content-Type: text/x-rst Requires: Replaces: 27 --- Abstract A new approach for user/group management is proposed. Regular packages in dedicated categories are used to represent and create user and group accounts. Dependencies are used to request users and group from within regular packages, and to track their usage. Motivation == User management in Gentoo is currently ad-hoc. Users and groups are created through calling system tools directly in packages needing them. There is no systematic way of tracking which packages need specific users or groups, and determining which ones are obsolete. Coordinating properties of users and groups used by multiple packages must be done manually by developers. GLEP 27 originally attempted to address the problem. Posted in 2004, it never had reached the reference implementation state, and became obsolete. [#GLEP27]_ A good system user and group management proposal should address: 1. Tracking usage of users and groups, and determining which ones are obsolete. 2. Sharing users and groups reliably between different packages. 3. Maintaining fixed UIDs/GIDs that are consistent between different systems. 4. Providing local overrides for user/group properties. 5. Ensuring that users and groups are not created unnecessarily at build time. At the same time, the proposal should avoid unnecessary complexity to avoid sharing the fate of GLEP 27. This proposal aims to address those points without requiring a new EAPI or any changes in the package manager. Specification = Logical structure - In this proposal, system users and groups are represented by regular packages. Those packages logically represent the ownership of the respective users and group, and technically implement their creation. User packages are placed in ``user`` category. Each user package defines the properties of the particular user, and should be named after the user it creates. It must depend at build and run time on the groups user belongs to. Group packages are placed in ``group`` category. Each group package defines the properties of the particular group, and should be named after the group it creates. All user and group packages must define preferred fixed UIDs/GIDs, and they must be unique within the repository. The packages should indicate whether the value needs to be strictly enforced, or whether another UID/GID can be used when the user exists already or requested UID/GID is taken. Packages needing a specific user or group use dependencies to pull the required user/group packages. If the user is needed at build time, a build time dependency (``DEPEND``) must be used. If the user is needed at install time, a run time dependency (``RDEPEND``) must be used. If the user is only needed after the package is installed, ``PDEPEND`` must be used. Maintaining users/groups The primary technical function of user and group packages is to create the users and groups. This is done via invoking the respective system tools at ``pkg_preinst`` phase. This is done only if the user/group does not exist on the system already. Normally, the packages should not modify existing users, to preserve local modifications. If such a modification is necessary, the package should verify the existing values of changed properties, and update them only if they match the previous value. The package must not remove users/groups. Any cleanup actions must be done with explicit user approval, and therefore should be addressed by separate tooling. Home directory ownership If the user in question uses a regular home directory (i.e. not ``/dev/null``), the user package should maintain the directory via ``keepdir`` command. This allows for clean removal of the home directory if it is no longer needed. The package manager will also apply correct permissions if the directory does not exist yet. Note that since the user is not created until ``pkg_preinst``, the permissions to home directory should not be applied earlier than that. User/group name/identifier collision detection -- The user/group packages can install
