Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

On Sat, 31 Mar 2018 09:39:47 + (UTC)
Farid BENAMROUCHE  wrote:

> interresting aproach.
> this could work. however, i can see a few limitations:
> - you must be root.

Actually you don't if you add -r to unshare, which gives you what is
sometimes called fakeroot. Obviously you still can't modify the files
if they are really owned by root but that's true of any solution.

> - this is specific to linux as of today.

True and I am only interested in Linux but I like to play nice. Other
platforms could potentially still briefly bind mount but it wouldn't be
isolated from the other processes so it wouldn't be entirely safe. Safe
enough though? You'd need to weigh this up against how many people use
ROOT!=/ on other platforms. Not many at all, I imagine.

> - if you want to hide the mechanism, i don't see how without doing
> the same portage modifications as in my solution.

You could handle this in the eclass functions but as you pointed out,
many things call chown/chgrp directly. Usage by ebuilds themselves can
be addressed but if a build system calls these then eclass functions
will not help. What would work is adding some identically-named
wrappers to the PATH.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer


pgpt1qN6u7M80.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[Farid BENAMROUCHE]

interresting aproach.
this could work. however, i can see a few limitations:
- you must be root.
- this is specific to linux as of today.
- if you want to hide the mechanism, i don't see how without doing the same 
portage modifications as in my solution.


but this is maybe worth investigating. my solution isn't perfect too, I admit.




En date de : Ven 30.3.18, James Le Cuirot <ch...@gentoo.org> a écrit :

 Objet: Re: [gentoo-dev] Re : Modification proposal for user/group creation 
when ROOT!="/"
 À: gentoo-dev@lists.gentoo.org
 Date: Vendredi 30 mars 2018, 21h56
 
 On Fri, 30 Mar 2018 20:47:20
 +0100
 James Le Cuirot <ch...@gentoo.org>
 wrote:
 
 > On Fri, 30 Mar
 2018 20:23:49 +0100
 > James Le Cuirot
 <ch...@gentoo.org>
 wrote:
 > 
 > > I did
 just have a lightbulb moment though. I've been playing
 with
 > > unshare recently and I
 wondered if we could leverage it here.
 >
 > 
 > >   $ sudo unshare -m /bin/sh
 -c "mount --bind /mnt/somewhere/etc /etc &&
 groupadd foo"
 > >   groupadd:
 Cannot determine your user name.  
 > 
 > Aha! I was trying to do this against an
 NFS share for a system with a
 > different
 architecture. If I use a local mount with a compatible
 > architecture, it actually does work.
 I'll explore this some more.
 
 Figured it out! The system I was doing this
 against has an ancient
 glibc (long story)
 with an old nsswitch.conf. I replaced this file with
 a newer one and it all started working. Do you
 agree this could be the
 way forwards?
 
 -- 
 James Le Cuirot (chewi)
 Gentoo Linux
 Developer

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

On Fri, 30 Mar 2018 20:47:20 +0100
James Le Cuirot  wrote:

> On Fri, 30 Mar 2018 20:23:49 +0100
> James Le Cuirot  wrote:
> 
> > I did just have a lightbulb moment though. I've been playing with
> > unshare recently and I wondered if we could leverage it here.
> > 
> >   $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc /etc && 
> > groupadd foo"
> >   groupadd: Cannot determine your user name.  
> 
> Aha! I was trying to do this against an NFS share for a system with a
> different architecture. If I use a local mount with a compatible
> architecture, it actually does work. I'll explore this some more.

Figured it out! The system I was doing this against has an ancient
glibc (long story) with an old nsswitch.conf. I replaced this file with
a newer one and it all started working. Do you agree this could be the
way forwards?

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer


pgpXQVm8fnITq.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

On Fri, 30 Mar 2018 20:23:49 +0100
James Le Cuirot  wrote:

> I did just have a lightbulb moment though. I've been playing with
> unshare recently and I wondered if we could leverage it here.
> 
>   $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc /etc && 
> groupadd foo"
>   groupadd: Cannot determine your user name.

Aha! I was trying to do this against an NFS share for a system with a
different architecture. If I use a local mount with a compatible
architecture, it actually does work. I'll explore this some more.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer


pgpi6lGElLSy7.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

On Fri, 30 Mar 2018 18:52:18 + (UTC)
Farid BENAMROUCHE  wrote:

> Yes, two years ago I've posted here to notify you about the creattion of 
> users and groups when using "ROOT=".
> As a reminder, if you currently emerge a package to a specific rootfs folder, 
> some packages will actually not create the user and groups correctly inside 
> this rootfs.
> It will also not check for the existance of the user/group inside of the 
> rootfs.
> Everytime, it will check "/".
> 
> This very old gentoo issue (I have to find again the GLEP talking about this 
> issue).
> 
> The solution is not possible without changing the behaviour of the tools used 
> by portage. For example, portage is using shadow in most systems (and shadow 
> is using the glibc).

Hi,

I have an interest this and was one of the early commenters in bug
#541406. I made my own suggestions about how this might work. Your
solution is cleaner in that it doesn't require modifying the users in
the / system but it does require significant changes to tools,
eclasses, and ebuilds so I'm on the fence about it.

I did just have a lightbulb moment though. I've been playing with
unshare recently and I wondered if we could leverage it here. First I
tried this.

  $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc/group 
/etc/group && groupadd foo"
  groupadd: failure while writing changes to /etc/group

It is possible to bind mount individual files but it doesn't work here
because it tries to rename /etc/group to made a backup. Next I tried
the whole directory but it gives a strange error.

  $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc /etc && 
groupadd foo"
  groupadd: Cannot determine your user name.

This reveals more.

  $ sudo unshare -m /bin/sh -c "id && mount --bind /mnt/utilite/mnt/moi/etc 
/etc && id"
  uid=0(root) gid=0(root) 
groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
  uid=0 gid=0 groups=0,1,2,3,4,6,10,11,20,26,27

I'm not sure why the IDs break like this and strace doesn't make it any
clearer. This seems like a route worth pursuing though because you
could create a bunch of wrappers for useradd, groupadd, chown and so on
and it would then all work transparently, even when not using the
eclass functions.

Regards,
-- 
James Le Cuirot (chewi)
Gentoo Linux Developer


pgp5TAk9z0pkh.pgp
Description: OpenPGP digital signature

[gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[Farid BENAMROUCHE]

Hi all,

Already two years
Yes, two years ago I've posted here to notify you about the creattion of users 
and groups when using "ROOT=".
As a reminder, if you currently emerge a package to a specific rootfs folder, 
some packages will actually not create the user and groups correctly inside 
this rootfs.
It will also not check for the existance of the user/group inside of the rootfs.
Everytime, it will check "/".

This very old gentoo issue (I have to find again the GLEP talking about this 
issue).

The solution is not possible without changing the behaviour of the tools used 
by portage. For example, portage is using shadow in most systems (and shadow is 
using the glibc).

I've submited a patch first to shadow upstream team to avec this feature, using 
a new argument (--prefix) to tools like usemod,useradd etc
You can create users in any rootfs you want, no need to be root (except if you 
local pam configuration requires it)

This is unfortunately not enough for gentoo: some packages are relying on 
chaonw directly. So I've added some wrappers (same as the current enewuser etc 
functions)
This enable people to build embedded systems rootfs from scratch, using a 
complete gentoo.
Some packages must be updated in order to behave properly in that case (replace 
chown with echown for example)
As a failsafe, I'm only triggering the new behavior in case ROOT env var is not 
"/" (and I think (I have to check) that if the command fails, it will fallback 
to the old behavior in case the --prefix argument is not implemented (incorrect 
shadow version for example)

This requires in that case a new EAPI I think. Just tell me what to do, where, 
and I will submit the proposal

I'm using the modifications for 2 years now.


Limitations: 
- selinux is not supported (ie: you will still use the system's selinux.). 
- Also, this only works with gentoo systems using the shadow package (the exact 
version will depend on when the new release will be done).
- Last one: I only tested on a linux configuration


Cheers



En date de : Mar 24.5.16, Farid BENAMROUCHE  a écrit :

 Objet: Modification proposal for user/group creation when ROOT!="/"
 À: gentoo-dev@lists.gentoo.org
 Date: Mardi 24 mai 2016, 21h00
 
 Hi all,
 
 Currently there is an old known
 limitation when using ROOT= option to install a package in a
 folder: user/groups are created in the host filesystem, not
 the target root filesystem.
 
 So I've pushed some modifications to
 the upstream shadow repo.
 Basically, I've added a --prefix option
 to user{add,mod,del} and group{add,mod,del}
 This option does the same as --root
 option, but whithout a chroot (so compatible when cross
 compiling)
 You can see more details (and the
 limitation of my implementation) in the shadow github repo:
 https://github.com/shadow-maint/shadow/issues/18
 
 
 Now, for the gentoo part, I do have a
 working solution that I've pushed in the following
 bugzilla:
 https://bugs.gentoo.org/show_bug.cgi?id=541406
 
 A new user.eclass file with modified
 enewuser,enewgroup and egetent that all supports ${ROOT}
 option via --prefix in shadow utilities.
 For now I've only added this option for
 linux.
 
 However, I've encountered some
 unexpected issues: some ebuilds are using direct calls to
 chown and fowners. Both are not compatible with ${ROOT}...
 
 To solve this, I've created 2 new calls
 in user.eclass: echown and efowners.
 The only thing the new functions are
 doing is to get the uid/gid from the correct passwd/group
 files from ${ROOT} using the modified egetent function and
 pass that to the native chown/fowners...
 
 For example, in sys-power/nut we can
 find:
 chown nut:nut ${ROOT}/var/lib/nut
 
 This should be changed to
 echown nut:nut ${ROOT}/var/lib/nut
 
 Same to fowners.
 If the modification is not done, either
 the ebuild will fail because the nut user does not exists in
 the host, or the incorrect uid will be user in ${ROOT}
 
 The solution is not perfect, but at
 least better than what we have today, and totally usable I
 believe.
 
 I've uploaded the patches for lighttpd
 and nut, plus my patch for user.eclass for review in this
 bug... we do have time until upstream shadow team reviews
 and commits my modifications (at least).
 
 Side note: it's a bit complicated to
 know when to add ${ROOT} and when not in a ebuild... For
 example, chown needs ${ROOT} but fowners must not!...
 Side note 2: maybe I should add a
 verification to check if useradd/groupadd supports my new
 --prefix solution, and fallback to original behavior if
 not?
 
 Tests: I've compiled a full working
 system with my above solution, and it works. (cross
 compilation in a dedicated target root path)
 
 Regards,
 Farid
 

Re : [gentoo-dev] Re: Modification proposal for user/group creation when ROOT!="/"

[Farid BENAMROUCHE]

Hi,

I only have a local overlay for now.
The shadow team still did'nt had time to merge my pull request on github.
But you can use my github branch:
https://github.com/fariouche/shadow.git

In the bugzilla you will find all the needed patches, except the shadow ebuild 
modified to use my branch.
Feel free to host the overlay if you want to test. It's pretty straightforward. 
:-)


I feel like an ebuild should never use directly the $ROOT prefix. I believe it 
is prone to errors and is better suited to be handled by the portage 
installation system.

Regards,
Farid


En date de : Mer 8.6.16, Benda Xu <hero...@gentoo.org> a écrit :

 Objet: [gentoo-dev] Re: Modification proposal for user/group creation when 
ROOT!="/"
 À: gentoo-dev@lists.gentoo.org
 Date: Mercredi 8 juin 2016, 6h44
 
 Hi Farid,
 
 This is an excellent idea! 
 It is very helpful for Gentoo Prefix/libc,
 where we maintain a set of nss databases
 (passwd, group, shadow, etc.)
 inside a
 directory prefix.
 
 Farid
 BENAMROUCHE <fariou...@yahoo.fr>
 writes:
 
 > Currently
 there is an old known limitation when using ROOT= option
 to
 > install a package in a folder:
 user/groups are created in the host
 >
 filesystem, not the target root filesystem.
 
 Exactly.
 
 > So I've pushed some modifications to
 the upstream shadow repo.
 > Basically,
 I've added a --prefix option to user{add,mod,del} and
 > group{add,mod,del} This option does the
 same as --root option, but
 > whithout a
 chroot (so compatible when cross compiling) 
 
 Cool.
 
 > You can see more details (and the
 limitation of my implementation) in
 > the
 shadow github repo:
 > https://github.com/shadow-maint/shadow/issues/18
 
 Hope the upstream accept your
 patch soon.
 
 > Now, for
 the gentoo part, I do have a working solution that
 I've
 > pushed in the following
 bugzilla:
 > https://bugs.gentoo.org/show_bug.cgi?id=541406
 >
 > A new user.eclass
 file with modified enewuser,enewgroup and egetent
 > that all supports ${ROOT} option via
 --prefix in shadow utilities.
 > For now
 I've only added this option for linux.
 
 The new user.eclass requires
 the new shadow.  After the upstream makes a
 new release, it will take a long time for
 Gentoo to use the feature.
 Because we should
 carefully handle the compatibility with the old
 systems.
 
 >
 However, I've encountered some unexpected issues: some
 ebuilds are
 > using direct calls to chown
 and fowners. Both are not compatible with
 > ${ROOT}...
 
 Those ebuilds are broken and should be
 fixed.
 
 > To solve this,
 I've created 2 new calls in user.eclass: echown and
 > efowners.  The only thing the new
 functions are doing is to get the
 >
 uid/gid from the correct passwd/group files from ${ROOT}
 using the
 > modified egetent function and
 pass that to the native chown/fowners...
 >
 > For example, in
 sys-power/nut we can find: chown nut:nut
 > ${ROOT}/var/lib/nut
 >
 > This should be
 changed to echown nut:nut ${ROOT}/var/lib/nut
 
 Brilliant.
 
 > Same to fowners.  If the modification is
 not done, either the ebuild
 > will fail
 because the nut user does not exists in the host, or the
 > incorrect uid will be user in ${ROOT}
 >
 > The solution is not
 perfect, but at least better than what we have
 > today, and totally usable I believe.
 >
 > I've uploaded the
 patches for lighttpd and nut, plus my patch for
 > user.eclass for review in this bug... we
 do have time until upstream
 > shadow team
 reviews and commits my modifications (at least).
 >
 > Side note: it's a
 bit complicated to know when to add ${ROOT} and when
 > not in a ebuild... For example, chown
 needs ${ROOT} but fowners must
 >
 not!...  
 
 Why not?  Could
 you give more details?
 
 >
 Side note 2: maybe I should add a verification to check
 if
 > useradd/groupadd supports my new
 --prefix solution, and fallback to
 >
 original behavior if not?
 
 IMHO, useradd/groupadd supporting --prefix will
 be captured by a new
 EAPI in the (far)
 future.  At present we don't need to worry about it.
 
 > Tests: I've compiled
 a full working system with my above solution, and
 > it works. (cross compilation in a
 dedicated target root path)
 
 Do you have an overlay for me reproduce your
 result?  I am also
 interested in hosting it
 in proj/android.git[1] if you do not have one
 yet.
 
 Keep on
 your good work.
 
 Yours,
 Benda   
 
 1.
 https://gitweb.gentoo.org/proj/android.git/
 

[gentoo-dev] Re: Modification proposal for user/group creation when ROOT!="/"

[Benda Xu]

Hi Farid,

This is an excellent idea!  It is very helpful for Gentoo Prefix/libc,
where we maintain a set of nss databases (passwd, group, shadow, etc.)
inside a directory prefix.

Farid BENAMROUCHE  writes:

> Currently there is an old known limitation when using ROOT= option to
> install a package in a folder: user/groups are created in the host
> filesystem, not the target root filesystem.

Exactly.

> So I've pushed some modifications to the upstream shadow repo.
> Basically, I've added a --prefix option to user{add,mod,del} and
> group{add,mod,del} This option does the same as --root option, but
> whithout a chroot (so compatible when cross compiling) 

Cool.

> You can see more details (and the limitation of my implementation) in
> the shadow github repo:
> https://github.com/shadow-maint/shadow/issues/18

Hope the upstream accept your patch soon.

> Now, for the gentoo part, I do have a working solution that I've
> pushed in the following bugzilla:
> https://bugs.gentoo.org/show_bug.cgi?id=541406
>
> A new user.eclass file with modified enewuser,enewgroup and egetent
> that all supports ${ROOT} option via --prefix in shadow utilities.
> For now I've only added this option for linux.

The new user.eclass requires the new shadow.  After the upstream makes a
new release, it will take a long time for Gentoo to use the feature.
Because we should carefully handle the compatibility with the old
systems.

> However, I've encountered some unexpected issues: some ebuilds are
> using direct calls to chown and fowners. Both are not compatible with
> ${ROOT}...

Those ebuilds are broken and should be fixed.

> To solve this, I've created 2 new calls in user.eclass: echown and
> efowners.  The only thing the new functions are doing is to get the
> uid/gid from the correct passwd/group files from ${ROOT} using the
> modified egetent function and pass that to the native chown/fowners...
>
> For example, in sys-power/nut we can find: chown nut:nut
> ${ROOT}/var/lib/nut
>
> This should be changed to echown nut:nut ${ROOT}/var/lib/nut

Brilliant.

> Same to fowners.  If the modification is not done, either the ebuild
> will fail because the nut user does not exists in the host, or the
> incorrect uid will be user in ${ROOT}
>
> The solution is not perfect, but at least better than what we have
> today, and totally usable I believe.
>
> I've uploaded the patches for lighttpd and nut, plus my patch for
> user.eclass for review in this bug... we do have time until upstream
> shadow team reviews and commits my modifications (at least).
>
> Side note: it's a bit complicated to know when to add ${ROOT} and when
> not in a ebuild... For example, chown needs ${ROOT} but fowners must
> not!...  

Why not?  Could you give more details?

> Side note 2: maybe I should add a verification to check if
> useradd/groupadd supports my new --prefix solution, and fallback to
> original behavior if not?

IMHO, useradd/groupadd supporting --prefix will be captured by a new
EAPI in the (far) future.  At present we don't need to worry about it.

> Tests: I've compiled a full working system with my above solution, and
> it works. (cross compilation in a dedicated target root path)

Do you have an overlay for me reproduce your result?  I am also
interested in hosting it in proj/android.git[1] if you do not have one
yet.

Keep on your good work.

Yours,
Benda   

1. https://gitweb.gentoo.org/proj/android.git/