Re: [systemd-devel] systemd-nspawn with filesystem id mapping

[Lennart Poettering]

On Fr, 04.06.21 14:53, systemd-de...@notandy.de (systemd-de...@notandy.de) 
wrote:

> Hi again,
>
> after some more debugging this EOVERFLOW seems to be the result of a call to 
> may_o_create in fs/namei.c in the kernel.
> There is a check:
>
> if (!fsuidgid_has_mapping(dir->dentry->d_sb, mnt_userns))
>   return -EOVERFLOW;
>
> This seems to be the one returning EOVERFLOW to nspawn and resulting in the 
> container spawn to fail.
> My guess would be that this is a systemd bug when combining filesystem id 
> mapping with --bind.
> Before I start spending more time debugging this, has anyone so far used 
> --bind with --private-users=pick and --private-users-ownership=map 
> successfull?
>
> As far as I understand the pull request #19438 , didn't add any handling to 
> the mount_bind function. Was this maybe overlooked?
> In my understanding there is a remount_idmap missing in that function well as 
> the touch needs to be done in the correct user namespace or with mapped 
> uid/gids.
>
> I'm new to the systemd source code, could somebody confirm that I'm on the 
> right track there and not heading in the wrong direction?

Let's follow up on the PR, it's the better place to development
discussions on specific bugs or problems. I replied on it the other
day.


Lennart

--
Lennart Poettering, Berlin
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04

[Aravindhan Krishnan]

Hi Reindl,

I have attached a minimalistic repro along with the codes of all the
scripts, service files. I suppose Silvio was able to see the files. Please
refer to
https://www.mail-archive.com/systemd-devel@lists.freedesktop.org/msg46055.html
.

I have re-attached the tar again in this email.


Regards,
Aravindhan Krishnan...


On Mon, 7 Jun 2021 at 21:53, Reindl Harald  wrote:

>
>
> Am 07.06.21 um 17:57 schrieb Aravindhan Krishnan:
> > Adding Raghav.
> >
> > And sorry the subject should have stated: Discrepancy in using dhclient
> > b/w ubuntu 20.04 and ubuntu 16.04
>
> and why didn't you fix it in your own reply?
>
> to your problem:
> you have a wild mix of docker, systemd-units and shellscripts but don't
> provide the source of the scripts nor the systemd unit
>
> overly complex for something that can be trivial as:
>
> [root@srv-rhsoft:~]$ cat /etc/systemd/system/network-wan-dhcp.service
> [Unit]
> Description=Internet DHCP-Client
>
> [Service]
> Type=forking
> ExecStart=/usr/sbin/dhclient -4 -q --no-pid --request-options
> subnet-mask,broadcast-address,routers br-wan
> PermissionsStartOnly=yes
> SuccessExitStatus=80
> Restart=always
> RestartSec=5
> ProtectSystem=strict
> ProtectHome=yes
> ReadWritePaths=-/var/lib/dhclient
> PrivateTmp=yes
> NoNewPrivileges=yes
> ProtectKernelTunables=yes
> ProtectKernelModules=yes
> ProtectControlGroups=yes
> MemoryDenyWriteExecute=yes
> CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
> CAP_NET_BROADCAST CAP_NET_RAW
> LockPersonality=yes
> PrivateDevices=yes
> ProtectHostname=yes
> RestrictNamespaces=yes
> RestrictRealtime=yes
> RestrictSUIDSGID=yes
> ProtectClock=true
> ProtectKernelLogs=true
> UMask=077
> SystemCallArchitectures=native
> SystemCallFilter=@system-service @network-io @privileged
> SystemCallFilter=~@aio @chown @clock @cpu-emulation @debug @keyring
> @module @mount @obsolete @raw-io @reboot @resources @swap
> InaccessiblePaths=-/boot
> InaccessiblePaths=-/efi
> InaccessiblePaths=-/root
>
> > On Mon, 7 Jun 2021 at 21:26, Aravindhan Krishnan
> > mailto:aravindhan...@gmail.com>> wrote:
> >
> > Hi Folks,
> >
> > I am finding anomalous behavior when I am trying to run dhclient
> > process inside my docker container in vanilla Ubuntu 16.04 host. The
> > service gets into "deactivating" state and is stuck forever. In the
> > mail I have attached a minimalistic reproduction of the issue seen.
> >
> > Working logic:
> >
> >   * There is a sample trial@.service script which invokes the
> > `trial` binary with the option passed to the systemd service via
> > @ option
> >   * The valid options are sleep and dhclient_
> >   * The binary either invokes a long-lived sleep process or dhclient
> > process on the said interface_name based on the input
> >   * The binary then spawns `kill_trial.sh` script. The script sleeps
> > for 20 seconds and kills the parent `trial` binary. The kill
> > signal is SIGKILL in the trial example. In the real-world, this
> > can be a SIGSEGV indicating a crash in the parent process.
> >   * If the trial binary was started for sleep process things work
> > fine and service goes into "failed" state as expected
> >   * However, in case of dhclient, the service is stuck in
> > "deactivating" state if the underlying host OS is Ubuntu 16.04.
> > This works well if the host is running Ubuntu 20.04.
> >   * We have kept TimeoutStopSec to infinity, because in real-word
> > deployments, the core collection post a crash takes varying time
> > depending on the memory config on the host.
> >
> >
> > Steps to reproduce
> > # tar -xf minimal_repro.tar -C minimal_repro/
> > # cd minimal_repro/
> > # docker build -t trial .
> > # docker rm -f trial
> > # docker run -it -d --net=host --privileged -v
> > /sys/fs/cgroup:/sys/fs/cgroup:ro --name trial trial
> > # docker exec -it trial bash
> >
> > # systemctl start trial@dhclient_eth1.service
> >
> > # #Keep monitoring trial@dhclient_eth1.service -- issue should be
> > seen within 20-30 seconds on Ubuntu 16.04 host
> >
> > # systemctl status trial@dhclient_eth1.service
> > ● trial@dhclient_eth1.service - Trial
> >   Loaded: loaded (/etc/systemd/system/trial@.service; static;
> > vendor preset: enabled)
> >   Active: deactivating (stop-sigterm) (Result: signal) since Mon
> > 2021-06-07 13:19:12 UTC; 1min 11s ago
> >  Process: 55 ExecStartPre=/bin/bash
> > /etc/systemd/system/trial_service_script.sh pre_start dhclient_eth1
> > (code=exited, status=0/SUCCESS)
> >  Process: 56 ExecStart=/bin/bash
> > /etc/systemd/system/trial_service_script.sh start dhclient_eth1
> > (code=killed, signal=KILL)
> > Main PID: 56 (code=killed, signal=KILL)
> >Tasks: 0 (limit: 38590)
> >   Memory: 588.0K
> >   CGroup:
> >
>  
> /docker/903fc

Re: [systemd-devel] rg...@outlook.com

[Aravindhan Krishnan]

Hi Lennart,

As I understand from
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.9_release_notes/index,
looks like centos 7.9 was released in Aug, 2020. Is this also considered to
be stale ?


Regards,
Aravindhan Krishnan...


On Tue, 8 Jun 2021 at 01:45, Lennart Poettering 
wrote:

> On Mo, 07.06.21 22:47, Aravindhan Krishnan (aravindhan...@gmail.com)
> wrote:
>
> > Hi Lennart,
> >
> > Thanks for the quick response. Yes, we are running systemd inside the
> > docker. We were also able to see the same issue even on top of
> > Centos 7.9.
>
> Unlike pretty much all other container managers Docker doesn't really
> make it easy to run systemd inside it. Docker upstream is pretty
> hostile towards systemd, so this is unlikely to change.
>
> We document pretty extensively what container managers have to do to
> make sure systemd just works inside containers. Pretty much all
> container managers just implement that, but Docker doesn't. This is
> what they need to implement:
>
> https://systemd.io/CONTAINER_INTERFACE
>
> Consider switching to a different container manager implementation,
> there are plenty others. (in particular podman is mostly a drop-in
> replacement for Docker, if you need Docker semantics. Podman upstream
> isn't hostile towards systemd, so things mostly just work there.)
>
> > Attaching the kernel and OS details of the centos host
> >
> > # uname -r
> > 3.10.0-1160.25.1.el7.x86_64
> >
> > # cat /etc/centos-release
> > CentOS Linux release 7.9.2009 (Core)
>
> This is very old. You might want to switch to a newer OS for this
> anyway.
>
> Lennart
>
> --
> Lennart Poettering, Berlin
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04

[Reindl Harald]

Am 08.06.21 um 14:50 schrieb Aravindhan Krishnan:

Hi Reindl,

I have attached a minimalistic repro along with the codes of all the 
scripts, service files. I suppose Silvio was able to see the files. 


i don't get the bash-nonsense for a handful of lines (most of them doing 
nothing at all) to begin with and given that there is no "Type=" in the 
unit file you may read the docs and try the different types


i also don't get the trial-binary

why in the world don't you trhow away all that crap inlcuding the docker 
container and start dhclient at your own from a trivial systemd-unit?


it's impressive how many layers and helpers one can wrap around simple 
tasks but to gain what except troubles?


keep it simple!

On Mon, 7 Jun 2021 at 21:53, Reindl Harald > wrote:




Am 07.06.21 um 17:57 schrieb Aravindhan Krishnan:
 > Adding Raghav.
 >
 > And sorry the subject should have stated: Discrepancy in using
dhclient
 > b/w ubuntu 20.04 and ubuntu 16.04

and why didn't you fix it in your own reply?

to your problem:
you have a wild mix of docker, systemd-units and shellscripts but don't
provide the source of the scripts nor the systemd unit

overly complex for something that can be trivial as:

[root@srv-rhsoft:~]$ cat /etc/systemd/system/network-wan-dhcp.service
[Unit]
Description=Internet DHCP-Client

[Service]
Type=forking
ExecStart=/usr/sbin/dhclient -4 -q --no-pid --request-options
subnet-mask,broadcast-address,routers br-wan
PermissionsStartOnly=yes
SuccessExitStatus=80
Restart=always
RestartSec=5
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=-/var/lib/dhclient
PrivateTmp=yes
NoNewPrivileges=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
MemoryDenyWriteExecute=yes
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
CAP_NET_BROADCAST CAP_NET_RAW
LockPersonality=yes
PrivateDevices=yes
ProtectHostname=yes
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectClock=true
ProtectKernelLogs=true
UMask=077
SystemCallArchitectures=native
SystemCallFilter=@system-service @network-io @privileged
SystemCallFilter=~@aio @chown @clock @cpu-emulation @debug @keyring
@module @mount @obsolete @raw-io @reboot @resources @swap
InaccessiblePaths=-/boot
InaccessiblePaths=-/efi
InaccessiblePaths=-/root

 > On Mon, 7 Jun 2021 at 21:26, Aravindhan Krishnan
 > mailto:aravindhan...@gmail.com>
>>
wrote:
 >
 >     Hi Folks,
 >
 >     I am finding anomalous behavior when I am trying to run dhclient
 >     process inside my docker container in vanilla Ubuntu 16.04
host. The
 >     service gets into "deactivating" state and is stuck forever.
In the
 >     mail I have attached a minimalistic reproduction of the issue
seen.
 >
 >     Working logic:
 >
 >       * There is a sample trial@.service script which invokes the
 >         `trial` binary with the option passed to the systemd
service via
 >         @ option
 >       * The valid options are sleep and dhclient_
 >       * The binary either invokes a long-lived sleep process or
dhclient
 >         process on the said interface_name based on the input
 >       * The binary then spawns `kill_trial.sh` script. The script
sleeps
 >         for 20 seconds and kills the parent `trial` binary. The kill
 >         signal is SIGKILL in the trial example. In the
real-world, this
 >         can be a SIGSEGV indicating a crash in the parent process.
 >       * If the trial binary was started for sleep process things work
 >         fine and service goes into "failed" state as expected
 >       * However, in case of dhclient, the service is stuck in
 >         "deactivating" state if the underlying host OS is Ubuntu
16.04.
 >         This works well if the host is running Ubuntu 20.04.
 >       * We have kept TimeoutStopSec to infinity, because in real-word
 >         deployments, the core collection post a crash takes
varying time
 >         depending on the memory config on the host.
 >
 >
 >     Steps to reproduce
 >     # tar -xf minimal_repro.tar -C minimal_repro/
 >     # cd minimal_repro/
 >     # docker build -t trial .
 >     # docker rm -f trial
 >     # docker run -it -d --net=host --privileged -v
 >     /sys/fs/cgroup:/sys/fs/cgroup:ro --name trial trial
 >     # docker exec -it trial bash
 >
 >     # systemctl start trial@dhclient_eth1.service
 >
 >     # #Keep monitoring trial@dhclient_eth1.service -- issue should be
 >     seen within 20-30 seconds on Ubuntu 16.04 host
 >
 >     # systemctl status trial@d

Re: [systemd-devel] Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04

[Aravindhan Krishnan]

Hi Reindl,

With all due respect, it would be very helpful if you can be a little bit
less snarky.

i don't get the bash-nonsense for a handful of lines (most of them doing
nothing at all) to begin with and given that there is no "Type=" in the
unit file you may read the docs and try the different types
>> As per the man page (
https://man7.org/linux/man-pages/man5/systemd.service.5.html), the default
Type is simple if ExecStart is specified.

i also don't get the trial-binary
why in the world don't you trhow away all that crap inlcuding the docker
container and start dhclient at your own from a trivial systemd-unit?
>> As the name indicates it is a trial or minimalistic reproduction of the
issue we are seeing. Actual issue: We have a binary, which starts, and stop
dhclient on the interface on demand (Please don't come back complaining why
you would need to start and stop dhclient on demand). In case the binary
crashes for some unforeseen reason (which I had also mentioned in my
initial mail. Requoting here:  In the real-world, this can be a SIGSEGV
indicating a crash in the parent process.), we are seeing that the service
is stuck in a limbo deactivating state forever. This is seen only in Ubuntu
16.04 and all Centos 7.x version and is not seen in Ubuntu 20.04.

why in the world don't you trhow away all that crap inlcuding the docker
container and start dhclient at your own from a trivial systemd-unit?
>> Again, in a real-world scenario we support upto 1000+ vLANs. Running
1000 different services for each of the dhclient could be too costly was
our initial assessment and thus we ran "dhclient  -nw" from the
parent process. If you feel systemd can handle such higher loads, without
causing a high perf impact, it would be helpful as well.

Regards,
Aravindhan Krishnan...


On Tue, 8 Jun 2021 at 18:35, Reindl Harald  wrote:

>
>
> Am 08.06.21 um 14:50 schrieb Aravindhan Krishnan:
> > Hi Reindl,
> >
> > I have attached a minimalistic repro along with the codes of all the
> > scripts, service files. I suppose Silvio was able to see the files.
>
> i don't get the bash-nonsense for a handful of lines (most of them doing
> nothing at all) to begin with and given that there is no "Type=" in the
> unit file you may read the docs and try the different types
>
> i also don't get the trial-binary
>
> why in the world don't you trhow away all that crap inlcuding the docker
> container and start dhclient at your own from a trivial systemd-unit?
>
> it's impressive how many layers and helpers one can wrap around simple
> tasks but to gain what except troubles?
>
> keep it simple!
>
> > On Mon, 7 Jun 2021 at 21:53, Reindl Harald  > > wrote:
> >
> >
> >
> > Am 07.06.21 um 17:57 schrieb Aravindhan Krishnan:
> >  > Adding Raghav.
> >  >
> >  > And sorry the subject should have stated: Discrepancy in using
> > dhclient
> >  > b/w ubuntu 20.04 and ubuntu 16.04
> >
> > and why didn't you fix it in your own reply?
> >
> > to your problem:
> > you have a wild mix of docker, systemd-units and shellscripts but
> don't
> > provide the source of the scripts nor the systemd unit
> >
> > overly complex for something that can be trivial as:
> >
> > [root@srv-rhsoft:~]$ cat
> /etc/systemd/system/network-wan-dhcp.service
> > [Unit]
> > Description=Internet DHCP-Client
> >
> > [Service]
> > Type=forking
> > ExecStart=/usr/sbin/dhclient -4 -q --no-pid --request-options
> > subnet-mask,broadcast-address,routers br-wan
> > PermissionsStartOnly=yes
> > SuccessExitStatus=80
> > Restart=always
> > RestartSec=5
> > ProtectSystem=strict
> > ProtectHome=yes
> > ReadWritePaths=-/var/lib/dhclient
> > PrivateTmp=yes
> > NoNewPrivileges=yes
> > ProtectKernelTunables=yes
> > ProtectKernelModules=yes
> > ProtectControlGroups=yes
> > MemoryDenyWriteExecute=yes
> > CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
> > CAP_NET_BROADCAST CAP_NET_RAW
> > LockPersonality=yes
> > PrivateDevices=yes
> > ProtectHostname=yes
> > RestrictNamespaces=yes
> > RestrictRealtime=yes
> > RestrictSUIDSGID=yes
> > ProtectClock=true
> > ProtectKernelLogs=true
> > UMask=077
> > SystemCallArchitectures=native
> > SystemCallFilter=@system-service @network-io @privileged
> > SystemCallFilter=~@aio @chown @clock @cpu-emulation @debug @keyring
> > @module @mount @obsolete @raw-io @reboot @resources @swap
> > InaccessiblePaths=-/boot
> > InaccessiblePaths=-/efi
> > InaccessiblePaths=-/root
> >
> >  > On Mon, 7 Jun 2021 at 21:26, Aravindhan Krishnan
> >  > mailto:aravindhan...@gmail.com>
> > >>
> > wrote:
> >  >
> >  > Hi Folks,
> >  >
> >  > I am finding anomalous behavior when I am trying to run
> dhclient
> >  > process inside my docker cont

Re: [systemd-devel] Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04

[Lennart Poettering]

On Di, 08.06.21 15:05, Reindl Harald (h.rei...@thelounge.net) wrote:

> > I have attached a minimalistic repro along with the codes of all the
> > scripts, service files. I suppose Silvio was able to see the files.
>
> i don't get the bash-nonsense for a handful of lines (most of them doing
> nothing at all) to begin with and given that there is no "Type=" in the unit
> file you may read the docs and try the different types
>
> i also don't get the trial-binary
>
> why in the world don't you trhow away all that crap inlcuding the docker
> container and start dhclient at your own from a trivial systemd-unit?

Reindl, I warned you very explicitly not to behave like this:

https://lists.freedesktop.org/archives/systemd-devel/2021-February/046028.html

You ignored that now. You are now blocked on this mailing list.

Lennart

--
Lennart Poettering, Berlin
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04

[Reindl Harald]

Am 08.06.21 um 15:44 schrieb Aravindhan Krishnan:
With all due respect, it would be very helpful if you can be a little 
bit less snarky.


With all due respect, it would be very helpful not use reply-all and not 
convert each and every response to HTML



i don't get the bash-nonsense for a handful of lines (most of them doing
nothing at all) to begin with and given that there is no "Type=" in the
unit file you may read the docs and try the different types
 >> As per the man page 
(https://man7.org/linux/man-pages/man5/systemd.service.5.html 
), the 
default Type is simple if ExecStart is specified.


which means the prcoess is supposed to stay and not fork


i also don't get the trial-binary
why in the world don't you trhow away all that crap inlcuding the docker
container and start dhclient at your own from a trivial systemd-unit?
 >> As the name indicates it is a trial or minimalistic reproduction of 
the issue we are seeing. Actual issue: We have a binary, which starts, 
and stop dhclient on the interface on demand (Please don't come back 
complaining why you would need to start and stop dhclient on demand). In 
case the binary crashes for some unforeseen reason (which I had also 
mentioned in my initial mail. 


this is handeled by systemd pretty fine - but not with an intermediate 
script and/or binary between



why in the world don't you trhow away all that crap inlcuding the docker
container and start dhclient at your own from a trivial systemd-unit?
 >> Again, in a real-world scenario we support upto 1000+ vLANs. Running 
1000 different services for each of the dhclient could be too costly was 
our initial assessment and thus we ran "dhclient  -nw" from 
the parent process. If you feel systemd can handle such higher loads, 
without causing a high perf impact, it would be helpful as well


with your template service you have 1000 systemd services anyways but 
throw additional load on the machine with your monitoring binary which 
is monitored by systemd anyways


you gain nothing with all that wrappers
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] socket activation SELinuxContextFromNet issue

[Ted Toth]

I'm working on a proxy to encrypt rsync network communications using
systemd socket activation (Accept=yes, SELinuxContextFromNet=true) so
that the proxy is run at the level of the connection (the system is
running selinux mls policy). rsync has the same systemd socket
activation configuration as I want it to also run at the level of the
connection. When the proxy is activated it connects to 873 (rsync) and
systemd logs an error:
Jun  7 18:19:25 comms systemd: Started fast remote file copy program
daemon (127.0.0.1:53456).
Jun  7 18:19:25 comms systemd: Failed at step SELINUX_CONTEXT spawning
/usr/bin/rsync: Protocol not available
Jun  7 18:19:25 comms systemd:
rsyncd@16-127.0.0.1:873-127.0.0.1:53456.service: main process exited,
code=exited, status=229/SELINUX_CONTEXT
Jun  7 18:19:25 comms systemd: Unit
rsyncd@16-127.0.0.1:873-127.0.0.1:53456.service entered failed state.

Prior to connecting to 873 the proxy calls setsockcreatecon. I think
that the error is coming from a systemd getpeercon call. What I'm
confused about is why the socket would not have a context. Any ideas?

Another data point is that if I netcat directly to 873 systemd
starts rsync without any issues. Also if I proxy to another port (ex.
1 instead of 873) and run a server on it getpeercon of the
connection from the proxy reports the expected context.

Ted
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel