Re: [systemd-devel] rg...@outlook.com
On Mo, 07.06.21 22:47, Aravindhan Krishnan (aravindhan...@gmail.com) wrote: > Hi Lennart, > > Thanks for the quick response. Yes, we are running systemd inside the > docker. We were also able to see the same issue even on top of > Centos 7.9. Unlike pretty much all other container managers Docker doesn't really make it easy to run systemd inside it. Docker upstream is pretty hostile towards systemd, so this is unlikely to change. We document pretty extensively what container managers have to do to make sure systemd just works inside containers. Pretty much all container managers just implement that, but Docker doesn't. This is what they need to implement: https://systemd.io/CONTAINER_INTERFACE Consider switching to a different container manager implementation, there are plenty others. (in particular podman is mostly a drop-in replacement for Docker, if you need Docker semantics. Podman upstream isn't hostile towards systemd, so things mostly just work there.) > Attaching the kernel and OS details of the centos host > > # uname -r > 3.10.0-1160.25.1.el7.x86_64 > > # cat /etc/centos-release > CentOS Linux release 7.9.2009 (Core) This is very old. You might want to switch to a newer OS for this anyway. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] rg...@outlook.com
Am Montag, dem 07.06.2021 um 21:26 +0530 schrieb Aravindhan Krishnan: > Hi Folks, > > I am finding anomalous behavior when I am trying to run dhclient > process inside my docker container in vanilla Ubuntu 16.04 host. The > service gets into "deactivating" state and is stuck forever. In the > mail I have attached a minimalistic reproduction of the issue seen. > Thanks, > Aravindhan > > Regards, > Aravindhan Krishnan... Hi Aravindhan, don't run systemd in a docker container in the first place? Also Ubuntu 16.04 is really old. IMHO all your problems are created by your setup itself. I really appreciate the minimal example you attached, but if your premise (running systemd in a docker container and not just one simple process) is already wrong, than no solution can be right. BR Silvio ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] rg...@outlook.com
Hi Lennart, Thanks for the quick response. Yes, we are running systemd inside the docker. We were also able to see the same issue even on top of Centos 7.9. Attaching the kernel and OS details of the centos host # uname -r 3.10.0-1160.25.1.el7.x86_64 # cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) Regards, Aravindhan Krishnan... On Mon, 7 Jun 2021 at 22:24, Lennart Poettering wrote: > On Mo, 07.06.21 21:26, Aravindhan Krishnan (aravindhan...@gmail.com) > wrote: > > > Hi Folks, > > > > I am finding anomalous behavior when I am trying to run dhclient process > > inside my docker container in vanilla Ubuntu 16.04 host. The service gets > > into "deactivating" state and is stuck forever. In the mail I have > attached > > a minimalistic reproduction of the issue seen. > > Are you running systemd inside of a Docker container on Ubuntu 16.04? > > Docker isn't really up to that. In particular not 5y old versions of it. > > Lennart > > -- > Lennart Poettering, Berlin > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] rg...@outlook.com
On Mo, 07.06.21 21:26, Aravindhan Krishnan (aravindhan...@gmail.com) wrote: > Hi Folks, > > I am finding anomalous behavior when I am trying to run dhclient process > inside my docker container in vanilla Ubuntu 16.04 host. The service gets > into "deactivating" state and is stuck forever. In the mail I have attached > a minimalistic reproduction of the issue seen. Are you running systemd inside of a Docker container on Ubuntu 16.04? Docker isn't really up to that. In particular not 5y old versions of it. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04
Am 07.06.21 um 17:57 schrieb Aravindhan Krishnan: Adding Raghav. And sorry the subject should have stated: Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04 and why didn't you fix it in your own reply? to your problem: you have a wild mix of docker, systemd-units and shellscripts but don't provide the source of the scripts nor the systemd unit overly complex for something that can be trivial as: [root@srv-rhsoft:~]$ cat /etc/systemd/system/network-wan-dhcp.service [Unit] Description=Internet DHCP-Client [Service] Type=forking ExecStart=/usr/sbin/dhclient -4 -q --no-pid --request-options subnet-mask,broadcast-address,routers br-wan PermissionsStartOnly=yes SuccessExitStatus=80 Restart=always RestartSec=5 ProtectSystem=strict ProtectHome=yes ReadWritePaths=-/var/lib/dhclient PrivateTmp=yes NoNewPrivileges=yes ProtectKernelTunables=yes ProtectKernelModules=yes ProtectControlGroups=yes MemoryDenyWriteExecute=yes CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW LockPersonality=yes PrivateDevices=yes ProtectHostname=yes RestrictNamespaces=yes RestrictRealtime=yes RestrictSUIDSGID=yes ProtectClock=true ProtectKernelLogs=true UMask=077 SystemCallArchitectures=native SystemCallFilter=@system-service @network-io @privileged SystemCallFilter=~@aio @chown @clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io @reboot @resources @swap InaccessiblePaths=-/boot InaccessiblePaths=-/efi InaccessiblePaths=-/root On Mon, 7 Jun 2021 at 21:26, Aravindhan Krishnan mailto:aravindhan...@gmail.com>> wrote: Hi Folks, I am finding anomalous behavior when I am trying to run dhclient process inside my docker container in vanilla Ubuntu 16.04 host. The service gets into "deactivating" state and is stuck forever. In the mail I have attached a minimalistic reproduction of the issue seen. Working logic: * There is a sample trial@.service script which invokes the `trial` binary with the option passed to the systemd service via @ option * The valid options are sleep and dhclient_ * The binary either invokes a long-lived sleep process or dhclient process on the said interface_name based on the input * The binary then spawns `kill_trial.sh` script. The script sleeps for 20 seconds and kills the parent `trial` binary. The kill signal is SIGKILL in the trial example. In the real-world, this can be a SIGSEGV indicating a crash in the parent process. * If the trial binary was started for sleep process things work fine and service goes into "failed" state as expected * However, in case of dhclient, the service is stuck in "deactivating" state if the underlying host OS is Ubuntu 16.04. This works well if the host is running Ubuntu 20.04. * We have kept TimeoutStopSec to infinity, because in real-word deployments, the core collection post a crash takes varying time depending on the memory config on the host. Steps to reproduce # tar -xf minimal_repro.tar -C minimal_repro/ # cd minimal_repro/ # docker build -t trial . # docker rm -f trial # docker run -it -d --net=host --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name trial trial # docker exec -it trial bash # systemctl start trial@dhclient_eth1.service # #Keep monitoring trial@dhclient_eth1.service -- issue should be seen within 20-30 seconds on Ubuntu 16.04 host # systemctl status trial@dhclient_eth1.service ● trial@dhclient_eth1.service - Trial Loaded: loaded (/etc/systemd/system/trial@.service; static; vendor preset: enabled) Active: deactivating (stop-sigterm) (Result: signal) since Mon 2021-06-07 13:19:12 UTC; 1min 11s ago Process: 55 ExecStartPre=/bin/bash /etc/systemd/system/trial_service_script.sh pre_start dhclient_eth1 (code=exited, status=0/SUCCESS) Process: 56 ExecStart=/bin/bash /etc/systemd/system/trial_service_script.sh start dhclient_eth1 (code=killed, signal=KILL) Main PID: 56 (code=killed, signal=KILL) Tasks: 0 (limit: 38590) Memory: 588.0K CGroup: /docker/903fca0cee1387b7c2113a36ee5efdb3a25edd1e60584fe5da5d0c5b5ffd8241/system.slice/system-trial.slice/trial@dhclient_eth1.service # #NOTE: `Active: deactivating` -- in stuck state # #Running `systemctl daemon-reload` forces the service to go to failed state # systemctl start trial@sleep.service # #Keep monitoring trial@sleep.service -- would be killed in 20-30 seconds and goes into failed state as expected # # systemctl status trial@sleep.service ● trial@sleep.service - Trial Loaded: loaded (/etc/systemd/system/trial@.service; static; vendor preset: enabled) Active: failed (Result: signal) since Mon 2021-06-07 13:38:19 UTC; 21s ago Process: 113
Re: [systemd-devel] rg...@outlook.com
Adding Raghav. And sorry the subject should have stated: Discrepancy in using dhclient b/w ubuntu 20.04 and ubuntu 16.04 Regards, Aravindhan Krishnan... On Mon, 7 Jun 2021 at 21:26, Aravindhan Krishnan wrote: > Hi Folks, > > I am finding anomalous behavior when I am trying to run dhclient process > inside my docker container in vanilla Ubuntu 16.04 host. The service gets > into "deactivating" state and is stuck forever. In the mail I have attached > a minimalistic reproduction of the issue seen. > > Working logic: > >- There is a sample trial@.service script which invokes the `trial` >binary with the option passed to the systemd service via @ option >- The valid options are sleep and dhclient_ >- The binary either invokes a long-lived sleep process or dhclient >process on the said interface_name based on the input >- The binary then spawns `kill_trial.sh` script. The script sleeps for >20 seconds and kills the parent `trial` binary. The kill signal is SIGKILL >in the trial example. In the real-world, this can be a SIGSEGV indicating a >crash in the parent process. >- If the trial binary was started for sleep process things work fine >and service goes into "failed" state as expected >- However, in case of dhclient, the service is stuck in "deactivating" >state if the underlying host OS is Ubuntu 16.04. This works well if the >host is running Ubuntu 20.04. >- We have kept TimeoutStopSec to infinity, because in real-word >deployments, the core collection post a crash takes varying time depending >on the memory config on the host. > > > Steps to reproduce > # tar -xf minimal_repro.tar -C minimal_repro/ > # cd minimal_repro/ > # docker build -t trial . > # docker rm -f trial > # docker run -it -d --net=host --privileged -v > /sys/fs/cgroup:/sys/fs/cgroup:ro --name trial trial > # docker exec -it trial bash > > # systemctl start trial@dhclient_eth1.service > > # #Keep monitoring trial@dhclient_eth1.service -- issue should be seen > within 20-30 seconds on Ubuntu 16.04 host > > # systemctl status trial@dhclient_eth1.service > ● trial@dhclient_eth1.service - Trial > Loaded: loaded (/etc/systemd/system/trial@.service; static; vendor > preset: enabled) > Active: deactivating (stop-sigterm) (Result: signal) since Mon > 2021-06-07 13:19:12 UTC; 1min 11s ago > Process: 55 ExecStartPre=/bin/bash > /etc/systemd/system/trial_service_script.sh pre_start dhclient_eth1 > (code=exited, status=0/SUCCESS) > Process: 56 ExecStart=/bin/bash > /etc/systemd/system/trial_service_script.sh start dhclient_eth1 > (code=killed, signal=KILL) >Main PID: 56 (code=killed, signal=KILL) > Tasks: 0 (limit: 38590) > Memory: 588.0K > CGroup: > /docker/903fca0cee1387b7c2113a36ee5efdb3a25edd1e60584fe5da5d0c5b5ffd8241/system.slice/system-trial.slice/trial@dhclient_eth1.service > > # #NOTE: `Active: deactivating` -- in stuck state > # #Running `systemctl daemon-reload` forces the service to go to failed > state > > # systemctl start trial@sleep.service > > # #Keep monitoring trial@sleep.service -- would be killed in 20-30 > seconds and goes into failed state as expected > > # # systemctl status trial@sleep.service > ● trial@sleep.service - Trial > Loaded: loaded (/etc/systemd/system/trial@.service; static; vendor > preset: enabled) > Active: failed (Result: signal) since Mon 2021-06-07 13:38:19 UTC; > 21s ago > Process: 113 ExecStartPre=/bin/bash > /etc/systemd/system/trial_service_script.sh pre_start sleep (code=exited, > status=0/SUCCESS) > Process: 114 ExecStart=/bin/bash > /etc/systemd/system/trial_service_script.sh start sleep (code=killed, > signal=KILL) > Process: 129 ExecStopPost=/bin/bash > /etc/systemd/system/trial_service_script.sh post_stop sleep (code=exited, > status=0/SUCCESS) >Main PID: 114 (code=killed, signal=KILL) > > Please advise on what can help us in alleviating the issue. > > Thanks, > Aravindhan > > Regards, > Aravindhan Krishnan... > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] rg...@outlook.com
Hi Folks, I am finding anomalous behavior when I am trying to run dhclient process inside my docker container in vanilla Ubuntu 16.04 host. The service gets into "deactivating" state and is stuck forever. In the mail I have attached a minimalistic reproduction of the issue seen. Working logic: - There is a sample trial@.service script which invokes the `trial` binary with the option passed to the systemd service via @ option - The valid options are sleep and dhclient_ - The binary either invokes a long-lived sleep process or dhclient process on the said interface_name based on the input - The binary then spawns `kill_trial.sh` script. The script sleeps for 20 seconds and kills the parent `trial` binary. The kill signal is SIGKILL in the trial example. In the real-world, this can be a SIGSEGV indicating a crash in the parent process. - If the trial binary was started for sleep process things work fine and service goes into "failed" state as expected - However, in case of dhclient, the service is stuck in "deactivating" state if the underlying host OS is Ubuntu 16.04. This works well if the host is running Ubuntu 20.04. - We have kept TimeoutStopSec to infinity, because in real-word deployments, the core collection post a crash takes varying time depending on the memory config on the host. Steps to reproduce # tar -xf minimal_repro.tar -C minimal_repro/ # cd minimal_repro/ # docker build -t trial . # docker rm -f trial # docker run -it -d --net=host --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name trial trial # docker exec -it trial bash # systemctl start trial@dhclient_eth1.service # #Keep monitoring trial@dhclient_eth1.service -- issue should be seen within 20-30 seconds on Ubuntu 16.04 host # systemctl status trial@dhclient_eth1.service ● trial@dhclient_eth1.service - Trial Loaded: loaded (/etc/systemd/system/trial@.service; static; vendor preset: enabled) Active: deactivating (stop-sigterm) (Result: signal) since Mon 2021-06-07 13:19:12 UTC; 1min 11s ago Process: 55 ExecStartPre=/bin/bash /etc/systemd/system/trial_service_script.sh pre_start dhclient_eth1 (code=exited, status=0/SUCCESS) Process: 56 ExecStart=/bin/bash /etc/systemd/system/trial_service_script.sh start dhclient_eth1 (code=killed, signal=KILL) Main PID: 56 (code=killed, signal=KILL) Tasks: 0 (limit: 38590) Memory: 588.0K CGroup: /docker/903fca0cee1387b7c2113a36ee5efdb3a25edd1e60584fe5da5d0c5b5ffd8241/system.slice/system-trial.slice/trial@dhclient_eth1.service # #NOTE: `Active: deactivating` -- in stuck state # #Running `systemctl daemon-reload` forces the service to go to failed state # systemctl start trial@sleep.service # #Keep monitoring trial@sleep.service -- would be killed in 20-30 seconds and goes into failed state as expected # # systemctl status trial@sleep.service ● trial@sleep.service - Trial Loaded: loaded (/etc/systemd/system/trial@.service; static; vendor preset: enabled) Active: failed (Result: signal) since Mon 2021-06-07 13:38:19 UTC; 21s ago Process: 113 ExecStartPre=/bin/bash /etc/systemd/system/trial_service_script.sh pre_start sleep (code=exited, status=0/SUCCESS) Process: 114 ExecStart=/bin/bash /etc/systemd/system/trial_service_script.sh start sleep (code=killed, signal=KILL) Process: 129 ExecStopPost=/bin/bash /etc/systemd/system/trial_service_script.sh post_stop sleep (code=exited, status=0/SUCCESS) Main PID: 114 (code=killed, signal=KILL) Please advise on what can help us in alleviating the issue. Thanks, Aravindhan Regards, Aravindhan Krishnan... minimal_repro.tar Description: Unix tar archive ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
