Re: [systemd-devel] Shutdown order in systemd
Zheng SHAO wrote on 03/08/2020 13:31: > Hello, > > We are finding a robust way to handle ACPI G2 soft off signal to graceful > shutdown our application. > To simplifier the problem, consider our instance is running with Nginx behind > a load balancer. > When the ACPI G2 soft off signal comes to the Nginx instance, we want to do > these jobs > > 1. Keep current HTTP connection works. > 2. Fail the health check in load balance side. > 3. Make sure new connection not comes from load balancer. > 4. Kill long connections if any connection exceeds to 60 seconds. > 5. Continue shutdown process. > > We are considering to achieve this by 2 options, > 1. Add a custom handler for `HandlePowerKey` in /etc/systemd/logind.conf. > 2. Add a system service so when systemd starting shutdown, this service will > be run first and block other service to be killed. Have you considered writing a service that takes a systemd-inhibit shutdown lock? This might not work but looking very quickly at https://www.freedesktop.org/wiki/Software/systemd/inhibit/ it would appear you get a PrepareForShutdown signal which could kick off your steps. Depending on how things work, you could just introduce a "delay" inhibit rather than a "block", e.g. a 70s delay could give you a bit of headroom to trigger fail the LB's health check. Or perhaps you could take a block lock, then when the prepareForShutdown() signal comes in, fail the LB, then when that is confirmed, add a new 60s delay inhibit (not sure if this works after shutdown has been triggered), then release the block inhibit and just wait for everything else to run it's course? Alternatively just keep the block inhibit right up until you want step 5 to begin. Again, this is pure speculation and the default handler for HandlePowerKey may bypass login (tho' I suspect not) and others may explain other reasons why this approach may not work. Good luck Col > The method[2] is a preferred way, but we can not find a correct implement for > this. > > ``` > [Unit] > Description=Delay shutdown > After=network-online.target network.target rsyslog.service > After=google-instance-setup.service google-network-daemon.service > After=systemd-user-sessions.service sshd.service google-fluentd.service > user.slice system.slice nss-user-lookup.target logind.service > Wants=network-online.target network.target rsyslog.service > google-instance-setup.service google-network-daemon.service > systemd-user-sessions.service sshd.service google-fluentd.service user.slice > system.slice multi-user.target nss-user-lookup.target logind.service > > [Service] > Type=oneshot > ExecStart=/bin/true > ExecStop=/root/shutdown.sh > RemainAfterExit=yes > KillMode=none > TimeoutStopSec=0 > StandardOutput=journal+console > > [Install] > WantedBy=multi-user.target > ``` > > /root/shutdown.sh > ``` > #!/bin/bash > > echo start shutdown > echo sleep 300 > sleep 300 > echo end shutdown > ``` > > We checked console output shows as follow, > ``` > CentOS Linux 7 (Core) > Kernel 3.10.0-1127.10.1.el7.x86_64 on an x86_64 > > shao-redis-prd-base login: Aug 3 21:19:25 shao-redis-prd-base chronyd[449]: > Selected source 169.254.169.254 > Aug 3 21:20:01 shao-redis-prd-base systemd: Created slice User Slice of root. > Aug 3 21:20:01 shao-redis-prd-base systemd: Started Session 1 of user root. > Aug 3 21:20:01 shao-redis-prd-base systemd: Removed slice User Slice of root. > Aug 3 21:20:11 shao-redis-prd-base systemd: Started Unbound recursive Domain > Name Server. > Aug 3 21:20:11 shao-redis-prd-base systemd: Reached target Host and Network > Name Lookups. > Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 0: > ipsecmod > Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 1: > validator > Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 2: > iterator > Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] info: start of service > (unbound 1.6.6). > Aug 3 21:20:59 shao-redis-prd-base systemd: Created slice User Slice of xxx. > Aug 3 21:20:59 shao-redis-prd-base systemd: Started Session 2 of user xxx. > Aug 3 21:21:29 shao-redis-prd-base systemd: Reached target Multi-User System. > Aug 3 21:21:29 shao-redis-prd-base systemd: Starting Update UTMP about > System Runlevel Changes... > Aug 3 21:21:29 shao-redis-prd-base systemd: Started Update UTMP about System > Runlevel Changes. > Aug 3 21:21:29 shao-redis-prd-base systemd: Startup finished in 768ms > (kernel) + 2.006s (initrd) + 2min 13.532s (userspace) = 2min 16.307s. > Aug 3 21:23:09 shao-redis-prd-base systemd: Removed slice > system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice. > Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Aug 3 21:23:09 > shao-redis-prd-base systemd: Stopped ACPI Event Daemon. > Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Authorization Manager. > Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Job spooling tools. > Aug
Re: [systemd-devel] [PATCH 0/6] RFC: Initial implementation of mount table handling using libmount kernelwatch
On Mon, 2020-08-03 at 09:30 +0100, Colin Guthrie wrote: > Ian Kent wrote on 29/07/2020 01:57: > > On Tue, 2020-07-28 at 16:13 +0200, Lennart Poettering wrote: > > > On Mo, 27.07.20 12:57, Ian Kent (ik...@redhat.com) wrote: > > > > > > > Further to my post about using the new mount table > > > > notifications in > > > > systemd I'd like to start by posting a few patches for > > > > discussion > > > > to > > > > hopefully get a better understanding of some aspects of how > > > > systemd > > > > mount unit state handling works, and also to get guidance on > > > > systemd > > > > coding style, and generally how I should be doing things in > > > > systemd > > > > as well as how best to debug problems. > > > > > > Thank you for working on this! > > > > > > We do reviews exclusively on github these days, could you please > > > submit this as PR on github? > > > > Thanks Lennart, will do. > > > > Although I was hoping for some generic discussion on this to set > > me going in the right direction. > > > > If that's not ok then I'll submit a PR when I think it's ready. > > Although I'm not as familiar with the development process as I used > to > be, you can file it now, even in it's current state for that > discussion > process. It can be an RFC Pull request. Just means the discussion can > all take place in Github and can go through various iterations there. > > So, in short, don't worry about it being fully "ready" before pushing > and doing a pull requests. I know I said I wasn't quite ready but your right, there's bound to be good advice about what I've done and what might need to change, at least now I have something that seems to behave as it's supposed to, ;) Ian ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Shutdown order in systemd
Hello, We are finding a robust way to handle ACPI G2 soft off signal to graceful shutdown our application. To simplifier the problem, consider our instance is running with Nginx behind a load balancer. When the ACPI G2 soft off signal comes to the Nginx instance, we want to do these jobs 1. Keep current HTTP connection works. 2. Fail the health check in load balance side. 3. Make sure new connection not comes from load balancer. 4. Kill long connections if any connection exceeds to 60 seconds. 5. Continue shutdown process. We are considering to achieve this by 2 options, 1. Add a custom handler for `HandlePowerKey` in /etc/systemd/logind.conf. 2. Add a system service so when systemd starting shutdown, this service will be run first and block other service to be killed. The method[2] is a preferred way, but we can not find a correct implement for this. ``` [Unit] Description=Delay shutdown After=network-online.target network.target rsyslog.service After=google-instance-setup.service google-network-daemon.service After=systemd-user-sessions.service sshd.service google-fluentd.service user.slice system.slice nss-user-lookup.target logind.service Wants=network-online.target network.target rsyslog.service google-instance-setup.service google-network-daemon.service systemd-user-sessions.service sshd.service google-fluentd.service user.slice system.slice multi-user.target nss-user-lookup.target logind.service [Service] Type=oneshot ExecStart=/bin/true ExecStop=/root/shutdown.sh RemainAfterExit=yes KillMode=none TimeoutStopSec=0 StandardOutput=journal+console [Install] WantedBy=multi-user.target ``` /root/shutdown.sh ``` #!/bin/bash echo start shutdown echo sleep 300 sleep 300 echo end shutdown ``` We checked console output shows as follow, ``` CentOS Linux 7 (Core) Kernel 3.10.0-1127.10.1.el7.x86_64 on an x86_64 shao-redis-prd-base login: Aug 3 21:19:25 shao-redis-prd-base chronyd[449]: Selected source 169.254.169.254 Aug 3 21:20:01 shao-redis-prd-base systemd: Created slice User Slice of root. Aug 3 21:20:01 shao-redis-prd-base systemd: Started Session 1 of user root. Aug 3 21:20:01 shao-redis-prd-base systemd: Removed slice User Slice of root. Aug 3 21:20:11 shao-redis-prd-base systemd: Started Unbound recursive Domain Name Server. Aug 3 21:20:11 shao-redis-prd-base systemd: Reached target Host and Network Name Lookups. Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 0: ipsecmod Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 1: validator Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 2: iterator Aug 3 21:20:11 shao-redis-prd-base unbound: [1204:0] info: start of service (unbound 1.6.6). Aug 3 21:20:59 shao-redis-prd-base systemd: Created slice User Slice of xxx. Aug 3 21:20:59 shao-redis-prd-base systemd: Started Session 2 of user xxx. Aug 3 21:21:29 shao-redis-prd-base systemd: Reached target Multi-User System. Aug 3 21:21:29 shao-redis-prd-base systemd: Starting Update UTMP about System Runlevel Changes... Aug 3 21:21:29 shao-redis-prd-base systemd: Started Update UTMP about System Runlevel Changes. Aug 3 21:21:29 shao-redis-prd-base systemd: Startup finished in 768ms (kernel) + 2.006s (initrd) + 2min 13.532s (userspace) = 2min 16.307s. Aug 3 21:23:09 shao-redis-prd-base systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped ACPI Event Daemon. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Authorization Manager. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Job spooling tools. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Getty on tty1. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Serial Getty on ttyS0. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped NTP client/server. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Google OSConfig Agent. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Command Scheduler. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Unbound recursive Domain Name Server. Aug 3 21:23:09 shao-redis-prd-base systemd: Removed slice system-serial\x2dgetty.slice. Aug 3 21:23:09 shao-redis-prd-base systemd: Starting Show Plymouth Power Off Screen... Aug 3 21:23:09 shao-redis-prd-base systemd: Removed slice system-getty.slice. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Session 2 of user kuma. Aug 3 21:23:09 shao-redis-prd-base systemd: Removed slice User Slice of kuma. Aug 3 21:23:09 shao-redis-prd-base systemd: Stopping Login Service... Aug 3 21:23:09 shao-redis-prd-base systemd: Stopped Login Service. [ 236.662420] shutdown.sh[1265]: start shutdown [ 236.662802] shutdown.sh[1265]: sleep 300 Aug 3 21:23:09 shao-redis-prd-base shutdown.sh: start shutdown Aug 3 21:23:09 shao-redis-prd-base shutdown.sh: sleep 300 Aug 3 21:23:09 shao-redis-prd-base GCEMetadataScripts[1266]:
Re: [systemd-devel] [PATCH 0/6] RFC: Initial implementation of mount table handling using libmount kernelwatch
On Mon, 2020-08-03 at 11:34 +0200, Lennart Poettering wrote: > On Mi, 29.07.20 08:57, Ian Kent (ik...@redhat.com) wrote: > > > On Tue, 2020-07-28 at 16:13 +0200, Lennart Poettering wrote: > > > On Mo, 27.07.20 12:57, Ian Kent (ik...@redhat.com) wrote: > > > > > > > Further to my post about using the new mount table > > > > notifications in > > > > systemd I'd like to start by posting a few patches for > > > > discussion > > > > to > > > > hopefully get a better understanding of some aspects of how > > > > systemd > > > > mount unit state handling works, and also to get guidance on > > > > systemd > > > > coding style, and generally how I should be doing things in > > > > systemd > > > > as well as how best to debug problems. > > > > > > Thank you for working on this! > > > > > > We do reviews exclusively on github these days, could you please > > > submit this as PR on github? > > > > Thanks Lennart, will do. > > > > Although I was hoping for some generic discussion on this to set > > me going in the right direction. > > > > If that's not ok then I'll submit a PR when I think it's ready. > > I am sorry, but we do code review in gitub. It's what it is good > for. If you want reviews on patches, even if thery are RFCs, post > them > as PR, like everyone else. > > PRs are general code commenting tools, they are not exclusively for > "final" versions. Yes, I've been looking at the systemd PR's and I see that. I'm fairly confident that the core functionality is working now so I'm not so concerned about feedback about what's happening in systemd any more. And it appears the reduction in overhead is quite significant as was expected. I've also made adjustments based on the systemd coding style notes which I found because you pointed me at github above, that's been useful too. But there's a bit more to do before I can raise a PR. Handling notification buffer overflow (and fall back to the old method) testing needs to be done. And also more testing of various cases and adjustments that may come out of it, as well as a meson build change so systemd builds against a libmount without the new changes. And finally I'll need to re-test all this when David's changes are merged upstream. Thanks for the feedback, Ian ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/6] RFC: Initial implementation of mount table handling using libmount kernelwatch
On Mi, 29.07.20 08:57, Ian Kent (ik...@redhat.com) wrote: > On Tue, 2020-07-28 at 16:13 +0200, Lennart Poettering wrote: > > On Mo, 27.07.20 12:57, Ian Kent (ik...@redhat.com) wrote: > > > > > Further to my post about using the new mount table notifications in > > > systemd I'd like to start by posting a few patches for discussion > > > to > > > hopefully get a better understanding of some aspects of how systemd > > > mount unit state handling works, and also to get guidance on > > > systemd > > > coding style, and generally how I should be doing things in systemd > > > as well as how best to debug problems. > > > > Thank you for working on this! > > > > We do reviews exclusively on github these days, could you please > > submit this as PR on github? > > Thanks Lennart, will do. > > Although I was hoping for some generic discussion on this to set > me going in the right direction. > > If that's not ok then I'll submit a PR when I think it's ready. I am sorry, but we do code review in gitub. It's what it is good for. If you want reviews on patches, even if thery are RFCs, post them as PR, like everyone else. PRs are general code commenting tools, they are not exclusively for "final" versions. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/6] RFC: Initial implementation of mount table handling using libmount kernelwatch
Ian Kent wrote on 29/07/2020 01:57: > On Tue, 2020-07-28 at 16:13 +0200, Lennart Poettering wrote: >> On Mo, 27.07.20 12:57, Ian Kent (ik...@redhat.com) wrote: >> >>> Further to my post about using the new mount table notifications in >>> systemd I'd like to start by posting a few patches for discussion >>> to >>> hopefully get a better understanding of some aspects of how systemd >>> mount unit state handling works, and also to get guidance on >>> systemd >>> coding style, and generally how I should be doing things in systemd >>> as well as how best to debug problems. >> >> Thank you for working on this! >> >> We do reviews exclusively on github these days, could you please >> submit this as PR on github? > > Thanks Lennart, will do. > > Although I was hoping for some generic discussion on this to set > me going in the right direction. > > If that's not ok then I'll submit a PR when I think it's ready. Although I'm not as familiar with the development process as I used to be, you can file it now, even in it's current state for that discussion process. It can be an RFC Pull request. Just means the discussion can all take place in Github and can go through various iterations there. So, in short, don't worry about it being fully "ready" before pushing and doing a pull requests. Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Problem understanding output of systemd-cgtop
Hi!
While examining a scheduling problem that might have resulted from restrictions
imposed by cgroups, I examined my machine using systemd-cgtop and systemd-cgls
(systemd-228-157.12.5.x86_64 of SLES12).
Among the things I don't understand are:
Why is systemd-cgtop outputting much less slices than systemd-cgls does?
Specifically I don't see the slice for the process I'm examining
("system-iotwatch.slice"). systemd-cgls shows it with three services.
Also, when using "systemd-cgtop --depth 1", I get this output:
Control GroupTasks %CPU Memory Input/s Output/s
/-1.1 564.7M--
/init.scope 1 ----
/system.slice 82 ----
/user.slice 18 ----
Does that output mean /init.scope, /system.slice, and /user.slice all don't
need any CPU and memory, while only / does?
If I don't restrict the depth, this is what I get (also when specifying a
higher depth):
Control GroupTasks %CPU Memory Input/s Output/s
/-3.5 565.8M--
/init.scope 1 ----
/system.slice 82 ----
/system.slice/auditd.service 2 ----
/system.slice/cron.service 1 ----
/system.slice/cups.service 1 ----
/system.slice/dbus.service 1 ----
/system.slice/haveged.service1 ----
/system.slice/icinga2.service 22 ----
/system.slice/logd.service 2 ----
/system.slice/lvm2-lvmetad.service 1 ----
/system.slice/monit.service 2 ----
/system.slice/nrpe.service 1 ----
/system.slice/nscd.service 11 ----
/system.slice/ntpd.service 2 ----
/system.slice/postfix.service3 ----
/system.slice/rpc-statd.service 1 ----
/system.slice/rpcbind.service1 ----
/system.slice/rsyslog.service5 ----
So why does / have CPU and memory usage, while all others don't?
Regards,
Ulrich
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Creating a fake logind seat with no devices [Experiment]
On Thursday, July 30, 2020 5:00:27 AM EDT Arseny Maslennikov wrote: > On Wed, Jul 29, 2020 at 11:17:51PM -0400, nerdopolis wrote: > > I am pretty much trying to cobble together tmux, cage, and vte, to create a > > full screen VT-like > > console lookalike. (as the kernel one takes input from all devices from all > > seats on a multiseat > > system). Mostly experimental. > > Instead of cage + vte, have you tried the following: > > https://github.com/Aetf/kmscon/ > > This is a fork (should I say, continuation?) of kmscon which builds from > source successfully on modern distributions. > > From then on you can run tmux there. > > Some distros even provide packages: > https://aur.archlinux.org/packages/kmscon-patched-git/ > http://git.altlinux.org/gears/k/kmscon.git > > I can confirm it worked for me as a VT replacement on a single seat and > properly integrated with logind, but I don't have the hardware to test a > multi-seat configuration. > > > (tmux server/client being used to avoid having to start vte and cage as > > root for /sbin/login) > > > > > > The problem is that the tmux PTY running /sbin/login , those sessions don't > > properly create a full > > pam_systemd sessions on seat0, as I get "VT number out of range", because > > as I understand seat0 > > sessions need to be on a valid TTY. Non-seat0 seats don't have this > > restriction, but non-seat0 > > seat hardware is far from a guarantee > > > > > > However I will note so far the only major side effect I notice to not > > having (sd-pam) started in > > this session is, for instance `pkexec` won't work, without sudo. And > > systemctl doesn't try to use > > PolicyKit and whatnot, as from what I understand, that needs to be on a > > session assigned to a seat > > to work I think > > > > Thanks > > > > Hi I wasn't aware that there was that recentish of a fork of kmscon. Interesting. It does run as root though. Now testing with it, it actually help me figure out what I had wrong. I just needed (in addition to Environment=XDG_SEAT=seat0 ) was to set XDG_VTNR=x before running /sbin/login and when logging in a full session with policykit working. (So I was tilting at the wrong windmill thinking it couldn't be done on seat0) However this ends up causing some conflicts, as logind sees it as two sessions on one TTY, and it tries to activate the second one most of the time, so the non-root display server ends up in an inactive state. I was trying an experiment where the display server and the (graphical) terminal emulator could provide something like KMSCON, but not run as root, and provide a valid login prompt. However I realize also that if bad terminal output could get some kind of (theoretical) VTE exploit to run arbitrary code, they will just be able to send commands to start a new session on the tmux socket. So I guess I solved the problem I initially had. Dang. Thanks. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
