Re: [systemd-devel] Requested transaction contradicts existing jobs: start is destructive
On Thu, 7 May 2020 at 21:48, Mark Bannister wrote: > I wasn't sure how to follow your exact steps because I wasn't clear > when you wrote session-N.scope whether I was expected to replace N > with a number and if so where that number comes from? Please confirm. The 'N' in session-N.scope is replaced by the active session ID, the N is what you see in loginctl under the SESSION column for your user. For example: $ loginctl SESSION UID USER SEAT TTY 75 1000 kkd seat0 tty2 1 sessions listed. Here, my session scope is session-75.scope. However, there may be multiple sessions, so the best way to find out on CentOS is doing cat /proc/self/cgroup, you will see something like "0::/user.slice/user-1000.slice/session-75.scope", the scope unit name is present there. `loginctl user-status` should list the active session with an asterisk (*) in the Sessions: field. You can also use: $ busctl get-property org.freedesktop.login1 /org/freedesktop/login1/session/self org.freedesktop.login1.Session Id which returns: s "75". If you are able to reproduce this on 219-67, please also attach the output of: systemctl show session-N.scope where 'N' is your session ID. -- Kartikeya ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Requested transaction contradicts existing jobs: start is destructive
At Thu May 7 15:13:14 UTC 2020, Kumar Kartikeya Dwivedi wrote: > Well, maybe you're mixing two issues into one. The original report was > about session scopes failing due to a conflict of job types, which you > should not be seeing on 219-67 Sorry if I confused the thread by mentioning systemd-run, in my ignorance I presumed that systemd-run was doing more or less the same as opening a new SSH session would have done inside the machinery of systemd. So let's forget about that for a moment, other than recording the fact that I reproduced it once with systemd-run on 219-67. Prior to my last response this entire thread was about one single issue occurring when SSH sessions were opening and closing. The original report is definitely happening on 219-67 which means that the github issue you found is not the cause (because that patch is not included in 219-67). This much is for certain. I wasn't sure how to follow your exact steps because I wasn't clear when you wrote session-N.scope whether I was expected to replace N with a number and if so where that number comes from? Please confirm. Thanks Mark ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Requested transaction contradicts existing jobs: start is destructive
On Thu, 7 May 2020 at 19:31, Mark Bannister wrote:
>
> You're missing something, although I don't know what, because we're
> definitely seeing this on 219-67 from
> http://vault.centos.org/7.7.1908/os/Source/SPackages/ and it
> definitely does not contain the change that you've linked to. The
> only custom patch we've applied is
> https://github.com/systemd/systemd/commit/cf99f8eacf1c864b19a6a02edea78c43f3185cb7.
> Something else is clearly going on here.
>
Well, maybe you're mixing two issues into one. The original report was
about session scopes failing due to a conflict of job types, which you
should not be seeing on 219-67 (I have no idea how you would without
the session scope having a Requires= dependency on the slice, which
wouldn't be the case on that version), but it is certainly possible to
get this message otherwise in other cases where two jobs conflict and
one of those cannot be deleted due to the relationship constraints. I
cannot tell without more context why, but it would be wise to first
try and reproduce the problem with the latest systemd version to see
if it is already fixed. These errors are less common now as a lot of
effort has been spent on finding and fixing some of these corner cases
over the years, especially after v219.
> Thanks for the suggestions on how to reproduce. I managed to
> reproduce it once (first time lucky) with:
>
> # service=$(systemd-run --uid=$RUNUSER /tmp/myscript 2>&1 | awk
> '{sub("\\.$", ""); print $NF}'); systemctl --no-block stop $service &
> systemctl --no-block stop user-$RUNUID.slice; sleep 1; systemctl
> list-jobs
>
> ... where /tmp/myscript is:
>
> #!/bin/sh
> trap '' INT TERM HUP
> sleep 1d
>
> However, after the first success, repeated attempts failed to reproduce it.
>
Sigh...
What you posted above is very different from what I asked you to
follow (where the conflicting job was queued by logind). I can't
really tell what you did there. How did you trigger the conflicting
job, and what was the job mode used?
If you tried logging in to see if it triggers the problem, well, this
unit runs under the system instance (it very likely does, as it
appears from your systemd-run invocation), it never creates any
dependency on the user slice at all, and therefore does *not* keep it
waiting, and hence the race is still as narrow, unless you specify
--slice=user-UID.slice.
Moreover, on 219-67, it would never create a conflict (given that
systemctl show lists user-UID.slice under Wants= and not Requires=).
Also, were you able to reproduce with the steps I listed?
> While most of the time this is triggered by SSH session, I have
> discovered that we have one script that is using systemd-run and which
> failed to launch a command with this 'transaction is destructive'
> error. Could it be that when this problem is triggered, a systemd-run
> could fail?
You should see which service is said to have conflicting job types
(given that you applied the verbose logging patch). However, if you
can't reproduce them with the latest systemd, it would be best to ask
your distribution maintainers to integrate the fixes back into the
systemd version they maintain.
> Alas systemd is so big and so baked into everything these days that we
> daren't do more than add a handful of custom patches to the version
> shipped with the release. So if there is a small patch we can add
> that fixes it in both 219-67 and 219-73, that would be immensely
> helpful.
I think the PR I mentioned before should help with the logind issue,
but I can't tell if it applies cleanly to 219 (and it really only
helps post 219-69+, as far I am concerned, the logind message atleast
cannot appear on 219-67 (where the scope Wants= the slice); convince
me by showing evidence to the contrary). If you're seeing this with
other cases, those changes won't help you.
--
Kartikeya
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Requested transaction contradicts existing jobs: start is destructive
On Wed May 6 17:49:02 UTC 2020, Kumar Kartikeya Dwivedi wrote:
> This issue is NOT reproducible with the version you reported
> originally. The stock CentOS image on my cloud provider comes with the
> same version (219-67.el7_7.4) that you said you see this issue with,
> but that can't be, as it doesn't include this change:
> https://github.com/systemd-rhel/rhel-7/commit/0600681f04e3818282a2d518ec3e6afee85f7978,
> unless I'm missing something. It took me a while to figure this out...
You're missing something, although I don't know what, because we're
definitely seeing this on 219-67 from
http://vault.centos.org/7.7.1908/os/Source/SPackages/ and it
definitely does not contain the change that you've linked to. The
only custom patch we've applied is
https://github.com/systemd/systemd/commit/cf99f8eacf1c864b19a6a02edea78c43f3185cb7.
Something else is clearly going on here.
Thanks for the suggestions on how to reproduce. I managed to
reproduce it once (first time lucky) with:
# service=$(systemd-run --uid=$RUNUSER /tmp/myscript 2>&1 | awk
'{sub("\\.$", ""); print $NF}'); systemctl --no-block stop $service &
systemctl --no-block stop user-$RUNUID.slice; sleep 1; systemctl
list-jobs
... where /tmp/myscript is:
#!/bin/sh
trap '' INT TERM HUP
sleep 1d
However, after the first success, repeated attempts failed to reproduce it.
> All in all, you can't do much about these errors, they're mostly
> harmless (at worst you need to retry the login)
While most of the time this is triggered by SSH session, I have
discovered that we have one script that is using systemd-run and which
failed to launch a command with this 'transaction is destructive'
error. Could it be that when this problem is triggered, a systemd-run
could fail?
> and upgrading to v228
> (the patchset mentioned in the previous exchange) or higher should fix
> all of this for you.
Alas systemd is so big and so baked into everything these days that we
daren't do more than add a handful of custom patches to the version
shipped with the release. So if there is a small patch we can add
that fixes it in both 219-67 and 219-73, that would be immensely
helpful.
Best regards
Mark
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Moving a service from one systemd slice to another..
On Do, 07.05.20 09:21, Tomasz Torcz (to...@pipebreaker.pl) wrote: > On Thu, May 07, 2020 at 08:56:24AM +0200, Lennart Poettering wrote: > > > > > Is there an API in systemd to move specific program to a dedicated > > > > > slice? > > > > > > > > Use Slice= in the service file. > > > > > > That would require having a service file. I don't have it. I just have > > > a > > > single process in session which I need to treat specially. Thus I move > > > it to separate slice. I have /etc/systemd/system/kodi.slice to define > > > this separate slice and resource controls for it. > > > > use "systemd-run --scope --slice=kodi.slice …" for that. > > That's not moving, that's starting a new instance. Yeah. Moving is not supported. As mentioned. > > > Really, if you muck around with the cgroup tree yourself you void all > > warranty. I mean, it's Linux you can do whatever you want, but it's > > certainly outside of how this is intended to be used, i.e. the > > cgroupsv2 "single writer" concept. > > > > Please read up on this here: > > > > https://systemd.io/CGROUP_DELEGATION > > It's documented in all length... > > From quick glance, it talks about a bit different concepts. > I'm not delegating, I'm using systemd to manage tree for me. > There are no multiple writers - only systemd (no kubernetes and > such). Well, if you manually muck with the cgroup tree, then *you* are the second writer. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Moving a service from one systemd slice to another..
On Thu, May 07, 2020 at 08:56:24AM +0200, Lennart Poettering wrote: > > > > Is there an API in systemd to move specific program to a dedicated > > > > slice? > > > > > > Use Slice= in the service file. > > > > That would require having a service file. I don't have it. I just have a > > single process in session which I need to treat specially. Thus I move > > it to separate slice. I have /etc/systemd/system/kodi.slice to define > > this separate slice and resource controls for it. > > use "systemd-run --scope --slice=kodi.slice …" for that. That's not moving, that's starting a new instance. > Really, if you muck around with the cgroup tree yourself you void all > warranty. I mean, it's Linux you can do whatever you want, but it's > certainly outside of how this is intended to be used, i.e. the > cgroupsv2 "single writer" concept. > > Please read up on this here: > > https://systemd.io/CGROUP_DELEGATION > It's documented in all length... From quick glance, it talks about a bit different concepts. I'm not delegating, I'm using systemd to manage tree for me. There are no multiple writers - only systemd (no kubernetes and such). But let me read this webpage carefully first. -- Tomasz Torcz 72->| 80->| to...@pipebreaker.pl 72->| 80->| ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Antw: [EXT] Re: Moving a service from one systemd slice to another..
On Do, 07.05.20 08:49, Ulrich Windl (ulrich.wi...@rz.uni-regensburg.de) wrote: > >>> Lennart Poettering schrieb am 06.05.2020 um 17:16 > in > Nachricht > <19374_1588778203_5EB2D4DB_19374_121_1_20200506151636.GL89018@gardel-login>: > > On Mo, 04.05.20 11:52, nitish nagesh (nagesh.nit...@gmail.com) wrote: > > > >> Hello, > >> > >>We have this odd want to move a daemon between different systemd > slices. > >> Not sure if that's even a valid thing to do, but here is the > >> requirement. > > > > The Linux kernel gets a bit confused with accounting if you migrate > > running processes between cgroups. One usually shouldn't do that > > except when forking off/execing new stuff. > > IMHO Linux should either handle the case correctly, or forbid it. Well, until recently there wasn't even a clone-into-cgroup API in the Linux kernel. Which meant you first had to fork off your process and then afterwards migrate it to the destination cgroup, i.e. it first had to be forked off and then migrated because there was no other API. This has been fixed now however, and we can clone directly into a cgroup, so that the process never exists elsewhere. we are not using that in systemd yet, but with that they could in theory forbid migration entirely and have accounting "atomic", i.e. that a process exists in a specific cgroup or in another for its entirely lifetime, but never migrates. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: [EXT] Re: network-online.target time
>>> Peter Morrow schrieb am 06.05.2020 um 17:56 in Nachricht <21624_1588780597_5EB2DE35_21624_6_1_aa19cc55fd1846a2344357bfb5465a645993bf1a.ca e...@microsoft.com>: [...] >> >> You can turn that off if you like: IPv6DuplicateAddressDetection= > > This option worked well when I tried it (total time came down to > ~700ms) but unfortunately disabling it entirely isn't an option. > > FWIW setting CONFIG_IPV6_OPTIMISTIC_DAD=y in the kernel is providing a > good middle ground. As a note from real life: A major IT company claiming to be the horsepower for enterprises, (so to say) once sent out a replacement motherboard for a defective server, and when we powered up that server again, a storage system from the same company failed. As it turned out the new motherboard had the same MAC address as that storage system's NIC... When the technician suggested to replace the server motherboard again, I was asking whether it wouldn't be easier for all if the MAC address would be re-programmed. The technician said that it's impossible to change the MAC addresses... ;-) Long story put short: Tests for duplicate addresses DO make sense > > Peter. > > >> Lennart >> >> ‑‑ >> Lennart Poettering, Berlin > > > ___ > systemd‑devel mailing list > systemd‑de...@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd‑devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Moving a service from one systemd slice to another..
On Do, 07.05.20 08:28, Tomasz Torcz (to...@pipebreaker.pl) wrote: > On Wed, May 06, 2020 at 11:53:14PM +0200, Lennart Poettering wrote: > > > > > > > > > > You can use low-level cgroup access to move it. Something like: > > > > > systemctl show --property MainPID --value your.service > > > > > > /sys/fs/cgroup/NEW.slice/cgroup.procs > > > > > > > > You void your warranty if you do that. Moreover, on cgroupvs2 this > > > > doesn't work really, since inner cgroups cannot have processes and > > > > slices are by definition inner cgroups. > > > > > > Doesn't work? I beg to differ, the following is on Fedora 31: > > > > > > Control group /: > > > -.slice > > > ├─kodi.slice > > > │ └─2872766 /usr/lib64/kodi/kodi-wayland ← here it has process > > > > > > > Is there an API in systemd to move specific program to a dedicated > > > slice? > > > > Use Slice= in the service file. > > That would require having a service file. I don't have it. I just have a > single process in session which I need to treat specially. Thus I move > it to separate slice. I have /etc/systemd/system/kodi.slice to define > this separate slice and resource controls for it. use "systemd-run --scope --slice=kodi.slice …" for that. Really, if you muck around with the cgroup tree yourself you void all warranty. I mean, it's Linux you can do whatever you want, but it's certainly outside of how this is intended to be used, i.e. the cgroupsv2 "single writer" concept. > > If you want to run kodi as user service, then assign your user the > > resources you want to assign to kodi, and then distribute them from > > there to kodi, and reduce it for the rest. > > This user has a bunch of other things running (compilation, emails, > batch jobs, torrents). I want Kodi to be isolated from the interference. > Slices/scopes (are they different? Manpage descriptions of them seem to say > the > same things using slightly different words) should do the trick. Slices are a way for hierarchially grouping services + scopes. They do not directly contain processes, but only indirectly because services + scopes contain them. Services encapsulates processes systemd starts for you. Scopes encapsulates processes you fork off yourself but want to manage in a separate own unit. When instantiating a scope you specify a bunch of processes which will be migrated to the new scope. Process generally are placed in services and scopes early on when they are forked off. Migrating already long running processes into a different scope or service is not something one wants to do because accounting is fucked then. Slices are the inner nodes of the cgroup tree basically, and scopes + services the leaves. Please read up on this here: https://systemd.io/CGROUP_DELEGATION It's documented in all length... Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Antw: [EXT] Re: Moving a service from one systemd slice to another..
>>> Lennart Poettering schrieb am 06.05.2020 um 17:16 in Nachricht <19374_1588778203_5EB2D4DB_19374_121_1_20200506151636.GL89018@gardel-login>: > On Mo, 04.05.20 11:52, nitish nagesh (nagesh.nit...@gmail.com) wrote: > >> Hello, >> >>We have this odd want to move a daemon between different systemd slices. >> Not sure if that's even a valid thing to do, but here is the >> requirement. > > The Linux kernel gets a bit confused with accounting if you migrate > running processes between cgroups. One usually shouldn't do that > except when forking off/execing new stuff. IMHO Linux should either handle the case correctly, or forbid it. > >>The daemon while booting up belongs to a systemd slice (say X) which has >> parameters like startupCPUShares tuned to bring the system up faster. >> However for normal operations after bootup, it would be apt to make it >> belong to another slice (say Y) with different cgroup parameters >> configurations (ex: CPUShares). The set of daemons that belong to X and Y >> are totally different, except for this one daemon. Also the cgroup >> parameter that are set via slice Y are totally different than those set via >> slice X. > > This is not supported. In systemd you can migrate services between > slices only by stopping them and starting them again. > >> A few basic questions: >> ‑ Can a daemon be a part of 2 slices? > > no. > >> ‑ If yes, going by the example above, if slice X loads first followed by >> slice Y, does it mean when slice X is in force this daemon will have >> startUpCPUShares set & when slice Y is loaded the CPUShares will be >> set? > > I don't grok this, sorry. > >>Please suggest if there are alternate ways in systemd to handle this >> requirement. > > We don't support that. Slice assignments are sticky during service > runtime. > > Lennart > > ‑‑ > Lennart Poettering, Berlin > ___ > systemd‑devel mailing list > systemd‑de...@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd‑devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Moving a service from one systemd slice to another..
On Wed, May 06, 2020 at 11:53:14PM +0200, Lennart Poettering wrote: > > > > > > > > You can use low-level cgroup access to move it. Something like: > > > > systemctl show --property MainPID --value your.service > > > > > /sys/fs/cgroup/NEW.slice/cgroup.procs > > > > > > You void your warranty if you do that. Moreover, on cgroupvs2 this > > > doesn't work really, since inner cgroups cannot have processes and > > > slices are by definition inner cgroups. > > > > Doesn't work? I beg to differ, the following is on Fedora 31: > > > > Control group /: > > -.slice > > ├─kodi.slice > > │ └─2872766 /usr/lib64/kodi/kodi-wayland ← here it has process > > > > Is there an API in systemd to move specific program to a dedicated > > slice? > > Use Slice= in the service file. That would require having a service file. I don't have it. I just have a single process in session which I need to treat specially. Thus I move it to separate slice. I have /etc/systemd/system/kodi.slice to define this separate slice and resource controls for it. > > In my usecase, I've created a top-level .slice for Kodi player. I want > > it to have priority over everything on my machine - over other users' > > processes, over virtual machines, over everything in system.slice. > > I achieve it by having this top-level slice with CPUShares, CPUWeight, > > BlockIOWeight, IOWeight much higher and IODeviceLatencyTargetSec > > much lower than rest of the slices. Seems to work. > > I presume you you mean a top-level slice in the system manager? If so > you need to run kodi as a system service too. That wouldn't work, Kodi is strictly tied to user session - file permissions, configurations, access to screen and audio (Kodi is a media player), interaction with other parts of user session. > If you want to run kodi as user service, then assign your user the > resources you want to assign to kodi, and then distribute them from > there to kodi, and reduce it for the rest. This user has a bunch of other things running (compilation, emails, batch jobs, torrents). I want Kodi to be isolated from the interference. Slices/scopes (are they different? Manpage descriptions of them seem to say the same things using slightly different words) should do the trick. P.S. I'm sorry for hijacking original poster's thread. -- Tomasz Torcz 72->| 80->| to...@pipebreaker.pl 72->| 80->| ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] No DHCP server is started with DHCPServer=yes
On Wed, May 6, 2020 at 11:47 PM Lennart Poettering wrote: > > On Mi, 06.05.20 23:43, Francis Moreau (francis.m...@gmail.com) wrote: > > > Hi, > > > > I'm running systemd v245 (in the host and in the containers) and I > > have 2 containers whose veth link is connected to a bridge. > > > > One container named "c1" uses "DHCPServer=yes" and is supposed to > > provide an ipv4 address to the second container "c2" but for some > > reasons it seems that no DHCP server is started in "c1". At least > > "journalctl -u systemd-networkd | grep DHCP" doesn't show anything > > interesting although the debug logs have been enabled. > > > > I monitored the DHCP traffic in "c1" andI could see the DISCOVER > > requests sent by "c2" but no answer from "c1". So it really seems that > > no DHCP server is running in "c1". > > > > Could anybody tell me how I could make sure that no DHCP server has > > been started and help me understanding why ? > > Are you sure your .network files match on the iface properly? What > does "networkctl status" say about the interface? > Yes I am pretty sure because the rest of the setting in my .network file has been taken into account, For example, it also defines a static address for "c1" which is correctly set. # networkctl status host0 ● 2: host0 Link File: n/a Network File: /etc/systemd/network/50-dhcp-server.network Type: ether State: routable (configured) HW Address: 36:1a:18:6b:19:52 MTU: 1500 (min: 68, max: 65535) Queue Length (Tx/Rx): 1/1 Auto negotiation: no Speed: 10Gbps Duplex: full Port: tp Address: 44.0.0.1 fe80::341a:18ff:fe6b:1952 Gateway: 44.0.0.1 May 07 01:56:00 node1 systemd-networkd[20]: host0: Addresses set May 07 01:56:00 node1 systemd-networkd[20]: host0: Configuring route: dst: n/a, src: n/a, gw: 44.0.0.1, p> May 07 01:56:00 node1 systemd-networkd[20]: host0: Setting routes May 07 01:56:00 node1 systemd-networkd[20]: host0: Received remembered route: dst: n/a, src: n/a, gw: 44.> May 07 01:56:00 node1 systemd-networkd[20]: host0: Routes set May 07 01:56:02 node1 systemd-networkd[20]: host0: Remembering foreign address: fe80::341a:18ff:fe6b:1952> May 07 01:56:02 node1 systemd-networkd[20]: host0: Gained IPv6LL May 07 01:56:02 node1 systemd-networkd[20]: host0: Discovering IPv6 routers May 07 01:56:02 node1 systemd-networkd[20]: host0: Remembering route: dst: fe80::341a:18ff:fe6b:1952/128,> May 07 01:56:14 node1 systemd-networkd[20]: host0: State changed: configuring -> configured I also tried to remove /usr/lib/systemd/network/80-container-host0.network because it's unclear to me if it's completely overriden by my .network file or not. According to the output of 'networkctl status' it seems so becuase only my network file is listed but after reading systemd.network man page, I'm really not sure. But without 80-container-host0.network it doesn't work. Thank you -- Francis ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
