Re: [systemd-devel] Synchronization Between Services at Shutdown

[Andrei Borzenkov]

В Wed, 1 Apr 2015 18:22:50 -0700
Kurt von Laven  пишет:

> Hello folks,
> 
> I am trying to ensure that my system DBus service has time to handle a DBus
> method call before it shuts down. My DBus service is implemented as a
> system systemd service; let's call it service A. It runs a single-threaded,
> synchronous event loop that listens for DBus method calls and handles them
> appropriately. This event loop also listens for SIGTERM. When SIGTERM is
> received, service A currently terminates the event loop, performs any
> necessary writes to disk, and releases its resources. I am assuming that
> when the system is shut down cleanly, systemd initially sends SIGTERM to
> systemd services and only sends SIGKILL to abort it abruptly if it takes
> too long to shut down when asked nicely.
> 
> I have another system systemd service, call it service B, that runs an
> event loop listening for SIGTERM. When service B receives a SIGTERM, it
> makes the aforementioned DBus method call and then flushes the system bus.
> Service B specifies Requires=
> Service
> A and After=
> Service
> A. Service A makes no mention of service B in its unit configuration file.
> Does systemd make any sort of guarantee that service A will get a chance to
> handle the DBus method call before receiving SIGTERM, or does my code as
> written have a race condition?
> 

systemd will not initiate stopping of A before B has stopped. If it
does, this should be considered a bug. So it is up to service B to
handle termination correctly. Note that systemd will send SIGKILL after
timeout if service has not terminated.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Kurt von Laven]

Thanks for the quick reply, Andrei. It sounds like there is indeed a race
condition in my code then since there is nothing explicit in my
implementation of service A that makes it wait until it has processed the
method call from service B before it stops listening for method calls. Did
I get that right?

On Wed, Apr 1, 2015 at 8:26 PM, Andrei Borzenkov 
wrote:

> В Wed, 1 Apr 2015 18:22:50 -0700
> Kurt von Laven  пишет:
>
> > Hello folks,
> >
> > I am trying to ensure that my system DBus service has time to handle a
> DBus
> > method call before it shuts down. My DBus service is implemented as a
> > system systemd service; let's call it service A. It runs a
> single-threaded,
> > synchronous event loop that listens for DBus method calls and handles
> them
> > appropriately. This event loop also listens for SIGTERM. When SIGTERM is
> > received, service A currently terminates the event loop, performs any
> > necessary writes to disk, and releases its resources. I am assuming that
> > when the system is shut down cleanly, systemd initially sends SIGTERM to
> > systemd services and only sends SIGKILL to abort it abruptly if it takes
> > too long to shut down when asked nicely.
> >
> > I have another system systemd service, call it service B, that runs an
> > event loop listening for SIGTERM. When service B receives a SIGTERM, it
> > makes the aforementioned DBus method call and then flushes the system
> bus.
> > Service B specifies Requires=
> > <
> http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Requires=
> >Service
> > A and After=
> > <
> http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=
> >Service
> > A. Service A makes no mention of service B in its unit configuration
> file.
> > Does systemd make any sort of guarantee that service A will get a chance
> to
> > handle the DBus method call before receiving SIGTERM, or does my code as
> > written have a race condition?
> >
>
> systemd will not initiate stopping of A before B has stopped. If it
> does, this should be considered a bug. So it is up to service B to
> handle termination correctly. Note that systemd will send SIGKILL after
> timeout if service has not terminated.
>



-- 
Kurt von Laven | Endless Mobile |  EndlessM.com 
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Wed, 01.04.15 18:22, Kurt von Laven (k...@endlessm.com) wrote:

> Hello folks,
> 
> I am trying to ensure that my system DBus service has time to handle a DBus
> method call before it shuts down. My DBus service is implemented as a
> system systemd service; let's call it service A. It runs a single-threaded,
> synchronous event loop that listens for DBus method calls and handles them
> appropriately. This event loop also listens for SIGTERM. When SIGTERM is
> received, service A currently terminates the event loop, performs any
> necessary writes to disk, and releases its resources. I am assuming that
> when the system is shut down cleanly, systemd initially sends SIGTERM to
> systemd services and only sends SIGKILL to abort it abruptly if it takes
> too long to shut down when asked nicely.
> 
> I have another system systemd service, call it service B, that runs an
> event loop listening for SIGTERM. When service B receives a SIGTERM, it
> makes the aforementioned DBus method call and then flushes the system bus.
> Service B specifies Requires=
> Service
> A and After=
> Service
> A. Service A makes no mention of service B in its unit configuration file.
> Does systemd make any sort of guarantee that service A will get a chance to
> handle the DBus method call before receiving SIGTERM, or does my code as
> written have a race condition?

The general rule in systemd is that the shutdown order is the reverse
startup order. If you declare After=foo.service in a service
bar.service, then this will hence have two effects: 1) when both are
started foo.service is started before bar.service. And 2) when both
are stopped bar.service is stopped before foo.service. 

If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down before
dbus is.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Michael Biebl]

2015-04-02 11:06 GMT+02:00 Lennart Poettering :
> If you want to ensure that bus communication still works in your
> shutdown code, you hence need to make sure you place
> After=dbus.service in your services, so that you are shut down before
> dbus is.

Type=dbus service currently only get a dependency on dbus.socket (via
After=basic.target).

Incidentally, pitti had to deal with bug reports the last couple of
days, where dbus.service was stopped too early during shutdown, so
e.g. wpa_supplicant.service and other D-Bus using services failed to
stop properly on shutdown.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Thu, 02.04.15 13:00, Michael Biebl (mbi...@gmail.com) wrote:

> 2015-04-02 11:06 GMT+02:00 Lennart Poettering :
> > If you want to ensure that bus communication still works in your
> > shutdown code, you hence need to make sure you place
> > After=dbus.service in your services, so that you are shut down before
> > dbus is.
> 
> Type=dbus service currently only get a dependency on dbus.socket (via
> After=basic.target).

Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Michael Biebl]

2015-04-02 13:03 GMT+02:00 Lennart Poettering :
> On Thu, 02.04.15 13:00, Michael Biebl (mbi...@gmail.com) wrote:
>
>> 2015-04-02 11:06 GMT+02:00 Lennart Poettering :
>> > If you want to ensure that bus communication still works in your
>> > shutdown code, you hence need to make sure you place
>> > After=dbus.service in your services, so that you are shut down before
>> > dbus is.
>>
>> Type=dbus service currently only get a dependency on dbus.socket (via
>> After=basic.target).
>
> Yeah, and rightfully so. I mean, a service really should be able to
> shutdown if dbus is dead. In fact, it should be able to shutdown in
> pretty much any situation...

Apparently they don't. There were all sorts of failures caused by dbus
being shut down too early.

https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
the related bug reports afair.

Since Martin has been working on this, I'll let him comment on this.
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Thu, 02.04.15 13:07, Michael Biebl (mbi...@gmail.com) wrote:

> 2015-04-02 13:03 GMT+02:00 Lennart Poettering :
> > On Thu, 02.04.15 13:00, Michael Biebl (mbi...@gmail.com) wrote:
> >
> >> 2015-04-02 11:06 GMT+02:00 Lennart Poettering :
> >> > If you want to ensure that bus communication still works in your
> >> > shutdown code, you hence need to make sure you place
> >> > After=dbus.service in your services, so that you are shut down before
> >> > dbus is.
> >>
> >> Type=dbus service currently only get a dependency on dbus.socket (via
> >> After=basic.target).
> >
> > Yeah, and rightfully so. I mean, a service really should be able to
> > shutdown if dbus is dead. In fact, it should be able to shutdown in
> > pretty much any situation...
> 
> Apparently they don't. There were all sorts of failures caused by dbus
> being shut down too early.
> 
> https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
> the related bug reports afair.

Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...

You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target. With that in place
during shutdown the mounts will be unmounted first, and NM/wpa only
shut down after that. See systemd.special(7) for details.

I really don't see what dbus has to do with all this...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Simon McVittie]

On 02/04/15 14:31, Lennart Poettering wrote:
> Hmm? I really don't see how the NFS vs wpa_supplicant issue has
> anything to do with dbus? NFS doesn't care about dbus at all...

It does inasmuch as it requires networking to be up, which *might*
require dbus (e.g. for NetworkManager).

> You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> and pull it it via Wants=remote-fs-pre.target.

AIUI services with the default dependencies depend on remote-fs.target?
Distributions have historically supported NFS /usr and /var if
networking is done via ifupdown or something, and used (the LSB
equivalent of) remote-fs.target to represent that. NFS /usr or /var with
NetworkManager or similar already didn't work, because that needs D-Bus,
which needs /usr and /var.

systemd mandates an initramfs that mounts /usr (which has thankfully
made it into Debian 8, although for a while it didn't look as though
that was going to happen), but AIUI /var is still something that comes
up later?

It's very easy to get into circular dependencies here; there's the
remote filesystem issue, and also the fact that dbus-daemon wants to
resolve users/groups, hence might need NIS or LDAP to be up. I
personally consider some sort of automatically-synched local cache like
sssd to be a far better answer than NIS or LDAP, and NFS root better
than NFS /usr and/or /var, but I am not the only one whose opinion
matters when considering whether a feature regression in distributions
is acceptable.

I think the best solution might be a way for "dbus-daemon --system" to
not be required to be Before early targets like sysinit.target (to avoid
circular dependencies), but for it to survive until the final killing
spree anyway. Does systemd have a way to say that, or are
startup/shutdown dependencies always symmetric?

Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
on this; one possible hack is to have dbus.service's ExecStop not
actually stop dbus-daemon, so that it stays around until the final
killing spree just before /usr and /var are unmounted.

I'm happy to modify dbus.service if that's (part or all of) the
solution, but before I can do that we need to come up with a solution
that doesn't cause new dependency cycles.

S

-- 
Simon McVittie
Collabora Ltd. 

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Martin Pitt]

Lennart Poettering [2015-04-02 13:03 +0200]:
> Yeah, and rightfully so. I mean, a service really should be able to
> shutdown if dbus is dead. In fact, it should be able to shutdown in
> pretty much any situation...

They do, but that's irrelevant here. The problem is that dbus.service
shuts down very early as nothing prevents it from that. All
dependencies just refer to dbus.socket, but TBH socket (re-)activation
totally doesn't work for dbus. On restarts it forgets all of its
previous state/connections, so a service which still needs to talk to
dbus during shutdown will just fail if dbus.service already stopped.

So what happens is that dbus.service gets stopped early, takes
NetworkManager, wpasupplicant etc. with it, and then the network
mounts can't be unmounted any more as the network is already down
(which isn't intended at this point yet, as network.target is still
active).

Synthesizing After=dbus.service dependencies instead of dbus.socket
ones for Type=dbus would help, as then you'd have the ordering which
prevents dbus.service from going down early. But this is prone to
dependency loops.

This has also been discussed with upstream on
https://bugs.freedesktop.org/show_bug.cgi?id=89847

I don't really have a good solution, the "don't really stop dbus" is a
relatively safe hack that's appropriate for the "we need to release in
two weeks" freeze. Obviously the really good solution is to get kdbus,
then dbus will always be available :-)

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Martin Pitt]

Lennart Poettering [2015-04-02 15:31 +0200]:
> You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> and pull it it via Wants=remote-fs-pre.target. With that in place
> during shutdown the mounts will be unmounted first, and NM/wpa only
> shut down after that.

wpa_supplicant and NM already have Before=network.target which is even
stronger. But that doesn't help -- it's D-Bus which we need to leave
running longer, as dbus going down early tears down all dbus clients as well
prematurely, thus rendering the above dependencies useless.

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Dan Williams]

On Thu, 2015-04-02 at 15:04 +0100, Simon McVittie wrote:
> On 02/04/15 14:31, Lennart Poettering wrote:
> > Hmm? I really don't see how the NFS vs wpa_supplicant issue has
> > anything to do with dbus? NFS doesn't care about dbus at all...
> 
> It does inasmuch as it requires networking to be up, which *might*
> require dbus (e.g. for NetworkManager).

NM requires wpa_supplicant running for WiFi because WiFi requires more
active management; the AP might go away, you might need to re-key, roam,
etc at any moment.  The supplicant can't really stop until you no longer
need the interface.  But in reality, (a) anyone mounting a rootfs over
WiFi is asking for trouble, and (b) why the F does NFS still not handle
network dropouts reliably?  But I digress.

> > You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> > and pull it it via Wants=remote-fs-pre.target.
> 
> AIUI services with the default dependencies depend on remote-fs.target?
> Distributions have historically supported NFS /usr and /var if
> networking is done via ifupdown or something, and used (the LSB
> equivalent of) remote-fs.target to represent that. NFS /usr or /var with
> NetworkManager or similar already didn't work, because that needs D-Bus,
> which needs /usr and /var.

NM hasn't needed dbus-daemon running for a long time in early boot, or
for any root-level operation.  So there's no requirement for /usr
or /var or anything else mounted, because root communication happens on
a private socket in /run.

wpa_supplicant does require dbus, since it has no private socket code.

Dan

> systemd mandates an initramfs that mounts /usr (which has thankfully
> made it into Debian 8, although for a while it didn't look as though
> that was going to happen), but AIUI /var is still something that comes
> up later?
> 
> It's very easy to get into circular dependencies here; there's the
> remote filesystem issue, and also the fact that dbus-daemon wants to
> resolve users/groups, hence might need NIS or LDAP to be up. I
> personally consider some sort of automatically-synched local cache like
> sssd to be a far better answer than NIS or LDAP, and NFS root better
> than NFS /usr and/or /var, but I am not the only one whose opinion
> matters when considering whether a feature regression in distributions
> is acceptable.
> 
> I think the best solution might be a way for "dbus-daemon --system" to
> not be required to be Before early targets like sysinit.target (to avoid
> circular dependencies), but for it to survive until the final killing
> spree anyway. Does systemd have a way to say that, or are
> startup/shutdown dependencies always symmetric?
> 
> Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
> on this; one possible hack is to have dbus.service's ExecStop not
> actually stop dbus-daemon, so that it stays around until the final
> killing spree just before /usr and /var are unmounted.
> 
> I'm happy to modify dbus.service if that's (part or all of) the
> solution, but before I can do that we need to come up with a solution
> that doesn't cause new dependency cycles.
> 
> S
> 


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Dan Williams]

On Thu, 2015-04-02 at 16:31 +0200, Martin Pitt wrote:
> Lennart Poettering [2015-04-02 15:31 +0200]:
> > You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> > and pull it it via Wants=remote-fs-pre.target. With that in place
> > during shutdown the mounts will be unmounted first, and NM/wpa only
> > shut down after that.
> 
> wpa_supplicant and NM already have Before=network.target which is even
> stronger. But that doesn't help -- it's D-Bus which we need to leave
> running longer, as dbus going down early tears down all dbus clients as well
> prematurely, thus rendering the above dependencies useless.

At least on the NM side, NM shouldn't be doing anything when dbus-daemon
exits, except logging something and trying to reconnect.  It shouldn't
touch network interfaces except those that require a management daemon,
like wpa_supplicant or Bluetooth or teamd, provided you use the internal
DHCP client.  But ethernet, bridge/bond/vlan/infiniband/etc should be
fine.  Obviously if the supplicant goes down, WiFi goes down, but
running something like NFS over WiFi is kinda asking for trouble, given
that NFS doesn't handle network dropouts well in the first place.

Dan

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Kurt von Laven]

Both services A and B specify Requires=dbus.service and After=dbus.service.
If I am interpreting everything that's being said correctly, there is a
race condition in my code since service A's implementation doesn't require
it to wait until it has processed the method call from service B before it
stops listening for method calls. Does that sound right?

On Thu, Apr 2, 2015 at 6:31 AM, Lennart Poettering 
wrote:

> On Thu, 02.04.15 13:07, Michael Biebl (mbi...@gmail.com) wrote:
>
> > 2015-04-02 13:03 GMT+02:00 Lennart Poettering :
> > > On Thu, 02.04.15 13:00, Michael Biebl (mbi...@gmail.com) wrote:
> > >
> > >> 2015-04-02 11:06 GMT+02:00 Lennart Poettering  >:
> > >> > If you want to ensure that bus communication still works in your
> > >> > shutdown code, you hence need to make sure you place
> > >> > After=dbus.service in your services, so that you are shut down
> before
> > >> > dbus is.
> > >>
> > >> Type=dbus service currently only get a dependency on dbus.socket (via
> > >> After=basic.target).
> > >
> > > Yeah, and rightfully so. I mean, a service really should be able to
> > > shutdown if dbus is dead. In fact, it should be able to shutdown in
> > > pretty much any situation...
> >
> > Apparently they don't. There were all sorts of failures caused by dbus
> > being shut down too early.
> >
> > https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
> > the related bug reports afair.
>
> Hmm? I really don't see how the NFS vs wpa_supplicant issue has
> anything to do with dbus? NFS doesn't care about dbus at all...
>
> You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> and pull it it via Wants=remote-fs-pre.target. With that in place
> during shutdown the mounts will be unmounted first, and NM/wpa only
> shut down after that. See systemd.special(7) for details.
>
> I really don't see what dbus has to do with all this...
>
> Lennart
>
> --
> Lennart Poettering, Red Hat
>



-- 
Kurt von Laven | Endless Mobile |  EndlessM.com 
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Kurt von Laven]

Actually I think I just answered my own question and in the process
realized that I didn't provide enough context in my original question.
There is a race condition in my code because when service B receives
SIGTERM, it calls g_dbus_connection_flush_sync

after
a g_dbus_proxy_call

to service A in an incorrect attempt to ensure that service A has handled
the DBus proxy call. The problem with this is that
g_dbus_connection_flush_sync ensures all outgoing messages are sent over
the DBus connection but doesn't wait around for replies to those messages.
Service B should instead call g_dbus_proxy_call_sync

and
call neither g_dbus_connection_flush_sync nor g_dbus_proxy_call. Please
correct me if I am missing something.

On a separate note, how long does systemd wait after sending a SIGTERM to a
service before sending it a SIGKILL? Is it reasonable to assume that a few
g_dbus_proxy_call_sync calls would complete in that time if the slowest
part of those calls is just the standard overhead of a DBus call? I ask
because currently I acquire a shutdown inhibitor
 in service B,
and I now suspect this is unnecessary.

Be well,
Kurt

On Thu, Apr 2, 2015 at 9:49 AM, Kurt von Laven  wrote:

> Both services A and B specify Requires=dbus.service and
> After=dbus.service. If I am interpreting everything that's being said
> correctly, there is a race condition in my code since service A's
> implementation doesn't require it to wait until it has processed the method
> call from service B before it stops listening for method calls. Does that
> sound right?
>
> On Thu, Apr 2, 2015 at 6:31 AM, Lennart Poettering  > wrote:
>
>> On Thu, 02.04.15 13:07, Michael Biebl (mbi...@gmail.com) wrote:
>>
>> > 2015-04-02 13:03 GMT+02:00 Lennart Poettering :
>> > > On Thu, 02.04.15 13:00, Michael Biebl (mbi...@gmail.com) wrote:
>> > >
>> > >> 2015-04-02 11:06 GMT+02:00 Lennart Poettering <
>> lenn...@poettering.net>:
>> > >> > If you want to ensure that bus communication still works in your
>> > >> > shutdown code, you hence need to make sure you place
>> > >> > After=dbus.service in your services, so that you are shut down
>> before
>> > >> > dbus is.
>> > >>
>> > >> Type=dbus service currently only get a dependency on dbus.socket (via
>> > >> After=basic.target).
>> > >
>> > > Yeah, and rightfully so. I mean, a service really should be able to
>> > > shutdown if dbus is dead. In fact, it should be able to shutdown in
>> > > pretty much any situation...
>> >
>> > Apparently they don't. There were all sorts of failures caused by dbus
>> > being shut down too early.
>> >
>> > https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
>> > the related bug reports afair.
>>
>> Hmm? I really don't see how the NFS vs wpa_supplicant issue has
>> anything to do with dbus? NFS doesn't care about dbus at all...
>>
>> You need to order wpa_supplicant and NM Before=remote-fs-pre.target
>> and pull it it via Wants=remote-fs-pre.target. With that in place
>> during shutdown the mounts will be unmounted first, and NM/wpa only
>> shut down after that. See systemd.special(7) for details.
>>
>> I really don't see what dbus has to do with all this...
>>
>> Lennart
>>
>> --
>> Lennart Poettering, Red Hat
>>
>
>
>
> --
> Kurt von Laven | Endless Mobile |  EndlessM.com 
>



-- 
Kurt von Laven | Endless Mobile |  EndlessM.com 
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Andrei Borzenkov]

On Fri, Apr 3, 2015 at 6:55 AM, Kurt von Laven  wrote:
>
> On a separate note, how long does systemd wait after sending a SIGTERM to a
> service before sending it a SIGKILL?

http://www.freedesktop.org/software/systemd/man/systemd.kill.html
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Martin Pitt]

Hello,

Dan Williams [2015-04-02  9:51 -0500]:
> On Thu, 2015-04-02 at 16:31 +0200, Martin Pitt wrote:
> > wpa_supplicant and NM already have Before=network.target which is even
> > stronger. But that doesn't help -- it's D-Bus which we need to leave
> > running longer, as dbus going down early tears down all dbus clients as well
> > prematurely, thus rendering the above dependencies useless.
> 
> At least on the NM side, NM shouldn't be doing anything when dbus-daemon
> exits, except logging something and trying to reconnect.

Ah, that's correct. It complains a lot, but indeed our bug reporters
say that everything works fine with ethernet on shutdown. The remote
fs specific problems are due to wpasupplicant going down early with
dbus instead of at the (much later) time when we intend it to go down.
Avahi, libvirt etc. are more "typical" dbus services which go down
when the system dbus does, and do cause trouble.

Martin

-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Thu, 02.04.15 15:04, Simon McVittie (simon.mcvit...@collabora.co.uk) wrote:

> On 02/04/15 14:31, Lennart Poettering wrote:
> > Hmm? I really don't see how the NFS vs wpa_supplicant issue has
> > anything to do with dbus? NFS doesn't care about dbus at all...
> 
> It does inasmuch as it requires networking to be up, which *might*
> require dbus (e.g. for NetworkManager).

In this case you should make sure that NM stays up until the NFS
shares are unmounted, and if NM wants to have dbus around until its
very end it should use After=dbus.service. However NFS mounts and dbus
have little to do with each other.

> > You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> > and pull it it via Wants=remote-fs-pre.target.
> 
> AIUI services with the default dependencies depend on
> remote-fs.target?

No. local-fs.target is part of the default dependencies, but
remote-fs.target is not.

> Distributions have historically supported NFS /usr and /var if
> networking is done via ifupdown or something, and used (the LSB
> equivalent of) remote-fs.target to represent that. NFS /usr or /var with
> NetworkManager or similar already didn't work, because that needs D-Bus,
> which needs /usr and /var.

Split-out /usr is not supported by systemd unless already mounted in
the initrd.

Split-out /var is supported and does not have to be done in the
initrd. It's incompatible of course with NM because NM needs dbus, and
dbus needs /var.

> systemd mandates an initramfs that mounts /usr (which has thankfully
> made it into Debian 8, although for a while it didn't look as though
> that was going to happen), but AIUI /var is still something that comes
> up later?

Yes.

> It's very easy to get into circular dependencies here; there's the
> remote filesystem issue, and also the fact that dbus-daemon wants to
> resolve users/groups, hence might need NIS or LDAP to be up. 

Well, no, this is not supported.

dbus is a late boot service, and it's difficult to change that. The
major reason is bus activation: as long as dbus-daemon is running all
its activatable names are activatable, and this easily results in
deadlocks, since actually activating many of them would mean
activating late boot services, which results in deadlocks all over the
place. (simply because there is so much code that tries to contact
some service if it can, but skips if it doesn't, that is run from udev
rules or weird other early-boot stuff).

In kdbus this problem goes away since bus name are not always
actviatable, but can be made activatable only during later boot. With
kdbus we can schedule when names become activtable, which dbus-daemon
simply doesn't support.

> I personally consider some sort of automatically-synched local cache
> like sssd to be a far better answer than NIS or LDAP, and NFS root
> better than NFS /usr and/or /var, but I am not the only one whose
> opinion matters when considering whether a feature regression in
> distributions is acceptable.

Well, the way this is supposed to work with systemd is that NIS/LDAP
becomes available with the nss-user-lookup.target target, and user
logins wait for that (as well as remote-fs.target). NIS/LDAP only adds
real (i.e. "human") users to the user database, and system users
*must* exist all the time before that.

Both remote-fs.target and nss-user-lookup.target are *not* included in
basic.target but are pulled in later.

> I think the best solution might be a way for "dbus-daemon --system" to
> not be required to be Before early targets like sysinit.target (to avoid
> circular dependencies), but for it to survive until the final killing
> spree anyway. Does systemd have a way to say that, or are
> startup/shutdown dependencies always symmetric?

Yes they are always symmetric.

dbus-daemon cannot really be started before basic.target

> Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
> on this; one possible hack is to have dbus.service's ExecStop not
> actually stop dbus-daemon, so that it stays around until the final
> killing spree just before /usr and /var are unmounted.

Urks! Don't do that.

> I'm happy to modify dbus.service if that's (part or all of) the
> solution, but before I can do that we need to come up with a solution
> that doesn't cause new dependency cycles.

Honestly, services should not require dbus to be around when shutting
down. And if they do anyway, they should simply specifiy the explicit
After=dbus.service dependency, and all is good. Don't fuck up the dbus
unit file, with lines like the above!

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Thu, 02.04.15 16:28, Martin Pitt (martin.p...@ubuntu.com) wrote:

> Lennart Poettering [2015-04-02 13:03 +0200]:
> > Yeah, and rightfully so. I mean, a service really should be able to
> > shutdown if dbus is dead. In fact, it should be able to shutdown in
> > pretty much any situation...
> 
> They do, but that's irrelevant here. The problem is that dbus.service
> shuts down very early as nothing prevents it from that. All
> dependencies just refer to dbus.socket, but TBH socket (re-)activation
> totally doesn't work for dbus.

systemd will not reactivate services when shutting down, we refuse to
go in circles...

And even if it would reactivate dbus, it wouldn't help you, since all
services disappeared from the bus then...

> On restarts it forgets all of its
> previous state/connections, so a service which still needs to talk to
> dbus during shutdown will just fail if dbus.service already stopped.

Well, if that service is really this broken that it needs dbus when
all it should do is exit, then it should declare After=dbus.service,
and all is good.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Thu, 02.04.15 16:31, Martin Pitt (martin.p...@ubuntu.com) wrote:

> Lennart Poettering [2015-04-02 15:31 +0200]:
> > You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> > and pull it it via Wants=remote-fs-pre.target. With that in place
> > during shutdown the mounts will be unmounted first, and NM/wpa only
> > shut down after that.
> 
> wpa_supplicant and NM already have Before=network.target which is even
> stronger. But that doesn't help -- it's D-Bus which we need to leave
> running longer, as dbus going down early tears down all dbus clients as well
> prematurely, thus rendering the above dependencies useless.

Well, it sounds really wrong btw, if NM shuts down the ifaces just
because it got kicked off the bus... That's massively wrong.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Thu, 02.04.15 22:28, Martin Pitt (martin.p...@ubuntu.com) wrote:

> Hello,
> 
> Dan Williams [2015-04-02  9:51 -0500]:
> > On Thu, 2015-04-02 at 16:31 +0200, Martin Pitt wrote:
> > > wpa_supplicant and NM already have Before=network.target which is even
> > > stronger. But that doesn't help -- it's D-Bus which we need to leave
> > > running longer, as dbus going down early tears down all dbus clients as 
> > > well
> > > prematurely, thus rendering the above dependencies useless.
> > 
> > At least on the NM side, NM shouldn't be doing anything when dbus-daemon
> > exits, except logging something and trying to reconnect.
> 
> Ah, that's correct. It complains a lot, but indeed our bug reporters
> say that everything works fine with ethernet on shutdown. The remote
> fs specific problems are due to wpasupplicant going down early with
> dbus instead of at the (much later) time when we intend it to go down.
> Avahi, libvirt etc. are more "typical" dbus services which go down
> when the system dbus does, and do cause trouble.

Avahi reconnects when dbus goes away, it does not exit.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Thu, 02.04.15 20:55, Kurt von Laven (k...@endlessm.com) wrote:

> On a separate note, how long does systemd wait after sending a SIGTERM to a
> service before sending it a SIGKILL? 

90s. This may be configured per-service in TimeoutStopSec= and
globally with DefaultTimeoutStopSec= in system.conf.

he default of 90s appears to be not documented anyway right
now. That's a bug. It used to be documented, but somehow was
lost... Will fix.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Andrei Borzenkov]

В Fri, 3 Apr 2015 14:04:10 +0200
Lennart Poettering  пишет:

> On Thu, 02.04.15 16:28, Martin Pitt (martin.p...@ubuntu.com) wrote:
> 
> > Lennart Poettering [2015-04-02 13:03 +0200]:
> > > Yeah, and rightfully so. I mean, a service really should be able to
> > > shutdown if dbus is dead. In fact, it should be able to shutdown in
> > > pretty much any situation...
> > 
> > They do, but that's irrelevant here. The problem is that dbus.service
> > shuts down very early as nothing prevents it from that. All
> > dependencies just refer to dbus.socket, but TBH socket (re-)activation
> > totally doesn't work for dbus.
> 
> systemd will not reactivate services when shutting down, we refuse to
> go in circles...
> 

Could you elaborate? systemd will obviously start services when
starting shutdown.target. How does systemd differentiate between
services that can and can not be started?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Martin Pitt]

Lennart Poettering [2015-04-03 14:06 +0200]:
> Well, it sounds really wrong btw, if NM shuts down the ifaces just
> because it got kicked off the bus... That's massively wrong.

It doesn't -- wpa_supplicant does.

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Dan Williams]

On Fri, 2015-04-03 at 14:01 +0200, Lennart Poettering wrote:
> On Thu, 02.04.15 15:04, Simon McVittie (simon.mcvit...@collabora.co.uk) wrote:
> 
> > On 02/04/15 14:31, Lennart Poettering wrote:
> > > Hmm? I really don't see how the NFS vs wpa_supplicant issue has
> > > anything to do with dbus? NFS doesn't care about dbus at all...
> > 
> > It does inasmuch as it requires networking to be up, which *might*
> > require dbus (e.g. for NetworkManager).
> 
> In this case you should make sure that NM stays up until the NFS
> shares are unmounted, and if NM wants to have dbus around until its
> very end it should use After=dbus.service. However NFS mounts and dbus
> have little to do with each other.

NM does not need D-Bus to be up and around, and it doesn't care if D-Bus
goes away.

The only issue with D-Bus going away is for interfaces that require
external management daemons, like WiFi or Bluetooth, where if those
daemons exit the interface must be shut down because they have much more
hands-on control that is essentially required for the interface to
operate.  Those daemons either need to not exit when D-Bus isn't
available, or they need to get ported to kdbus or something.

Dan

> > > You need to order wpa_supplicant and NM Before=remote-fs-pre.target
> > > and pull it it via Wants=remote-fs-pre.target.
> > 
> > AIUI services with the default dependencies depend on
> > remote-fs.target?
> 
> No. local-fs.target is part of the default dependencies, but
> remote-fs.target is not.
> 
> > Distributions have historically supported NFS /usr and /var if
> > networking is done via ifupdown or something, and used (the LSB
> > equivalent of) remote-fs.target to represent that. NFS /usr or /var with
> > NetworkManager or similar already didn't work, because that needs D-Bus,
> > which needs /usr and /var.
> 
> Split-out /usr is not supported by systemd unless already mounted in
> the initrd.
> 
> Split-out /var is supported and does not have to be done in the
> initrd. It's incompatible of course with NM because NM needs dbus, and
> dbus needs /var.
> 
> > systemd mandates an initramfs that mounts /usr (which has thankfully
> > made it into Debian 8, although for a while it didn't look as though
> > that was going to happen), but AIUI /var is still something that comes
> > up later?
> 
> Yes.
> 
> > It's very easy to get into circular dependencies here; there's the
> > remote filesystem issue, and also the fact that dbus-daemon wants to
> > resolve users/groups, hence might need NIS or LDAP to be up. 
> 
> Well, no, this is not supported.
> 
> dbus is a late boot service, and it's difficult to change that. The
> major reason is bus activation: as long as dbus-daemon is running all
> its activatable names are activatable, and this easily results in
> deadlocks, since actually activating many of them would mean
> activating late boot services, which results in deadlocks all over the
> place. (simply because there is so much code that tries to contact
> some service if it can, but skips if it doesn't, that is run from udev
> rules or weird other early-boot stuff).
> 
> In kdbus this problem goes away since bus name are not always
> actviatable, but can be made activatable only during later boot. With
> kdbus we can schedule when names become activtable, which dbus-daemon
> simply doesn't support.
> 
> > I personally consider some sort of automatically-synched local cache
> > like sssd to be a far better answer than NIS or LDAP, and NFS root
> > better than NFS /usr and/or /var, but I am not the only one whose
> > opinion matters when considering whether a feature regression in
> > distributions is acceptable.
> 
> Well, the way this is supposed to work with systemd is that NIS/LDAP
> becomes available with the nss-user-lookup.target target, and user
> logins wait for that (as well as remote-fs.target). NIS/LDAP only adds
> real (i.e. "human") users to the user database, and system users
> *must* exist all the time before that.
> 
> Both remote-fs.target and nss-user-lookup.target are *not* included in
> basic.target but are pulled in later.
> 
> > I think the best solution might be a way for "dbus-daemon --system" to
> > not be required to be Before early targets like sysinit.target (to avoid
> > circular dependencies), but for it to survive until the final killing
> > spree anyway. Does systemd have a way to say that, or are
> > startup/shutdown dependencies always symmetric?
> 
> Yes they are always symmetric.
> 
> dbus-daemon cannot really be started before basic.target
> 
> > Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
> > on this; one possible hack is to have dbus.service's ExecStop not
> > actually stop dbus-daemon, so that it stays around until the final
> > killing spree just before /usr and /var are unmounted.
> 
> Urks! Don't do that.
> 
> > I'm happy to modify dbus.service if that's (part or all of) the
> > solution, but before I can do that we need to come up with a solution
> > th

Re: [systemd-devel] Synchronization Between Services at Shutdown

[Lennart Poettering]

On Fri, 03.04.15 19:37, Andrei Borzenkov (arvidj...@gmail.com) wrote:

> В Fri, 3 Apr 2015 14:04:10 +0200
> Lennart Poettering  пишет:
> 
> > On Thu, 02.04.15 16:28, Martin Pitt (martin.p...@ubuntu.com) wrote:
> > 
> > > Lennart Poettering [2015-04-02 13:03 +0200]:
> > > > Yeah, and rightfully so. I mean, a service really should be able to
> > > > shutdown if dbus is dead. In fact, it should be able to shutdown in
> > > > pretty much any situation...
> > > 
> > > They do, but that's irrelevant here. The problem is that dbus.service
> > > shuts down very early as nothing prevents it from that. All
> > > dependencies just refer to dbus.socket, but TBH socket (re-)activation
> > > totally doesn't work for dbus.
> > 
> > systemd will not reactivate services when shutting down, we refuse to
> > go in circles...
> > 
> 
> Could you elaborate? systemd will obviously start services when
> starting shutdown.target. How does systemd differentiate between
> services that can and can not be started?

Socket units for which a stop job is queued will suppress
activations until the stop job is cancelled (ot it is stopped).

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel