Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction

2015-05-01 Thread Uoti Urpala
On Mon, 2015-04-27 at 18:07 +0200, Lennart Poettering wrote:
 On Wed, 04.02.15 23:48, Uoti Urpala (uoti.urp...@pp1.inet.fi) wrote:
  If you mean something like systemctl restart --no-block
  mydaemon-convert-config.service; systemctl reload mydaemon.service, I
  don't see why you'd ever /expect/ this to work with any reload semantics
  - isn't this clear user error, and will be racy with current systemd
  code just as much as the proposed fix? 
 
 Yupp, this is what I mean. (though I'd actually specify the --no-block
 in the second command too, though this doesn't make much of a
 difference...)


  And in any case I'd consider the semantics of reload to be switch
  to configuration equal or newer than what existed *when the reload
  was requested*, without any guarantees that changes from operations
  queued but not finished before calling reload would be taken into
  account.
 
 The queue is really a work queue, and the After= and Before= deps
 dictate how the work can be parallelized or needs to be serialized. As
 such if i have 5 jobs enqueued that depend on each other, i need to
 make sure they are executed in the order specified and can operateon
 the results of the previous job.
 
 I hope this makes sense...

After those clarifications I believe I now understand what kind of
example case you meant, and it does now seem a meaningful case to
consider; however, I still think that you're wrong, as your example case
turns out to work fine and is not actually a counterexample to the kind
of changes I was talking about.

If I understood correctly, you're talking about a case where service B
has After=A.service, both A and B have queued jobs where the B job is
a reload, and the queued job for A might change the configuration for B
(so the reload needs to happen after that); and you're worried that
immediately returning success for the reload could create a violation of
the after job A requirement. Is this reload property of After
documented anywhere? The code does seem to apply it to reloads, but
systemd.unit documentation only starts about start/stop. Anyway, when
you consider what actually happens with my suggested change, it turns
out that even these After semantics for reload still work.

The situation where my changes would result in different behavior is
when B has a start job queued, but no code for B is running yet, and you
request a reload for B; current code waits for the start of B before the
reload is considered complete, whereas my change makes the reload return
immediate success. This does not actually change the semantics above:
the only difference is when the reload operation is CONSIDERED COMPLETE,
there is NO difference in what operations are actually run or in which
order! [1] Current code merges RELOAD to existing START and returns
success for reload after START has completed, whereas my change returns
success immediately; but both run exactly the same START operation with
the same ordering constraints, which already ensure that it happens
after A.service (START already has the ordering constraints from
After=; merging the RELOAD to START does not add any additional
ordering that START would not already have had).

[1] So this difference only really matters when something blocks to wait
until the reload completes.


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-04-27 Thread Lennart Poettering
On Wed, 04.02.15 23:48, Uoti Urpala (uoti.urp...@pp1.inet.fi) wrote:

Sorry for the late reply,

 On Wed, 2015-02-04 at 21:57 +0100, Lennart Poettering wrote:
  OK, let's try this again, with an example:
  
  a) you have one service mydaemon.service
  
  b) you have a preparation service called
 mydaemon-convert-config.service that takes config from somewhere,
 converts it into a suitable format for mydaemon.service's binary
  
  Now, you change that config that is located somewhere, issue a restart
  request for m-c-c.s, issue a reload request for mydaemon.service.
  
  Now, something like this should always have the result that your
  config change is applied to mydaemon.service. Regardless if
  mydaemon.service's start was queued, is already started or is
  currently being started. You are suggesting that the reload can
  suppressed when a start is already enqueued, but that's really not the
  case, because you first have to run m-c-c.s, before you can reload...
 
 I do not see why that would cause any problems with removing the
 blocking.
 
 If you mean literally running systemctl restart
 mydaemon-convert-config.service; systemctl reload mydaemon.service then
 this should still work fine - the first restart will block until the
 operation is complete and new config exists, and then the reload
 guarantees that no old config is in use. 

No, the commands you suggest above don't just enqueue the operations,
they enqueue them and then wait for them to finish. Of course, if you
synchronously wait for them to finish then all races are gone, but
this really should work without that, so that things can be enqueued
and work correctly.

 However, I don't see why you'd
 include the part about creating the new configuration via
 mydaemon-convert-config.service in this case - the new configuration
 already exists before any reload functionality is invoked, so why the
 irrelevant complication of creating that configuration via another
 service? It seems you are implicitly assuming some kind of parallel
 execution of the restart and the reload?

Well, this is an example of the way people do this, and yes, i was
talking about enqueuing, and that really means just that: it won't
be the client anymore that controls execution order, but it is solely
the queue and its semantics that enforce how things are run and what
guarantees are made.

 If you mean something like systemctl restart --no-block
 mydaemon-convert-config.service; systemctl reload mydaemon.service, I
 don't see why you'd ever /expect/ this to work with any reload semantics
 - isn't this clear user error, and will be racy with current systemd
 code just as much as the proposed fix? 

Yupp, this is what I mean. (though I'd actually specify the --no-block
in the second command too, though this doesn't make much of a
difference...)

I am pretty sure that enqueueing these two commands should be sufficient
to get the config that was written out by the first service to be in
effect in the second service.

 There are no ordering constraints
 here, any more than there would be about starting two services and
 expecting the first request to be started first. 

hmm? if you start two services, and they are ordered against each
other, then yes, the second service should only be started after the
first one completed startup.

 And in any case I'd consider the semantics of reload to be switch
 to configuration equal or newer than what existed *when the reload
 was requested*, without any guarantees that changes from operations
 queued but not finished before calling reload would be taken into
 account.

The queue is really a work queue, and the After= and Before= deps
dictate how the work can be parallelized or needs to be serialized. As
such if i have 5 jobs enqueued that depend on each other, i need to
make sure they are executed in the order specified and can operateon
the results of the previous job.

I hope this makes sense...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction

2015-03-10 Thread Uoti Urpala
On Wed, 2015-02-04 at 23:48 +0200, Uoti Urpala wrote:
 On Wed, 2015-02-04 at 21:57 +0100, Lennart Poettering wrote:
  currently being started. You are suggesting that the reload can
  suppressed when a start is already enqueued, but that's really not the
  case, because you first have to run m-c-c.s, before you can reload...

 If you mean literally running systemctl restart

...

 So unless I completely misunderstood your example, it seems that this
 does NOT demonstrate any problems with removing the blocking.


Discussion seems to have died again. How to proceed with fixing this? Is
there anything more I can clarify about why the current behavior (that
is, when service startup is queued, have reload requests block until
the service is up) is wrong and why the fix would be valid?


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-02-04 Thread Martin Pitt
Lennart Poettering [2015-02-04 16:38 +0100]:
 Sure, I can only recommend again: in the the glue code that calls out
 to systemctl from service, you can add the code to use --no-block
 or --job-mode=ignore-dependencies , if you notice you are in shutdown
 mode...

Yeah, I agree that given all the options that's the best heuristics
for now.

   Modern code, that talks directly to systemctl, that uses hook scripts,
   should always use --no-block for these cases.
  
  Yeah, for sure. We just don't have that luxury easily, as first these
  scripts don't call systemctl but service (for the Debianists: or
  invoke-rc.d, but same difference), it will take some time to find
  all these occurrences, and people will hate us for having to add stuff
  like
  
if (running under systemd)
 systemctl --no-block reload foo
else
 service foo reload
 
 I am not proposing anything like that.

Right, this was just a reply to modern code that talks directly to
systemctl, and we can't do that while we support more than one init
system. Sorry for the misunderstanding.

 - Don't enqueue a reload if the service to be reloaded isn't running.
   E. g. postfix.service inactive/dead in
   https://bugs.debian.org/635777 or smbd.service start/waiting in
   https://launchpad.net/bugs/1417010.  This would completely avoid
   the deadlock in most situations already, and doesn't change the
   semantics for working use cases either, so this should even be
   applicable for upstream?

For the record, this was already discussed here:
http://lists.freedesktop.org/archives/systemd-devel/2014-July/021457.html
continuation:
http://lists.freedesktop.org/archives/systemd-devel/2014-August/022048.html

 I mean, if you want precise sysv semantics you can just use
 --job-mode=ignore-dependencies always, since sysv ignore all deps too
 when sysv scripts are involved.

Always sounds rather unsafe to me, as we would totally ignore a
unit's Requires/Wants= then. Doing it if !systemctl is-system-running
sounds ok.

 I'd strongly encourage you to work around this on client side in the
 sysv glue, not breaking the guarantess that systemd-aware code needs
 when issuing commands.

Yes, I prefer that as well. (That's what I meant with distro side..)

Thanks,

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-02-04 Thread Uoti Urpala
On Wed, 2015-02-04 at 19:36 +0100, Lennart Poettering wrote:
 On Wed, 04.02.15 20:19, Uoti Urpala (uoti.urp...@pp1.inet.fi) wrote:
  You're missing an essential point here: there's a distinction between
  skipping reloads for services which have not not been dispatched, and
  skipping reloads for services for which startup code is already running
  (and may be using existing configuration) but which have not reached
  full running status yet.
  
  The former is the correct behavior (but currently handled wrong by
  systemd!), and never causes races. Only the latter can cause races like
  described above.
 
 These two cases aren't that different. If somebody pushes an
 additional job into the queue that wants to run before the reload but
 after the service is up you cannot ot flush out the reload just
 because the service has not started yet...

I cannot parse what you're trying to say here, if it's anything
meaningful. Your wants to run before the reload sounds like you're
talking about guaranteeing that a reload NOT happen before something
else runs, but that would be nonsense - the guarantee would guarantee
nothing semantically relevant (if systemd only starts executing the
service binary *after* the reload has been queued, it cannot use any
pre-reload-order config at any point; there's no guaranteed to use old
config guarantee of any form possible!).


 Whether you change config in your current context, or you do so from a
 new unit's context is no difference: we cannot move anything that is
 supposed to happen after that change before it, and we cannot remove it
 either...

If no code from a service is currently running, it's already guaranteed
that every request issued to the service in the future will use the new
config (no old state exists, and any newly started process will
obviously load the new config). Thus the requirements for a reload are
already fulfilled; the operation is complete, and there is nothing more
to do. Unnecessary waiting only causes deadlocks for no benefit
whatsoever.

 There are some forms of coalescing possible, but we already do all of
 the ones that are safe...

This is not exactly coalescing - it's just immediately returning
success if there is no service code running (either in running state
or in startup state where a process already exists and could have read
the old config before it was changed).

Removing the current incorrect blocking and returning success
immediately is 100% safe, in the following strictly defined sense:
All requests handled by the service after systemctl reload has
returned will use a version of config equal or newer than the one that
was in effect when the reload call was started.

If you still want claim that removing the blocking would not be safe,
please try to construct a sequence of operations where such non-blocking
behavior would lead to failure (failure defined as: the service
processes a request using configuration older than what existed when
reload was requested). I'm confident that it is impossible to
construct such a counterexample.


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-02-04 Thread Lennart Poettering
On Wed, 04.02.15 20:19, Uoti Urpala (uoti.urp...@pp1.inet.fi) wrote:

 On Wed, 2015-02-04 at 16:38 +0100, Lennart Poettering wrote:
  On Wed, 04.02.15 15:25, Martin Pitt (martin.p...@ubuntu.com) wrote:
   Lennart Poettering [2015-02-04 13:27 +0100]:
On Wed, 04.02.15 08:56, Martin Pitt (martin.p...@ubuntu.com) wrote:
  - Don't enqueue a reload if the service to be reloaded isn't running.
E. g. postfix.service inactive/dead in
https://bugs.debian.org/635777 or smbd.service start/waiting in
https://launchpad.net/bugs/1417010.  This would completely avoid
the deadlock in most situations already, and doesn't change the
semantics for working use cases either, so this should even be
applicable for upstream?

No, this would open up the door for races. The guarantee we give
around blocking operations, is that by the time they return the
operation or an equivalent has been executed. More specifically, if
you start a service, and it is in starting, and then issue a
reload or restart, and it returns you *know* that the
configuration that was on disk at the time you issued the reload or
restart -- or a newer one -- is in place. If you'd suppress the
reload/restart in this case, then you will not get that guarantee,
because the configuration ultimately loaded might be the one from the
time the daemon was first put into starting mode at.
 
 You're missing an essential point here: there's a distinction between
 skipping reloads for services which have not not been dispatched, and
 skipping reloads for services for which startup code is already running
 (and may be using existing configuration) but which have not reached
 full running status yet.
 
 The former is the correct behavior (but currently handled wrong by
 systemd!), and never causes races. Only the latter can cause races like
 described above.

These two cases aren't that different. If somebody pushes an
additional job into the queue that wants to run before the reload but
after the service is up you cannot ot flush out the reload just
because the service has not started yet... 

Whether you change config in your current context, or you do so from a
new unit's context is no difference: we cannot move anything that is
supposed to happen after that change before it, and we cannot remove it
either...

There are some forms of coalescing possible, but we already do all of
the ones that are safe...

 Fixing the systemd semantics should fix most of the bootup deadlock
 cases. This is not a sysv workaround or anything like that. The
 current systemd semantics are wrong and undesirable for new code,
 regardless of any legacy compatibility issues. Fixing them would give
 semantics that are more logically correct and work better in
 practice.

No, totally not. THe current semantics give the necessary guarantees
that changing a config file from any context you like or queing a file
config change from any config you like, and then queuing a reload will
take effect, regardless if there's a job for the unit already queued,
running or anything else.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-02-04 Thread Lennart Poettering
On Wed, 04.02.15 22:10, Uoti Urpala (uoti.urp...@pp1.inet.fi) wrote:

 On Wed, 2015-02-04 at 19:36 +0100, Lennart Poettering wrote:
  On Wed, 04.02.15 20:19, Uoti Urpala (uoti.urp...@pp1.inet.fi) wrote:
   You're missing an essential point here: there's a distinction between
   skipping reloads for services which have not not been dispatched, and
   skipping reloads for services for which startup code is already running
   (and may be using existing configuration) but which have not reached
   full running status yet.
   
   The former is the correct behavior (but currently handled wrong by
   systemd!), and never causes races. Only the latter can cause races like
   described above.
  
  These two cases aren't that different. If somebody pushes an
  additional job into the queue that wants to run before the reload but
  after the service is up you cannot ot flush out the reload just
  because the service has not started yet...
 
 I cannot parse what you're trying to say here, if it's anything
 meaningful. 

No, usually what I am babbling is not meaningful at all...

 Your wants to run before the reload sounds like you're
 talking about guaranteeing that a reload NOT happen before something
 else runs, but that would be nonsense - the guarantee would guarantee
 nothing semantically relevant (if systemd only starts executing the
 service binary *after* the reload has been queued, it cannot use any
 pre-reload-order config at any point; there's no guaranteed to use old
 config guarantee of any form possible!).

OK, let's try this again, with an example:

a) you have one service mydaemon.service

b) you have a preparation service called
   mydaemon-convert-config.service that takes config from somewhere,
   converts it into a suitable format for mydaemon.service's binary

Now, you change that config that is located somewhere, issue a restart
request for m-c-c.s, issue a reload request for mydaemon.service.

Now, something like this should always have the result that your
config change is applied to mydaemon.service. Regardless if
mydaemon.service's start was queued, is already started or is
currently being started. You are suggesting that the reload can
suppressed when a start is already enqueued, but that's really not the
case, because you first have to run m-c-c.s, before you can reload...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-02-04 Thread Uoti Urpala
On Wed, 2015-02-04 at 21:57 +0100, Lennart Poettering wrote:
 OK, let's try this again, with an example:
 
 a) you have one service mydaemon.service
 
 b) you have a preparation service called
mydaemon-convert-config.service that takes config from somewhere,
converts it into a suitable format for mydaemon.service's binary
 
 Now, you change that config that is located somewhere, issue a restart
 request for m-c-c.s, issue a reload request for mydaemon.service.
 
 Now, something like this should always have the result that your
 config change is applied to mydaemon.service. Regardless if
 mydaemon.service's start was queued, is already started or is
 currently being started. You are suggesting that the reload can
 suppressed when a start is already enqueued, but that's really not the
 case, because you first have to run m-c-c.s, before you can reload...

I do not see why that would cause any problems with removing the
blocking.

If you mean literally running systemctl restart
mydaemon-convert-config.service; systemctl reload mydaemon.service then
this should still work fine - the first restart will block until the
operation is complete and new config exists, and then the reload
guarantees that no old config is in use. However, I don't see why you'd
include the part about creating the new configuration via
mydaemon-convert-config.service in this case - the new configuration
already exists before any reload functionality is invoked, so why the
irrelevant complication of creating that configuration via another
service? It seems you are implicitly assuming some kind of parallel
execution of the restart and the reload?

If you mean something like systemctl restart --no-block
mydaemon-convert-config.service; systemctl reload mydaemon.service, I
don't see why you'd ever /expect/ this to work with any reload semantics
- isn't this clear user error, and will be racy with current systemd
code just as much as the proposed fix? There are no ordering constraints
here, any more than there would be about starting two services and
expecting the first request to be started first. And in any case I'd
consider the semantics of reload to be switch to configuration equal or
newer than what existed *when the reload was requested*, without any
guarantees that changes from operations queued but not finished before
calling reload would be taken into account.

So unless I completely misunderstood your example, it seems that this
does NOT demonstrate any problems with removing the blocking.


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-02-04 Thread Lennart Poettering
On Wed, 04.02.15 08:56, Martin Pitt (martin.p...@ubuntu.com) wrote:

 Lennart Poettering [2015-02-03 21:40 +0100]:
  It's really about synchronous waiting on jobs. If you synchronously
  wait for completion of jobs that are ordered against the job your are
  part of yourself, then things will deadlock.
 
 Indeed. The problem is that if you reload e. g. postfix from a DHCP or
 network up/down hook, such a script doesn't have the slightest idea
 whether it was run because the network changed at runtime (i. e. udev
 event or the user just selected a new network) or whether it happens
 as part of a systemd transaction (boot/shutdown). In the former case
 you do want to block, in the latter case you mustn't.

I don't see why you'd want to block in either case. Networking is
asynchronous anyway, there's really no point in blocking here. What
does this give you?

Modern code, that talks directly to systemctl, that uses hook scripts,
should always use --no-block for these cases.

That said, non-crap code does not rely on hook scripts like this
anyway, and just subscribes to rtnl on its own.

  Now, regardless which option you choose it's always a good idea to
  keep this change as local as possible. Altering the state engine for
  all operations is the worst solution...
 
 Well, it's a problem which can happen in a lot of scenarios and isn't
 specific to which kind of service or hook script you have, so what's
 local is actually quite hard to define here.
 
 I agree with Michael that involving a lot of shell commands which we
 then have to copy to lots of places (and find these places at all) is
 also not the best solution. So perhaps we could have some middle
 ground here and make systemctl a bit more clever?

Hmm? I don't follow here at all?

I mean, you must have some code in Debian that forwards old sysv
restart/reload requests to systemctl. That's probably going to be in
/usr/sbin/service and maybe a few other places. Why can't you just add
the --no-block logic there, conditionalized to shutdown/reboot?

  - Don't enqueue a reload if the service to be reloaded isn't running.
E. g. postfix.service inactive/dead in
https://bugs.debian.org/635777 or smbd.service start/waiting in
https://launchpad.net/bugs/1417010.  This would completely avoid
the deadlock in most situations already, and doesn't change the
semantics for working use cases either, so this should even be
applicable for upstream?

No, this would open up the door for races. The guarantee we give
around blocking operations, is that by the time they return the
operation or an equivalent has been executed. More specifically, if
you start a service, and it is in starting, and then issue a
reload or restart, and it returns you *know* that the
configuration that was on disk at the time you issued the reload or
restart -- or a newer one -- is in place. If you'd suppress the
reload/restart in this case, then you will not get that guarantee,
because the configuration ultimately loaded might be the one from the
time the daemon was first put into starting mode at.

Or in other words: if a script does this:

   # sed -i -e 's/something/somethingelse/' /etc/mydaemon.conf
   # systemctl restart mydaemon.service

Then we *must* give the guarantee that when the second command returns
the change just made to /etc/mydaemon.conf is in place, and not a
previous config loaded. And we must guarantee that in any case,
regardless if the daemon is about to start, already started, reloaded
or anything else...

(Now, this example is about restarts, but reload is pretty much the
same, except that many people use async ExecReload= commands (for
example send a signal), which will break this guarantee, but that's on
them then, we recommend synchronous reload operations, to make this
race-free).

 
 And/or
 
  - systemctl reload/restart could imply --no-block if the service is
already enqueued in the current transaction. That would avoid this
deadlock situation in more cases.

Opens the same race, as described above.

Also: *why* for heaven's sake? Just add --no-block when you are
running from these contexts, and all is good. I really don't get why
you don't want to do that.

 With that the remaining deadlock case would be trying to reload an
 already running service which isn't affected by the current
 transaction, but we haven't seen that in practice yet.
 
 If you don't want this upstream, I'd keep it as a patch in Debian. But
 I can't really imagine that this wouldn't happen in Fedora or other
 distros? I mean, things like the ISC DHCP hooks aren't a Debianism,
 and a lot of existing software wasn't written with this be careful on
 service reloads and guess whether you need --no-block approach in
 mind, as it has never been a problem with other init systems.

On Fedora, these hook scripts use --no-block correctly, and I am
completely puzzled why you don't want this on Debian!

Please, can you elaborate what your issue with --no-block is?

Lennart

Re: [systemd-devel] Deadlocks with reloading jobs which are part of current transaction [was: [PATCH] Avoid reloading services when shutting down]

2015-02-04 Thread Martin Pitt
Lennart Poettering [2015-02-04 13:27 +0100]:
 On Wed, 04.02.15 08:56, Martin Pitt (martin.p...@ubuntu.com) wrote:
 
  Lennart Poettering [2015-02-03 21:40 +0100]:
   It's really about synchronous waiting on jobs. If you synchronously
   wait for completion of jobs that are ordered against the job your are
   part of yourself, then things will deadlock.
  
  Indeed. The problem is that if you reload e. g. postfix from a DHCP or
  network up/down hook, such a script doesn't have the slightest idea
  whether it was run because the network changed at runtime (i. e. udev
  event or the user just selected a new network) or whether it happens
  as part of a systemd transaction (boot/shutdown). In the former case
  you do want to block, in the latter case you mustn't.
 
 I don't see why you'd want to block in either case. Networking is
 asynchronous anyway, there's really no point in blocking here. What
 does this give you?

So far, if a hook or shell script calls e. g. service foo restart,
it can count on the service actually being reloaded after it finishes,
and thus you can then interact with it immediately. That's not
important for many scripts, but we can't just always make service foo
reload async by using --no-block, as that would break compatibility
with scripts which do rely on that behaviour. So we do need to
conditionalize it to avoid the deadlocks under systemd.

 Modern code, that talks directly to systemctl, that uses hook scripts,
 should always use --no-block for these cases.

Yeah, for sure. We just don't have that luxury easily, as first these
scripts don't call systemctl but service (for the Debianists: or
invoke-rc.d, but same difference), it will take some time to find
all these occurrences, and people will hate us for having to add stuff
like

  if (running under systemd)
   systemctl --no-block reload foo
  else
   service foo reload

as that special case appears rather pointless for someone who hasn't
dealt with the details of systemd job transactions. That's not to say
that it shouldn't happen (perhaps adding a --no-block option to
service), but that takes time.

 I mean, you must have some code in Debian that forwards old sysv
 restart/reload requests to systemctl. That's probably going to be in
 /usr/sbin/service and maybe a few other places. Why can't you just add
 the --no-block logic there, conditionalized to shutdown/reboot?

Well, we can; it's just as much an unreliable hack as the two existing
patches :-) and I was wondering if we can't solve this in a more
generic/distro agnostic way.

   - Don't enqueue a reload if the service to be reloaded isn't running.
 E. g. postfix.service inactive/dead in
 https://bugs.debian.org/635777 or smbd.service start/waiting in
 https://launchpad.net/bugs/1417010.  This would completely avoid
 the deadlock in most situations already, and doesn't change the
 semantics for working use cases either, so this should even be
 applicable for upstream?
 
 No, this would open up the door for races. The guarantee we give
 around blocking operations, is that by the time they return the
 operation or an equivalent has been executed. More specifically, if
 you start a service, and it is in starting, and then issue a
 reload or restart, and it returns you *know* that the
 configuration that was on disk at the time you issued the reload or
 restart -- or a newer one -- is in place. If you'd suppress the
 reload/restart in this case, then you will not get that guarantee,
 because the configuration ultimately loaded might be the one from the
 time the daemon was first put into starting mode at.

Hm, I don't quite understand this. If you reload a service which isn't
currently running, systemctl will fail. It's just currently going to
enqueue the reload request, and executing the job will fail due to the
not active check in unit_reload(). The problem with that is just
that systemctl doesn't fail immediately with the unit is not active,
but blocks on the queued request. So if you don't have pending jobs,
the net result is the same, but with pending jobs you can easily get a
deadlock.

 Or in other words: if a script does this:
 
# sed -i -e 's/something/somethingelse/' /etc/mydaemon.conf
# systemctl restart mydaemon.service
 
 Then we *must* give the guarantee that when the second command returns
 the change just made to /etc/mydaemon.conf is in place, and not a
 previous config loaded.

Yes, that's fine. I only said reload above, as that's what
DHCP/network scripts usually do.

 (Now, this example is about restarts, but reload is pretty much the
 same

reload on an inactive unit will fail, restart will start it.

 Also: *why* for heaven's sake? Just add --no-block when you are
 running from these contexts, and all is good. I really don't get why
 you don't want to do that.

Well, I do, but special-casing that on boot/shutdown isn't enough (you
can have other bigger transactions which lock up); and it's rather
expensive to do