Bug#733452: init system daemon readiness protocol

[Tollef Fog Heen]

]] Ian Jackson 

> (Sorry, 2nd copy here because I missed up the change of To field in
> the previous one.)
> 
> cameron writes ("Re: Bug#733452: init system daemon readiness protocol"):
> > I was curious: why should SOCK_STREAM be used instead of SOCK_DGRAM in 
> > your proposed protocol?
> 
> SOCK_DGRAM sockets do not offer reliable delivery (at least, not on
> all unices).
> 
> > Have you seen Lennart Poettering's pastebin of a short daemon side 
> > implementation of that protocol: http://fpaste.org/64821/32737713/? It 
> > meets all your desired criteria, it is used in one init system already, 
> > and it is very extensible. Now that you know that systemd does not 
> > actually use SOCK_SEQPACKET, but SOCK_DGRAM, do you have any changes in 
> > opinion of the systemd approach?
> 
> I still think it would be simpler to pass the ready-connected socket
> (or whatever) to the daemon by inheritance, rather than having the
> daemon call socket() etc.
> 
> Do you know why in systemd it was done the way it was ?

Passing a file descriptor around reliably so it only ends up in the
right place is harder than doing the same with an environment variable.
Many daemons close all fds as part of the startup (this is documented as
best practice in quite a few places).

You also need to ensure that the fd given in the environment variable is
actually a valid file descriptior.  As per Lennart when asked on IRC:

17:16 < poettering> Mithrandir: anyway, the one line summary is that i
wanted this to be as simple as possible: it should suffice to place a
single sd_notify() at the appropriate place to make this work. With fd
passing that would not work, as you first have to make sure the passed
fd is not closed as part of setting up the execution environment during
daemon initialization.
17:17 < poettering> Mithrandir: sd_notify() as it stands now is really
really isolated. As long as the env var is there it doesn't need
anything else
17:17 < poettering> Mithrandir: what ian wants otoh is not so isolated,
you need to have a look at the entire daemon initialization and make
sure the fd is never closed or handled in its process, and then you
*also* need to look at the env var for it
17:17 < poettering> hope that makes sense

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


-- 
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/m2y530f6r4@rahvafeir.err.no

Bug#733452: init system daemon readiness protocol

[cameron]

On Mon, Dec 30, 2013 at 9:38 AM, Ian Jackson 
 wrote:

(Sorry, 2nd copy here because I missed up the change of To field in
the previous one.)

cameron writes ("Re: Bug#733452: init system daemon readiness 
protocol"):
 I was curious: why should SOCK_STREAM be used instead of SOCK_DGRAM 
in 
 your proposed protocol?



SOCK_DGRAM sockets do not offer reliable delivery (at least, not on
all unices).



I am pretty sure they are reliable for //local// sockets, at least on 
Linux.
see this reddit comment: 
http://www.reddit.com/r/linux/comments/1tya0c/lennarts_take_on_the_proposed_debian_daemon/cecstgq


 Have you seen Lennart Poettering's pastebin of a short daemon side 
 implementation of that protocol: http://fpaste.org/64821/32737713/? 
It 
 meets all your desired criteria, it is used in one init system 
already, 
 and it is very extensible. Now that you know that systemd does not 
 actually use SOCK_SEQPACKET, but SOCK_DGRAM, do you have any 
changes in 
 opinion of the systemd approach?



I still think it would be simpler to pass the ready-connected socket
(or whatever) to the daemon by inheritance, rather than having the
daemon call socket() etc.

Do you know why in systemd it was done the way it was ?



Yes, here are Lennart's words:

>We use SOCK_DGRAM because we are interested in the message boundary 
and to get SCM_CREDENTIALS attached to each datagram by the kernel. 
Note that systemd only has a single notification socket set up for all 
the services it starts. All service hence queue their messages into the 
same socket, and we need to be able to identify exactly from which 
process each message originated, and need to make sure that the 
boundaries are intact and not messages from one service are half 
written and then mixed with messages from other services which write 
inbetween. By using SOCK_DGRAM we can be sure that each datagram is 
either fully written or never fully written, but never half-written 
interleaved with another half message from somebody else. And the 
kernel implicitly attaches SCM_CREDENTIALS to each of these datagrams, 
but this does not translate to SOCK_STREAM.



Thanks,
Ian.


Bravo,
Cameron Norman

Bug#733452: init system daemon readiness protocol

[Ian Jackson]

(Sorry, 2nd copy here because I missed up the change of To field in
the previous one.)

cameron writes ("Re: Bug#733452: init system daemon readiness protocol"):
> I was curious: why should SOCK_STREAM be used instead of SOCK_DGRAM in 
> your proposed protocol?

SOCK_DGRAM sockets do not offer reliable delivery (at least, not on
all unices).

> Have you seen Lennart Poettering's pastebin of a short daemon side 
> implementation of that protocol: http://fpaste.org/64821/32737713/? It 
> meets all your desired criteria, it is used in one init system already, 
> and it is very extensible. Now that you know that systemd does not 
> actually use SOCK_SEQPACKET, but SOCK_DGRAM, do you have any changes in 
> opinion of the systemd approach?

I still think it would be simpler to pass the ready-connected socket
(or whatever) to the daemon by inheritance, rather than having the
daemon call socket() etc.

Do you know why in systemd it was done the way it was ?

Thanks,
Ian.


-- 
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/21185.44921.223653.273...@chiark.greenend.org.uk

Bug#733452: init system daemon readiness protocol

[Ian Jackson]

Tollef Fog Heen writes ("Bug#733452: init system daemon readiness protocol"):
> Ian Jackson:
> > I conclude therefore that we should design another simple protocol -
> > preferably, a variation on one of the existing ones - and have (at
> > least) both Debian's systemd and Debian's upstart implement it.
> 
> I think you're into ever-multiplying power socket standards territory
> here.  I am not going to carry patches in systemd in Debian for a
> Debian-only notification protocol because you don't want to use the
> upstream protocol.

I would like an easier protocol to be supported by upstream too.
Perhaps that wasn't clear from my message.  I would certainly prefer
to avoid some Debian-specific protocol.

Later:
> It seems Lennart read the proposal and responded in
> https://plus.google.com/+LennartPoetteringTheOneAndOnly/posts/37AWJLE3XcJ

Nevertheless, even if SCM_CREDENTIALS is not required by systemd, my
proposed protocol is still simpler than the systemd one.

Lennart doesn't seem to have addressed this point.

Ian.


-- 
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/21185.42952.83261.226...@chiark.greenend.org.uk

Re: Bug#733452: init system daemon readiness protocol

[cameron]

Hello Mr. Jackson,

I was curious: why should SOCK_STREAM be used instead of SOCK_DGRAM in 
your proposed protocol?


Have you seen Lennart Poettering's pastebin of a short daemon side 
implementation of that protocol: http://fpaste.org/64821/32737713/? It 
meets all your desired criteria, it is used in one init system already, 
and it is very extensible. Now that you know that systemd does not 
actually use SOCK_SEQPACKET, but SOCK_DGRAM, do you have any changes in 
opinion of the systemd approach?


Thank you for your time,

Cameron Norman

Bug#733452: init system daemon readiness protocol

[Tollef Fog Heen]

]] Tollef Fog Heen 

> ]] Ian Jackson 
> 
> > I conclude therefore that we should design another simple protocol -
> > preferably, a variation on one of the existing ones - and have (at
> > least) both Debian's systemd and Debian's upstart implement it.
> 
> I think you're into ever-multiplying power socket standards territory
> here.  I am not going to carry patches in systemd in Debian for a
> Debian-only notification protocol because you don't want to use the
> upstream protocol.
> 
> As I've said in other messages, feel free to talk to upstream, but I'm
> not going to pass on suggestions which I'm not going to support.

It seems Lennart read the proposal and responded in
https://plus.google.com/+LennartPoetteringTheOneAndOnly/posts/37AWJLE3XcJ

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


-- 
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/m27ganlj2k@rahvafeir.err.no

Bug#733452: init system daemon readiness protocol

[Tollef Fog Heen]

]] Ian Jackson 

> I conclude therefore that we should design another simple protocol -
> preferably, a variation on one of the existing ones - and have (at
> least) both Debian's systemd and Debian's upstart implement it.

I think you're into ever-multiplying power socket standards territory
here.  I am not going to carry patches in systemd in Debian for a
Debian-only notification protocol because you don't want to use the
upstream protocol.

As I've said in other messages, feel free to talk to upstream, but I'm
not going to pass on suggestions which I'm not going to support.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


-- 
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/m2ppogkpdr@rahvafeir.err.no

Bug#733452: init system daemon readiness protocol

[Nikolaus Rath]

Ian Jackson  writes:
> (I have cloned the bug for this, to keep this particular
> sub-discussion separable.)
>
> As I have reported, we have a problem with non-forking daemon
> readiness protocols.

"We have a problem" seems a bit exxagerated to me. So far, the only
problem that I have seen is that you don't like the systemd protocol,
(and there's nothing wrong with that, I even agree to some extent). But
does that translate to a general problem with readiness protocols in
Debian? In other words, is there a significant number of important
packages where neither upstream nor the Debian maintainer is willing to
tolerate systemd's protocol, and that cannot use socket activation
either?

[...]
> I conclude therefore that we should design another simple protocol -
> preferably, a variation on one of the existing ones - and have (at
> least) both Debian's systemd and Debian's upstart implement it.

Could you elaborate a bit on the advantages of this proposal for Debian?
(Maybe I don't see them because I don't see the general problem that
you're trying to solve in the first place).

I think that most likely this standard wouldn't be used by anyone other
than Debian, so every daemon needs a Debian specific patch to support
it.


Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«


--
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ppogfsce@vostro.rath.org

Bug#733452: init system daemon readiness protocol

[Russ Allbery]

Ian Jackson  writes:

> The systemd protocol is troublesome because it requires too much code
> in the daemon, leaving the daemon author with the trilemma which has
> previously been discussed.

This statement as written is only true if you're unwilling to link with a
shared library, a stance that I personally do not understand.  I do
realize that's the position you hold, and you're entitled to your opinions
on that front, but I think it's important to include that disclaimer here.

When linking with libsystemd-daemon, which is a tiny shared library that
adds no particular difficulties and that can be trivially stubbed out on
systems that don't have it, adding systemd daemon readiness to my package
required eight lines of upstream code (three for the actual call, five to
stub the call out if the library was not found).  The build system support
for optionally compiling with the appropriate libraries was comparable:
six lines of code, two variables added to CPPFLAGS and LDADD lines in
Makefile.am, and an autogen-time dependency on pkg-config (which many
packages are already using for other reasons).  So a total of 14 lines of
code, which I certainly didn't find to be "too much."

I think one's position on this depends heavily on whether one considers
optionally linking with a shared library to provide systemd integration
and not providing that integration if the library was not available at
build time to be reasonable.  Personally, I find that entirely reasonable,
and therefore cannot agree with your characterization of the systemd
situation.

> I conclude therefore that we should design another simple protocol -
> preferably, a variation on one of the existing ones - and have (at
> least) both Debian's systemd and Debian's upstart implement it.
> Obviously this needs input from both upstreams.  Given the poor
> relationship between the two main projects which would need to implement
> the same protocol, and the possibility that we might have to carry this
> in a local patch in Debian if we can't reach agreement with one or other
> of the upstreams, I think it would be best if this design discussion
> were refereed by the TC.

Despite the fact that I personally have no problems with the existing
systemd notification protocol, I would certainly welcome a compromise that
more people found reasonable.  I'm a bit skeptical that such a thing would
happen, but if people would like to work on it, by all means.

However, I don't think this fits the role of the TC, and indeed I think
refereeing that discussion would be contrary to my view of section 6.3.5
of the Debian constitution.

> Also, though should not block the decision on the default init system
> on this more open-ended design work (and associated negotation); but
> it is probably worth waiting with starting a mass package conversion
> until we know what startup protocol we want.

Agreed, with the caveat that I don't think we should discourage upstreams
from adding support for one or both of the synchronization protocols.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ha9sr1d1@windlord.stanford.edu

Bug#733452: init system daemon readiness protocol

[Ian Jackson]

(I have cloned the bug for this, to keep this particular
sub-discussion separable.)

As I have reported, we have a problem with non-forking daemon
readiness protocols.

There are two competing protocols that I'm aware of (not counting the
protocol implied by daemon(3)): upstart's, where the daemon raises
SIGSTOP, and systemd's where the daemon connects to an AF_UNIX socket
(whose name is found in an environment variable) and uses sendmsg to
send a message with SCM_CREDENTIALS.

(I checked djb's daemontools and it doesn't seem to address this
problem.  If there is another existing approach please would someone
let me know.)

The systemd protocol is troublesome because it requires too much code
in the daemon, leaving the daemon author with the trilemma which has
previously been discussed.  The upstart protocol is "elegant" or
"horrible" depending who you listen to; and has the minor practical
problem that it can make sending stop signals to starting daemons
ineffective or damaging.

The upstart protocol could be somewhat improved by the use of SIGTTOU
rather than SIGSTOP, but the result still doesn't provide 100%
reliability for an administrator's use of SIGSTOP (a SIGSTOP sent
between the daemon raising TTOU and being sent CONT by the supervisor
will be lost).  And anyway that doesn't answer the key aesthetic
objection that signalling readiness by stopping is perverse.  (I
certainly do not discount this objection simply because it's
aesthetic, even though as it happens I don't share it.)

I conclude therefore that we should design another simple protocol -
preferably, a variation on one of the existing ones - and have (at
least) both Debian's systemd and Debian's upstart implement it.
Obviously this needs input from both upstreams.  Given the poor
relationship between the two main projects which would need to
implement the same protocol, and the possibility that we might have to
carry this in a local patch in Debian if we can't reach agreement with
one or other of the upstreams, I think it would be best if this design
discussion were refereed by the TC.

Also, though should not block the decision on the default init system
on this more open-ended design work (and associated negotation); but
it is probably worth waiting with starting a mass package conversion
until we know what startup protocol we want.


Onto the concrete question, it seems to me that the requirements for a
protocol that everyone should be able to accept include:
 - it should be implementable on any reasonable unix
 - it should have no (not just minor) undesired side effects
 - it should require minimal code in the daemon
 - it should not introduce new dependencies

Furthermore, I would say that:
 - the indication to the daemon that it is to use the new protocol
   should be providable either by command-line option or environment
   variable - at the option of the daemon author; if it is done by
   environment variable this should happen only if needed by the
   specific daemon, and ideally by an environment variable specific to
   the target program (to reduce the chances of the daemon's
   descendants seeing and trying to honour it);
 - the choice of fd to use should not be "baked in" to the protocol,
   to make the protocol the easiest fit with other software.

I think there is only one realistic possible basic structure for such
a protocol: the supervisor passes a fd to the daemon by inheritance;
when the daemon is ready it calls write(2) to write a fixed string to
the socket.

The systemd approach of using a SOCK_SEQPACKET socket is attractive,
but unfortunately I don't think it's suitable because: many people are
unfamiliar with SOCK_SEQPACKET sockets (and we want a protocol which
daemon authors will be confident that they have implemented
correctly); it is difficult to debug with ordinary utilities (so a
daemon author can't check their implementation); and I have heard that
some kernels have idiosyncracies in their handling of these sockets.

However, we don't need to extend this protocol to continue throughout
the life of the daemon - if we wanted the kind of facilities provided
by the systemd messaging protocol, the extra library dependency is
tolerable so we could just use the systemd protocol in those cases.
Our new protocol is just for the very basic case.  So we can just
indicate the end of the message by closing the socket.


I therefore propose the following specification:

 * This protocol is called "simple non-forking daemon readiness
   indication protocol" or "readiness protocol" for short.

 * A service supervisor which implements the readiness protocol:

- MUST permit its configuration to specify command line arguments
  and environment variables to the daemon;

- MUST NOT pass any startup protocol related environment variables
  unless explicitly configured to do so as just described;

- MUST permit its configuration to specify the use of the
  readiness protocol

- The fd number to be used