Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 21.04.2017 21:31, k...@aspodata.se wrote: > a, If you have that impression, then why daemonize() which sounds as >the same thing ? > > b, No it isn't. That's why the -f, which is nice to have when debugging. If you dont want the service to daemonize itself, a debug flag doesn't seem the correct approach, just a workaround. So they should at least implemented some --foreground option that just prevents daemonizing, but doesn't enable extra logging. And here we are - lots of code duplication in all the individual daemons. And operators have to look up how to actually call it in the individual case. We could consolidate that into exactly one place. (actually, I'd prefer services never daemonize themselves and always leave that to a separate program of operators choice) >> That '-f' switch would be in >> the lib in a different form, eg. via env variables. > > Yes, it's messier to pick just the -f from argc/argv in a lib than > calling getenv(). You're assuming that everybody implements an '-f' switch, and it's always exactly '-f' - that flag could mean something completely different (perhaps f for file). Again and again, operators have to look up the manpage and be careful not to mix things up. (or even patch up services that don't even have that flag yet) With that library function, we have all of that consolidated in one place and don't even have to change any command line syntax anymore. >> My goal here is not voting for any particular way of daemonizing, >> service monitoring, etc, but just to provide an abstraction layer, so >> all that stuff can be moved out of the individual application. > ... > > Well, that's fine, but replaceing -f and daemon() with your entry > doesn't seem worth it. Maybe it's better to write up some > "best practices" ? Sure, best practices would be nice, but then we can run into problems like -f already used for something else. And again the operator needs to handle it individually in his startup scripts / service rules / etc. The best practice could also state: "please use that library if can, and leave the rest to the operator / distro". Or perhaps "don't ever let your programs daemonize themselves". > Then perhaps it's better to make it easier to use "standard" > command line options, and put that in a lib. E.g. a few funcs. > that > parses argc/argv according to some data, extracts values, puts in > default values, prints error messages, usage text or version > strings without all the usual mess with option handling. That would be nice, too, but I would put it into a different library, and certain point the program would still carry the code looking at the flag and deciding whether to daemonize. Yes, not much code (OTOH: rewriting to do the command line parsing by the lib would be more invasive, and you'll have trouble if -f is already used otherwise), but we'd have it consolidated in one place and operators can easily replace that whenever they feel necessary. > I think it is more fruitful in that case to write a lib that unifies the > command line, then environment and the config file handling, and make > it easy for the daemon writer to use it. Same as above. > Also of value could be a lib that unifies logging, or rather removes the > distinction between logging to syslog(), to a file or to stdout/stderr; > possible with default callback for SIGHUP to reopen/rotate log file. Actually, applications should either use syslog (except for debugging) or just use stderr and leave it to the daemonizing/monitor program. > Yes, and as a intermittent daemon writer, I try to understand why I > should use it. If I cannot find any reason then maybe we should think > of something better. Well, lots of developers see the need for having such an reporting. I, personally, never really had a real usecase for that, but these folks want that, and therefore use the only API they have - systemd. > There are two things, > 1, the monitor needs > 2, the (possible monitored) process/daemon needs > > To a daemon writer, point 1 is simply uninteresting. To get point 1 > satisfied, you have to provide something that first and formost > fulfill point 2, and by providing for point 2, you in your lib, can > introduce things that makes it easier for the monitor. > > So, instead of your proposal, I propose a lib that streamlines the > dull tasks of option/env/config_file, logging, and possible signal > handling. With it you can introduce things that is helpful for a > monitor. That would be a much wide scope. I was just trying to design a sane API for what many applications link in systemd now. --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Enrico Weigelt:
> On 18.04.2017 14:46, k...@aspodata.se wrote:
>
> > Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in
> > either camp, end case.
>
> I've had the impression that daemon() wasnt't good idea for everybody,
a, If you have that impression, then why daemonize() which sounds as
the same thing ?
b, No it isn't. That's why the -f, which is nice to have when debugging.
> and there might be some need for anyone doing something these.
The any other thing I can think of is running in the foreground and
"logging" to stdout/stderr, and possible being able to receive
commands via stdin.
> So I
> proposed to move that out to some library, which the operator easily
> replace, w/o touching the application itself. (patching up libc to their
> needs isn't quite what operators do ...).
if (do_daemon) { daemon(0,0); }
isn't much code to justify a lib entry, so what else is there in it to
lure a daemon writer to use it ?
> That '-f' switch would be in
> the lib in a different form, eg. via env variables.
Yes, it's messier to pick just the -f from argc/argv in a lib than
calling getenv().
> My goal here is not voting for any particular way of daemonizing,
> service monitoring, etc, but just to provide an abstraction layer, so
> all that stuff can be moved out of the individual application.
...
Well, that's fine, but replaceing -f and daemon() with your entry
doesn't seem worth it. Maybe it's better to write up some
"best practices" ?
> Certainly, everybody can implement the '-f' switch on his own (and it
> shouldn't be so difficult), but all of that repeated work (and the
> operator's work of reading manpages to find out which exact parameters
> he has to pass to an individual program to switch it on/off), could
> be saved if there was a single point for that. If you (as a service
> author) don't wanna use that, it's still your decision.
Then perhaps it's better to make it easier to use "standard"
command line options, and put that in a lib. E.g. a few funcs. that
parses argc/argv according to some data, extracts values, puts in
default values, prints error messages, usage text or version
strings without all the usual mess with option handling.
> > Why srvmgt_droppriv(), either accept that the process set's it itself
> > or force it to some uid:gid, what is it more to it ?
>
> It could even detect (eg. via env variables) whether it's already suid'd
> and then not even try and fail, it could fetch the target uid/gid from
> env, instead of having it's own (application specific) config or cmdline
> options. Again, consolidating repeated code into one place. Nothing
> more, nothing less. Similiar to srvmgt_daemonize(), it's only meant for
> cases where the service can't be started with the final uid/gid in the
> first place.
I think it is more fruitful in that case to write a lib that unifies the
command line, then environment and the config file handling, and make
it easy for the daemon writer to use it. In that kind of lib, you could
include default options for things like to daemon or not, uid/gid,
and where to log things.
I, as a daemon writer can see the value in having that handling defaulted
to some lib, and by it you can introduce default handling for -f and
--uid/--gid, so a possible monitor can set that what it to whatever it
wants.
Also of value could be a lib that unifies logging, or rather removes the
distinction between logging to syslog(), to a file or to stdout/stderr;
possible with default callback for SIGHUP to reopen/rotate log file.
You have to introduce what you want the other way around.
> > Why srvmgt_report_state(), just do a normal query to the process and
> > you'll know if it is ready to serve you or not,
>
> There might be cases when the process being alive doesn't necessarily
> mean it's ready to serve (usually it takes *some* time). By the call,
> the application can tell it to the outside world (however the actual
> signalling might happen, and who might listen).
>
> NOTE: I'm just proposing an *API*, *not* an particular implementation.
Yes, and as a intermittent daemon writer, I try to understand why I
should use it. If I cannot find any reason then maybe we should think
of something better.
> All I wanna get is that individual applications have an API (and ABI)
> to do these things w/o ever having to care whatever init system,
> service monitor, etc, there might or might not be. And the actual
> implementation shall be replacable w/o ever recompiling applications.
There are two things,
1, the monitor needs
2, the (possible monitored) process/daemon needs
To a daemon writer, point 1 is simply uninteresting. To get point 1
satisfied, you have to provide something that first and formost
fulfill point 2, and by providing for point 2, you in your lib, can
introduce things that makes it easier for the monitor.
So, instead of your proposal, I propose a lib that streamlines the
dull tasks of option/env/config_file, logging, and possible sign
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18.04.2017 12:21, Alessandro Selli wrote: > Again, as far as I understood what Enrico Weigelt wrote, the monitor > does not want to know the user:group the daemons runs under, it needs to > *set* them to the appropriate values when it launches the daemon to have > them reflect config file settings or local policies. Right. In that case, srvmgt_droppriv() would essentially do nothing (execpt perhaps some logging note). OTOH, we might have situations where the daemon needs to do some root operations first (eg. opening privileged sockets) before dropping privs. In that case the monitor/init can pass the target uid:gid via env, so it doesn't need to be configured within the application. > Again, the relevant info is supposed to be set in config files. The > user:group the daemon must run under is not up to the daemon itself, > they are set in config files, either the deamon's or the monitor's. Exactly. My approach also gives the choice of passing that from the init/monitor, even if the application needs to to the setuid() stuff itself. At that point we can consolidate all these individual assignments from dozens of separate configfiles to one place, somewhere within the init system / service monitor (which ever the operator might have chosen). In the end, the application doesn't even care about that anymore. > Again: if you do not want to take advantage of the monitor's policing > capabilities, don't use them. Exactly. And ease deployment and configuration for both cases, I'm just proposing to move these things out of individual applications and consolidate them in one point (the library). If one doesn't want any monitor, then he'll just install a minimalistic version, which wouldn't be much more than dummies or thin wrappers around things like daemon(). Anybody can choose his preferred implementation easily, w/o ever recompiling or even patching. --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18.04.2017 14:46, k...@aspodata.se wrote: > Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in > either camp, end case. I've had the impression that daemon() wasnt't good idea for everybody, and there might be some need for anyone doing something these. So I proposed to move that out to some library, which the operator easily replace, w/o touching the application itself. (patching up libc to their needs isn't quite what operators do ...). That '-f' switch would be in the lib in a different form, eg. via env variables. My goal here is not voting for any particular way of daemonizing, service monitoring, etc, but just to provide an abstraction layer, so all that stuff can be moved out of the individual application. All the srvmgt_daemonize() call tells is that after that point the process will be daemonized - whatever that *actually* means in the exact setup (which is up to the operator's choice). This is *only* for the usecase where an service wants to have a straight line at all (eg. before that it might print out error messages to stdout). Those who don't need that at all, probably just run in foreground and delegate daemonizing to the caller (monitor, start-stop-daemon, etc, etc, etc) Certainly, everybody can implement the '-f' switch on his own (and it shouldn't be so difficult), but all of that repeated work (and the operator's work of reading manpages to find out which exact parameters he has to pass to an individual program to switch it on/off), could be saved if there was a single point for that. If you (as a service author) don't wanna use that, it's still your decision. > Why srvmgt_droppriv(), either accept that the process set's it itself > or force it to some uid:gid, what is it more to it ? It could even detect (eg. via env variables) whether it's already suid'd and then not even try and fail, it could fetch the target uid/gid from env, instead of having it's own (application specific) config or cmdline options. Again, consolidating repeated code into one place. Nothing more, nothing less. Similiar to srvmgt_daemonize(), it's only meant for cases where the service can't be started with the final uid/gid in the first place. > Why srvmgt_report_state(), just do a normal query to the process and > you'll know if it is ready to serve you or not, There might be cases when the process being alive doesn't necessarily mean it's ready to serve (usually it takes *some* time). By the call, the application can tell it to the outside world (however the actual signalling might happen, and who might listen). NOTE: I'm just proposing an *API*, *not* an particular implementation. All I wanna get is that individual applications have an API (and ABI) to do these things w/o ever having to care whatever init system, service monitor, etc, there might or might not be. And the actual implementation shall be replacable w/o ever recompiling applications. --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18.04.2017 15:15, Arnt Gulbrandsen wrote: > There's hypothetical stuff, what if service x needs service y. Well, > what if it does. Should it demand that y be running at every moment and > on the same host? DOES it demand that? nfs daemons (plural) need portmap. many years ago, I've patched portmap to be fully stateless (directly maintains the table in some state files), so it can even be started on per-request basis. no idea whether my patches ever went into mainline ... > This isn't a good thing to spend effort on. The trend today is towards > services that require only an OS absolutely. In general a good idea (unless they don't duplicate so dozens of things over and over again). But there're still services out there that have those constraints - and somehow we gotta cope w/ that. Of course, often it's just bad sw architecture - just like most of the stuff that distros have to cope with. We could start w/ dist patches ... --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18.04.2017 16:08, Alessandro Selli wrote: Hi folks, seems that things got a bit overheated here ... > Where did Enrico Weigelt write the monitor should get it's child PID > from the process itself? This subthread started from these lines: > >>> * srvmgt_droppriv(...) >>> --> drop root privileges (if we are still root) >>> --> several versions, eg. with fetching the target uid/gid from env >> Given the pid, it isn't that hard for a monitor to find the uid/gid of >> the process, why has this have to go through the monitor ? Perhaps I should clarify this a bit: This library functions just shall drop privileges if the process is still running as root - and therefore might fetch the target uid/gid from environment (defined by start script, monitor, etc), instead of fetching from it's own config file. It's just implementing common repeated stuff in once central place and giving it a uniform interface. As that library would be provided by $whatever_initsystem, any further customizations would be done *there* instead of individual applications. The whole thing only applies to services which *need* to be started as root, instead of directly under the final uid/gid. (whether such things are good design at all is a whole different discussion). > «All daemons detach from the control terminal. The monitor and the > daemon must coordinate each other about who's doing the detach to avoid > launch time failures.» ACK. That's why I would put the code that does all (on the daemon side) into a separate library, which can be customized to whatever init/ monitor system the operator might choose, and the daemon itself doesn't need to be touched (not even recompiled) anymore. In that scenario, each init system (distro) package would provide an implementation of that library - all of them share the same API/ABI, so they can be easily replaced, w/o ever having to rebuild any daemon. > Daemons should behave the same no matter under what monitor they are > running under, they are not to care about this. ACK. But there are some points where they potentially need to care about, eg. who does the actual dataching (unfortunately, there isn't any ultimate standard) - and I'd like to get rid of individual per-daemon configuration (in both daemon and monitor) and delegate that part to a library. The operator (or the init system package maintainer) is free to put in the appropriate implementation. --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 11:10:24PM +0200, Alessandro Selli wrote: > On 18/04/2017 at 19:01, KatolaZ wrote: > > On Tue, Apr 18, 2017 at 06:48:31PM +0200, Alessandro Selli wrote: > > > > [cut] > > > >>> There is really no other reasonable alternative, IMVHO. > >> Yes, there is. It's called a good compromise. Then, outside of it, > >> you find the purists and the extremists. > >> > > We have seen what an irresistible avalanche of compromises has created > > in the systemd camp. > > Today I read someone (perhaps Enrico Weigelt) stating that one of > systemd's > pitfalls is wanting to be the perfect init. I agree. systemd does not > compromise, it wants thing to be done the way it wants, it forces all > userland to comply to it's dictates. systemd is the opposite of compromise, > it's dictatorial. > > > It turns out that I am a purist. And an extremist. > > This is one of the best recipies for failure. > It would be very good if you would like to actually contribute to Devuan :) There are lots of thigs to do, besides deciding who will fail and who will succeed. HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Il 18/04/2017 23:10, Alessandro Selli ha scritto: > On 18/04/2017 at 19:01, KatolaZ wrote: >> On Tue, Apr 18, 2017 at 06:48:31PM +0200, Alessandro Selli wrote: >> >> [cut] >> There is really no other reasonable alternative, IMVHO. >>> Yes, there is. It's called a good compromise. Then, outside of it, >>> you find the purists and the extremists. >>> >> We have seen what an irresistible avalanche of compromises has created >> in the systemd camp. > Today I read someone (perhaps Enrico Weigelt) stating that one of > systemd's > pitfalls is wanting to be the perfect init. Found it, it was Steve Litt instead: «There's no need to search for the perfect init or supervisor. Long ago we got a bunch of them that are all good enough, and can be combined to fill almost any need. This continuing search for a perfect init is just wheel spinning, or perhaps an excuse for profit-driven complexification.» -- Alessandro Selli Tel. 3701355486 VOIP SIP: dhatarat...@ekiga.net Chiave PGP/GPG key: B7FD89FD ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18/04/2017 at 19:01, KatolaZ wrote: > On Tue, Apr 18, 2017 at 06:48:31PM +0200, Alessandro Selli wrote: > > [cut] > >>> There is really no other reasonable alternative, IMVHO. >> Yes, there is. It's called a good compromise. Then, outside of it, >> you find the purists and the extremists. >> > We have seen what an irresistible avalanche of compromises has created > in the systemd camp. Today I read someone (perhaps Enrico Weigelt) stating that one of systemd's pitfalls is wanting to be the perfect init. I agree. systemd does not compromise, it wants thing to be done the way it wants, it forces all userland to comply to it's dictates. systemd is the opposite of compromise, it's dictatorial. > It turns out that I am a purist. And an extremist. This is one of the best recipies for failure. > And I am not at all > interested in blurry concepts like "market shares", If this feeling is shared by over 25% of Devuan's managers and developers, it's going to be the 1000th distribution so nice, so good, so "perfect" it's going to starve out of it's own failure to gain any significant foothold into the marketplace. You might dislike the market as much as you can, but that's what feeds projects and keeps distributions alive. Voluntary donations might get you started, but will not prop you up indefinitely into a viable OS capable of sustaining itself. > since the quest to > let Linux conquest "new market shares" has only brought problems and > created monsters. Get your conclusions. The quest to get Linux into as many lucrative markets as possible has made it what it is, that is a success, unlike worthy, perfectionist projects that failed nonetheless, like Plan9 or many of the successors to OpenSolaris. Who mentioned GNU? -- Alessandro Selli VOIP SIP: dhatarat...@ekiga.net Chiave PGP/GPG key: B7FD89FD ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 06:48:31PM +0200, Alessandro Selli wrote: [cut] > > > There is really no other reasonable alternative, IMVHO. > > Yes, there is. It's called a good compromise. Then, outside of it, > you find the purists and the extremists. > We have seen what an irresistible avalanche of compromises has created in the systemd camp. It turns out that I am a purist. And an extremist. And I am not at all interested in blurry concepts like "market shares", since the quest to let Linux conquest "new market shares" has only brought problems and created monsters. Get your conclusions. HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18/04/2017 at 16:16, KatolaZ wrote: > On Tue, Apr 18, 2017 at 02:48:54PM +0100, Arnt Gulbrandsen wrote: >> kato...@freaknet.org writes: >>> Unfortunately we are already paying the consequences of badly-written >>> software implementing oddly-designed solutions to non-existing >>> problems... >> Indeed. But what's your point? >> > Oh, you are right, I didn't make it explicit: my humble opinion is > that the whole thread is not particularly interesting or useful in > this context, since all this fuss about monitoring/supervision boils > down to two alternative options: > > a) you *really* need *strict* monitoring/supervsion. In this case, > you *don't* use a unix-like operating system, since you are always > left at least with the problem of monitoring the monitor (or > supervising the supervisor). But this is not a real problem, since > in those environments where *strict* supervision/monitoring is > *really* necessary, unix-like systems are rarely used, if ever, and > the scheduler also acts as a monitor/supervisor. So if you need > *real* *strict* monitoring, the whole topic is off-topic here. I do not agree, as Linux systems are employed both in real-time scenarios (though I admit this is a really niche use case) and in High Availability infrastructures. > b) you need *loose* monitoring/supervision. In this case, you pick > and choose among what is available, or write your own. In any case, > you get a monitor/supervisor running as a user process on top of any > general-purpose operating system, possibly a unix-like one, and your > applications must still be able to deal with faults, > inconsistencies, delays, and other issues. And sometimes you will > still need manual intervention nonetheless. The "everything will be > all right" theme is just dust in your eyes, because things are never > going to be all right all the time. This is an argument that sounds as an excuse to me: "Since you'll never get perfection, just do nothing." Everyone knows that even top500 supercomputers do need some manual intervention from time to time. This is not a good reason to put the sysadmin in charge of manually managing all the processes and events that they produce. Some OS could go this way, of course, free to. But you know very well that such an OS, though philosophically and academically fine and technically elegantly constructed, is not going to win any significant market share, as today everywhere the utmost of automation is a prime requirement. > There is really no other reasonable alternative, IMVHO. Yes, there is. It's called a good compromise. Then, outside of it, you find the purists and the extremists. Alessandro ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18/04/2017 at 14:46, k...@aspodata.se wrote: [...] > Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in > either camp, end case. This is well *if* the daemon has -f and does daemon(). I know the only programs that do not do it are those that were not intended to be run as daemons, still from time to time people do that, with netcat for instance. > Why srvmgt_droppriv(), either accept that the process set's it itself > or force it to some uid:gid, what is it more to it ? "What if you do not trust the program?" (quote) Then you answered yourself already: "force it to some uid:gid": this is the monitor's job. One of them. > Why srvmgt_report_state(), just do a normal query to the process and > you'll know if it is ready to serve you or not, This way you only know that the process is up, not that "it is ready to serve you or not". I'm sure you've had experiences of processes that are running but are not delivering the services you expected from them. > if you want to know > if it is up to speed, do a benchmark, if you want to test the > integrity, do a regression test; what is it more to it ? Knowing if it is ready to serve you or not", for instance. Learning if it got stuck on something, if that is temporary or definitive, if it's best to kill it or maybe raise it's allotted max number of open files. Or just write a line in the logs and let the sysadmin pick it up and see for himself what to do. If you want an init that only launches programs and daemons and as soon forgets all about them, fine. I'm not in principle against such a minimalistic init. Just don't call it a monitor thought, 'cause it ain't. Alessandro ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18/04/2017 at 16:42, k...@aspodata.se wrote: > Alessandro: >> On 18/04/2017 at 15:11, k...@aspodata.se wrote: >>> Alessandro Selli: >>> ... And what I don't like of Karl Aspo's idea is that it takes any instrument of policy checking and enforcing out of the monitor, which ends up not being able to monitor anything, it becomes just a shell: fire and forget. >>> I honestly do not understand what I have written which gives you that idé. >> Your many lines that the monitor is not to do any monitoring. Like: >> "Why do any monitor program need to know if the program has detached or >> not, the only thing it needs to know is the pid and the state,". > You cut out the thing after the "," which makes that a question for > someone else to fill in the blanks, the things that's not obvious > to me. I got you had a question, I did understand you had no idea why would "any monitor program need to know if the program has detached or not". Again I quote myself: «All daemons detach from the control terminal. The monitor and the daemon must coordinate each other about who's doing the detach to avoid launch time failures.» It's been obvious to all Unix sysadmins for thirty sound years, if it's still not clear to you I think I must give up. > If a process knows the pid of another process and have sufficient > rights, can it not manage that process or is there anything else it > needs to know, what am I missing ? It manages them based on what? Local config files, events transmitted, signals? You're missing the whole "monitoring" and policing the monitor does besides launching the daemon. Alessandro ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Alessandro: > On 18/04/2017 at 15:11, k...@aspodata.se wrote: > > Alessandro Selli: > > ... > >> And > >> what I don't like of Karl Aspo's idea is that it takes any instrument of > >> policy checking and enforcing out of the monitor, which ends up not > >> being able to monitor anything, it becomes just a shell: fire and forget. > > I honestly do not understand what I have written which gives you that idé. > Your many lines that the monitor is not to do any monitoring. Like: > "Why do any monitor program need to know if the program has detached or > not, the only thing it needs to know is the pid and the state,". You cut out the thing after the "," which makes that a question for someone else to fill in the blanks, the things that's not obvious to me. If a process knows the pid of another process and have sufficient rights, can it not manage that process or is there anything else it needs to know, what am I missing ? Regards, /Karl Hammar --- Aspö Data Lilla Aspö 148 S-742 94 Östhammar Sweden +46 173 140 57 ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Katola2: > On Tue, Apr 18, 2017 at 03:11:51PM +0200, k...@aspodata.se wrote: ... > > If I can get a value from the kernel instead of from the process, > > I'd take the kernel value. > > > > Why do a process have to query the kernel to get a value and then > > sending it to a monitor over a communication link; why don't the > > monitor query the kernel itself, saving one step. > I genuinely don't understand why the kernel should know about the > internals of running processes, or get notified if a process is ... What is the problem with you guys, I did write "if". That "if" applies to thing like PID, uid, gid and such of processes, not wheter they are in some sense ready or not. Please, don't continously misinterpret me. Regards, /Karl Hammar --- Aspö Data Lilla Aspö 148 S-742 94 Östhammar Sweden +46 173 140 57 ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 02:48:54PM +0100, Arnt Gulbrandsen wrote: > kato...@freaknet.org writes: > >Unfortunately we are already paying the consequences of badly-written > >software implementing oddly-designed solutions to non-existing > >problems... > > Indeed. But what's your point? > Oh, you are right, I didn't make it explicit: my humble opinion is that the whole thread is not particularly interesting or useful in this context, since all this fuss about monitoring/supervision boils down to two alternative options: a) you *really* need *strict* monitoring/supervsion. In this case, you *don't* use a unix-like operating system, since you are always left at least with the problem of monitoring the monitor (or supervising the supervisor). But this is not a real problem, since in those environments where *strict* supervision/monitoring is *really* necessary, unix-like systems are rarely used, if ever, and the scheduler also acts as a monitor/supervisor. So if you need *real* *strict* monitoring, the whole topic is off-topic here. b) you need *loose* monitoring/supervision. In this case, you pick and choose among what is available, or write your own. In any case, you get a monitor/supervisor running as a user process on top of any general-purpose operating system, possibly a unix-like one, and your applications must still be able to deal with faults, inconsistencies, delays, and other issues. And sometimes you will still need manual intervention nonetheless. The "everything will be all right" theme is just dust in your eyes, because things are never going to be all right all the time. There is really no other reasonable alternative, IMVHO. But I am not an expert in the field (though I am quite happy about not being an expert, having seen what the "experts in the field" have been able to come up with in the last couple of years...) My2Cents KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18/04/2017 at 15:11, k...@aspodata.se wrote: > Alessandro Selli: > ... >> And >> what I don't like of Karl Aspo's idea is that it takes any instrument of >> policy checking and enforcing out of the monitor, which ends up not >> being able to monitor anything, it becomes just a shell: fire and forget. > I honestly do not understand what I have written which gives you that idé. Your many lines that the monitor is not to do any monitoring. Like: "Why do any monitor program need to know if the program has detached or not, the only thing it needs to know is the pid and the state,". >> I was also been ironic on Aspo, as many times he can only counter >> another person's ideas asking "What if cannot be trusted?", >> as if this constitutes a valid argument against being able to set >> policies for the monitor to enforce on the daemons it runs. > If I can get a value from the kernel instead of from the process, > I'd take the kernel value. What value? Of course there are values that are set and managed by the kernel, not by the process, and are thus most dependable when extracted from the kernel itself. However other values are specific to the daemon and the kernel doesn't care/know anything of, like what virtual domains the webserver is handling, of how many domains the DNS server is master. Anyway, this has very little (if anything) to do with the monitor's need to be able to... monitor the processes it launches, set them to the wanted user:group and the daemons knowing whether they are to detach from the controlling terminal or not. > Why do a process have to query the kernel to get a value and then > sending it to a monitor over a communication link; why don't the > monitor query the kernel itself, saving one step. Where did Enrico Weigelt write the monitor should get it's child PID from the process itself? This subthread started from these lines: >> * srvmgt_droppriv(...) >> --> drop root privileges (if we are still root) >> --> several versions, eg. with fetching the target uid/gid from env > Given the pid, it isn't that hard for a monitor to find the uid/gid of > the process, why has this have to go through the monitor ? >> In this >> case, asking "why should a program/daemon care if it has a monitor or >> not ?" is moot, because that the daemon is aware or not that it was >> launched by a monitor and if so by what monitor, does not change >> anything about the monitor's functions and duties. The daemon doesn't >> care if it's run by a monitor? The monitor is still there, and it does >> care about the program. > It would help my understanding if you simply answered my question. An answer to a moot question is Mu! (https://en.wikipedia.org/wiki/Mu_%28negative%29) The only point of any technical interest was already answered: «All daemons detach from the control terminal. The monitor and the daemon must coordinate each other about who's doing the detach to avoid launch time failures.» And again, so long as it's configured the right way, that the daemon is aware that it is run under a monitor or not is irrelevant: it is the monitor that need to be aware of it's children processes, their status and operational parameters, because it is responsible of what they do, starting from setting the user:group they run under. Daemons should behave the same no matter under what monitor they are running under, they are not to care about this. This is one of the reasons systemd is bad. Still, the monitor does the monitoring. >> A driver does not care if the traffic light is >> red or green? The street police does. > This is not as a suitable metaphor as you think it is. It is, no matter what you think about it. Alessandro ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
kato...@freaknet.org writes: Unfortunately we are already paying the consequences of badly-written software implementing oddly-designed solutions to non-existing problems... Indeed. But what's your point? Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
kato...@freaknet.org writes: I genuinely don't understand why the kernel should know about the internals of running processes, or get notified if a process is "ready" to do whatever it is supposed to do, or get queried by other processes which would like to access this kind of information, or maintain such information in the first instance... Indeed. Particularly in light of the modern fashion of running things on other hosts reachable across the network, perhaps via load balancers that complicate the availability state machine. Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 02:15:38PM +0100, Arnt Gulbrandsen wrote: > I've read this thread fairly thoroughly, but fail to see much of a use > case... > > There's hypothetical stuff, what if service x needs service y. Well, what if > it does. Should it demand that y be running at every moment and on the same > host? DOES it demand that? > Unfortunately we are already paying the consequences of badly-written software implementing oddly-designed solutions to non-existing problems... HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 03:11:51PM +0200, k...@aspodata.se wrote: [cut] > > > I was also been ironic on Aspo, as many times he can only counter > > another person's ideas asking "What if cannot be trusted?", > > as if this constitutes a valid argument against being able to set > > policies for the monitor to enforce on the daemons it runs. > > If I can get a value from the kernel instead of from the process, > I'd take the kernel value. > > Why do a process have to query the kernel to get a value and then > sending it to a monitor over a communication link; why don't the > monitor query the kernel itself, saving one step. > I genuinely don't understand why the kernel should know about the internals of running processes, or get notified if a process is "ready" to do whatever it is supposed to do, or get queried by other processes which would like to access this kind of information, or maintain such information in the first instance... A general-purpose unix-like kernel must do what it has been designed for: providing access to hardware resources (memory and devices) and schedule processes. Maybe you need another (non-unix-like and non-general-purpose) operating system? HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Alessandro Selli: ... > And > what I don't like of Karl Aspo's idea is that it takes any instrument of > policy checking and enforcing out of the monitor, which ends up not > being able to monitor anything, it becomes just a shell: fire and forget. I honestly do not understand what I have written which gives you that idé. > I was also been ironic on Aspo, as many times he can only counter > another person's ideas asking "What if cannot be trusted?", > as if this constitutes a valid argument against being able to set > policies for the monitor to enforce on the daemons it runs. If I can get a value from the kernel instead of from the process, I'd take the kernel value. Why do a process have to query the kernel to get a value and then sending it to a monitor over a communication link; why don't the monitor query the kernel itself, saving one step. > In this > case, asking "why should a program/daemon care if it has a monitor or > not ?" is moot, because that the daemon is aware or not that it was > launched by a monitor and if so by what monitor, does not change > anything about the monitor's functions and duties. The daemon doesn't > care if it's run by a monitor? The monitor is still there, and it does > care about the program. It would help my understanding if you simply answered my question. > A driver does not care if the traffic light is > red or green? The street police does. This is not as a suitable metaphor as you think it is. Regards, /Karl Hammar --- Aspö Data Lilla Aspö 148 S-742 94 Östhammar Sweden +46 173 140 57 ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
I've read this thread fairly thoroughly, but fail to see much of a use case... There's hypothetical stuff, what if service x needs service y. Well, what if it does. Should it demand that y be running at every moment and on the same host? DOES it demand that? This isn't a good thing to spend effort on. The trend today is towards services that require only an OS absolutely. A modern service requires e.g. a read/write file system and/or a network interface with addressing and routing. If a modern service uses something like an SQL database, it'll paper over inavailability, it'll delay, retry, reconnect. The fashionable rationale is cloudy wizbangery. But a notable side effect is that postgresql and services that use postgresql can start at the same time. I'm not saying that ALL services cope with unavailable dependencies. What I am saying is rather that spending effort on supporting this kind of dependency management has low and decreasing value. AWS and other systems that provision instances according to need push strongly on server software to cope with momentary inavailability, odd starting order, etc. Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Allesandro, I'm not interested in a debate (people talking past each other), I'm interested in understanding why the proposed library and it function calls are such a good idé, since I don't think it is. Wheter to run a monitor or not is a different question, which I consider is up to the local admin. Why srvmgt_daemonize(), use -f and daemon() and you'd be fine in either camp, end case. Why srvmgt_droppriv(), either accept that the process set's it itself or force it to some uid:gid, what is it more to it ? Why srvmgt_report_state(), just do a normal query to the process and you'll know if it is ready to serve you or not, if you want to know if it is up to speed, do a benchmark, if you want to test the integrity, do a regression test; what is it more to it ? (And, yes, a monitoring program can do the above for you.) I don't get the point of all this, can you explain ? Regards, /Karl Hammar --- Aspö Data Lilla Aspö 148 S-742 94 Östhammar Sweden +46 173 140 57 ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 01:32:01PM +0200, Adam Borowski wrote: [cut] > > > > This is a totally wrong assumption. Being a sysadmin is all about > > setting and implementing *policies*. And the choice between systemd or > > anything else is exactly a matter of policy. So yes, a sysadmin *must* > > care if the OS he/she uses runs systemd or anything else. > > I don't care whether the OS runs systemd, sysv-rc, openrc or an enslaved > young boy sitting at the console and starting daemons by hand. What I do > care about is whether it: > * implements standard interfaces > * is correct while doing so > This is exactly my point :) A sysadmin *must* care about init because not all the enslaved young boys sitting at the root console react or complain in the same way. Some of them just limit their actions to obeying to what the sysadmin asks them to do, and let the sysadmin choose and implement the policy he/she wants. Some other are quite opinionated guys. They come with their pre-defined policies (often called "typical use cases"), which are often terribly hard to work around, and stubbornly refuse to obey if the sysadmin does not align to those "typical use cases". This is something most sysadmin cannot (and should not) tolerate... HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 11:32:00AM +0100, KatolaZ wrote: > On Tue, Apr 18, 2017 at 12:21:05PM +0200, Alessandro Selli wrote: > > Why should a car driver care if the traffic light has the red or the > > green light turned on? > > Why should a sysadmin care if the OS he uses runs systemd or not? > > This is a totally wrong assumption. Being a sysadmin is all about > setting and implementing *policies*. And the choice between systemd or > anything else is exactly a matter of policy. So yes, a sysadmin *must* > care if the OS he/she uses runs systemd or anything else. I don't care whether the OS runs systemd, sysv-rc, openrc or an enslaved young boy sitting at the console and starting daemons by hand. What I do care about is whether it: * implements standard interfaces * is correct while doing so My beef with systemd is that it has egregious bugs (some due to mistakes, some due to intentional brain damage), and that working around bugs or places where it doesn't meet your particular use case is a matter of a single line of shell on sysv-rc but on systemd it's a hopeless case of digging through a giant blob of spaghetti code. A sample openrc (init script?) bug: [ ok ] Asking all remaining processes to terminate...done. Currently running processes (pstree): init-+-2*[nbd-client] `-rc---openrc---openrc-run---sendsigs---pstree [FAIL] Killing all remaining processes...failed. Stopping NBD client process: disconnect, [17892.932316] block nbd0: NBD_DISCONNECT sock, [17892.937274] block nbd0: shutting down sockets done disconnect, [17892.951774] block nbd1: NBD_DISCONNECT sock, [17892.956734] block nbd1: shutting down sockets done rmmod: ERROR: Module nbd is in use nbd-client. [warn] not deconfiguring network interfaces: network devices still mounted. ... (warning). [info] Saving the system clock. [info] Hardware Clock updated to Tue Apr 18 13:18:23 CEST 2017. [ ok ] Deactivating swap...done. [] Unmounting local filesystems...[17896.568144] block nbd0: Attempted send on invalid socket [17896.573501] blk_update_request: I/O error, dev nbd0, sector 4257856 [17896.579804] block nbd0: Attempted send on invalid socket [17896.585125] blk_update_request: I/O error, dev nbd0, sector 4258112 [17896.591402] BTRFS error (device nbd0): bdev /dev/nbd0 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 [17896.600422] block nbd0: Attempted send on invalid socket [17896.605741] blk_update_request: I/O error, dev nbd0, sector 4262496 [17896.612018] block nbd0: Attempted send on invalid socket A sysadmin who doesn't know the proper way to fix this can still trivially deal with the issue. Don't know how to make nbd-client stop _after_ network filesystems are unmounted? Just plop in a manual unmount somewhere. Here, your system is fixed, you can then learn when you got a bit of time. A sample systemd bug: A common way to backup/search/etc a filesystem that has something mounted in random subdirectories is to "mount --bind / /mnt/ref/rootfs; mount --bind /home /mnt/ref/home" and so on, so you don't risk a naive script getting into a loop. Except, systemd sometimes decides to convert that --bind into a --rbind (specifically what you wanted to avoid!); it might happen a fraction of second after the mount, several minutes or apparently never. So you can't even rely on that bug happening or not. It will also sometimes decide to _unmount_ what you just mounted (most often on a degraded raid). Now try to work around this... -- ⢀⣴⠾⠻⢶⣦⠀ Meow! ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋⠀ Collisions shmolisions, let's see them find a collision or second ⠈⠳⣄ preimage for double rot13! ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18/04/2017 at 12:32, KatolaZ wrote: > On Tue, Apr 18, 2017 at 12:21:05PM +0200, Alessandro Selli wrote: > > [cut] > >>> And why should a program/daemon care if >>> it has a monitor or not ? >> Why should a car driver care if the traffic light has the red or the >> green light turned on? >> Why should a sysadmin care if the OS he uses runs systemd or not? > This is a totally wrong assumption. Being a sysadmin is all about > setting and implementing *policies*. And the choice between systemd or > anything else is exactly a matter of policy. So yes, a sysadmin *must* > care if the OS he/she uses runs systemd or anything else. Which is exactly my point. I am not advocating in favor of systemd any more than I'm advocating letting drivers ignore traffic lights. And what I don't like of Karl Aspo's idea is that it takes any instrument of policy checking and enforcing out of the monitor, which ends up not being able to monitor anything, it becomes just a shell: fire and forget. I was also been ironic on Aspo, as many times he can only counter another person's ideas asking "What if cannot be trusted?", as if this constitutes a valid argument against being able to set policies for the monitor to enforce on the daemons it runs. In this case, asking "why should a program/daemon care if it has a monitor or not ?" is moot, because that the daemon is aware or not that it was launched by a monitor and if so by what monitor, does not change anything about the monitor's functions and duties. The daemon doesn't care if it's run by a monitor? The monitor is still there, and it does care about the program. A driver does not care if the traffic light is red or green? The street police does. Alessandro ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, 18 Apr 2017 11:32:00 +0100 KatolaZ wrote: > > Why should a car driver care if the traffic light has the red or the > > green light turned on? > > Why should a sysadmin care if the OS he uses runs systemd or not? > > > > This is a totally wrong assumption. Being a sysadmin is all about > setting and implementing *policies*. And the choice between systemd or > anything else is exactly a matter of policy. So yes, a sysadmin *must* > care if the OS he/she uses runs systemd or anything else. He should care, as he has to know whether it is a unix/linux system or a systemd system, the two being rather different in their management... Cheers, Ron. -- Eat drink and be merry, for tomorrow we may diet. -- http://www.olgiati-in-paraguay.org -- ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 12:21:05PM +0200, Alessandro Selli wrote: [cut] > > > And why should a program/daemon care if > > it has a monitor or not ? > > Why should a car driver care if the traffic light has the red or the > green light turned on? > Why should a sysadmin care if the OS he uses runs systemd or not? > This is a totally wrong assumption. Being a sysadmin is all about setting and implementing *policies*. And the choice between systemd or anything else is exactly a matter of policy. So yes, a sysadmin *must* care if the OS he/she uses runs systemd or anything else. HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 18/04/2017 at 10:35, k...@aspodata.se wrote: > Alessandro Selli: >> On Fri, 14 Apr 2017 at 12:06:44 +0200 (CEST) >> k...@aspodata.se wrote: >>> Enrico Weigelt: >>> ... Let's just take some example: libsrvmgt with funcs like that: * srvmgt_daemonize() --> detach from controlling terminal, etc >>> Why do any monitor program need to know if the program has detached >>> or not, the only thing it needs to know is the pid and the state, >>> which would/could be provided by the *_state() functions, >>> or am I wrong ? >> All daemons detach from the control terminal. > A daemon from a daemon ? Isn't init a daemon? Isn't inetd/xinetd a daemon? >> The monitor and the >> daemon must coordinate each other about who's doing the detach to avoid >> launch time failures. > The program to become the daemon, how does it knows if it has a monitor > or not ? In the case of init being the monitor, it can check if it's parent has PID 1. The classical approach is to configure the daemon to do the right thing, i.e. to do the detach himself when run with the --daemon switch or when the config file has a similar item configured, to let the monitor do it otherwise or if explicitly told not to daemonize (some programs have a --no-daemon or --foreground or --debug switch to override the config file settings). This is old-school Unix, classical Unix daemon have had these items (switches and config files) for many ten years... the modern approach in Linux entails IPC, dbus, sockets to have messages passed from the monitor to daemons and back. > And if it doesn't know, how can one demand that it should > coordinate with something unknown ? Config files, command line switches. > And why should a program/daemon care if > it has a monitor or not ? Why should a car driver care if the traffic light has the red or the green light turned on? Why should a sysadmin care if the OS he uses runs systemd or not? * srvmgt_droppriv(...) --> drop root privileges (if we are still root) --> several versions, eg. with fetching the target uid/gid from env >>> Given the pid, it isn't that hard for a monitor to find the uid/gid of >>> the process, why has this have to go through the monitor ? >> I don't think here the monitor is used to extract process info (such >> as user/group it's running under), rather the environment is used to >> determine what user:group the daemon must be set to by the monitor. > I don't follow you. If the monitor shouldn't know much about the child > as stated by someone somewhere else in this thread, why should the > monitor need to know user:group ? Again, as far as I understood what Enrico Weigelt wrote, the monitor does not want to know the user:group the daemons runs under, it needs to *set* them to the appropriate values when it launches the daemon to have them reflect config file settings or local policies. > And if it should know it, why trust > the monitored program when the kernel can give you the info ? Again, the relevant info is supposed to be set in config files. The user:group the daemon must run under is not up to the daemon itself, they are set in config files, either the deamon's or the monitor's. >>> Also, given that you can do the above without this lib opens gives the >>> program the option too cheat, say, if the monitor wishes to have a >>> tight control of the process. >> It depends by where is that environment set. It might be a config >> file. You're supposed to trust your config files. > Or, rather, the user is responsible for the config file, not the program > reading it, so yes the program generally trusts the config file "since I > demand it". And what is your point ? That the relevant info is set in the config files, they are not made up by the program itself and trusted by the monitor. The trust is placed into the config files. * srvmgt_report_state(...) >>> --> report the service state to the supervisor >>> ... >>> >>> There could be something like *_a_would_like_to_be_monitored() and >>> *_I_wish_to_regain_my_free_will(). >> The monitor is supposed to implement local security/service >> continuity/resource allocation policies the daemon must not be able to >> bail itself out of. > It seems you have the view that a monitor has to police a program. Many monitor also do this, some are setup mainly for this reason. Xinetd has several config items related to security or resource policing: some "flags" (like IDONLY and INTERCEPT), the "user" and "group" settings, "libwrap", "only_from", "no_access", "access_times", "redirect", "bind", "per_source", "cps" (connections per second), "max_load", "umask", "rlimit_(as|cpu|data|rss|stack)", "passenv", "deny_time". It's good they are available, that you use them or don't. > I don't think one should have such a strict view. If you don't want them, don't use them. Let other people decide otherwise, though. There is not a single use case that
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Alessandro Selli: > On Fri, 14 Apr 2017 at 12:06:44 +0200 (CEST) > k...@aspodata.se wrote: > > Enrico Weigelt: > > ... > > > Let's just take some example: libsrvmgt with funcs like that: > > > > > > * srvmgt_daemonize() > > > --> detach from controlling terminal, etc > > > > Why do any monitor program need to know if the program has detached > > or not, the only thing it needs to know is the pid and the state, > > which would/could be provided by the *_state() functions, > > or am I wrong ? > All daemons detach from the control terminal. A daemon from a daemon ? > The monitor and the > daemon must coordinate each other about who's doing the detach to avoid > launch time failures. The program to become the daemon, how does it knows if it has a monitor or not ? And if it doesn't know, how can one demand that it should coordinate with something unknown ? And why should a program/daemon care if it has a monitor or not ? > > > * srvmgt_droppriv(...) > > > --> drop root privileges (if we are still root) > > > --> several versions, eg. with fetching the target uid/gid from env > > > > Given the pid, it isn't that hard for a monitor to find the uid/gid of > > the process, why has this have to go through the monitor ? > > I don't think here the monitor is used to extract process info (such > as user/group it's running under), rather the environment is used to > determine what user:group the daemon must be set to by the monitor. I don't follow you. If the monitor shouldn't know much about the child as stated by someone somewhere else in this thread, why should the monitor need to know user:group ? And if it should know it, why trust the monitored program when the kernel can give you the info ? > > Also, given that you can do the above without this lib opens gives the > > program the option too cheat, say, if the monitor wishes to have a > > tight control of the process. > > It depends by where is that environment set. It might be a config > file. You're supposed to trust your config files. Or, rather, the user is responsible for the config file, not the program reading it, so yes the program generally trusts the config file "since I demand it". And what is your point ? > > > * srvmgt_report_state(...) > > --> report the service state to the supervisor > > ... > > > > There could be something like *_a_would_like_to_be_monitored() and > > *_I_wish_to_regain_my_free_will(). > > The monitor is supposed to implement local security/service > continuity/resource allocation policies the daemon must not be able to > bail itself out of. It seems you have the view that a monitor has to police a program. I don't think one should have such a strict view. > > > --> states could be eg. > > > * SRVMGT_STATE_STARTUP -- still within the startup phase > > > * SRVMGT_STATE_READY_LOCAL -- ready for local clients only > > > * SRVMGT_STATE_READY_ALL -- ready for all clients > > > * SRVMGT_STATE_BUSY-- too busy to process new requests > > > * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing > > > queued requests > > > * SRVMGT_STATE_DEFERRED-- temporarily can't accept new > > > requests (eg. overload) > > > * SRVMGT_STATE_WAITING -- wait for resource (eg. printer > > > needs paper or ink) > > > * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some > > > fatal error) > > > > Given the choises given, it seems that the target of the monitor is > > network servers. Couldn't the monitor find out the ready_local, > > ready_all and shutdown by itself by monitoring which ports are open ? > > The monitor is not supposed to always grant any daemon any port it > wishes. ... Why, the monitor is not a firewall. It seems you and I have different views about what a monitor is and should/shouldn't do. > > And, what if the monitor cannot trust the program it monitors ? > > Given the lib suggestion, it seems that we are trusting the program. > > Well... what if you do not trust the monitor? Then I wouldn't run it, would you ? > What do you mean by what you wrote? How do you think you can teach a > monitor if some program can be trusted? ... The monitor doesn't have to trust a specific program if it can get the the wanted info from the kernel. If I can get the info I want from the kernel, why should I ask the program ? If the kernel say something and the program something else, which one do you trust more ? > > But what if if we don't know if we can trust the walues provided by > > the program, the we have to check what the program is really doing, > > and then the lib have no value, isn't that so ? > > Could you please provide us with an example of untrusted > behaviour/program provided data that the monitor is supposed to detect > and handle accordingly? First off, we seems to have dif
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Tue, Apr 18, 2017 at 03:36:09AM -0400, Steve Litt wrote: > On Mon, 17 Apr 2017 10:09:02 +0200 > exha...@posteo.net wrote: > > > in the same way that there's not a perfect graphical desktop for all > > the people, > > Quoted for truth! > > You can't *download* or *install* the perfect GUI desktop, you must > construct the GUI desktop that's best for you. And that requires > interchangeable parts. > > I sense a theme here. > The theme is quite easy to spot, and has been around for the best part of the last 50 years: small, reusable, interchangeable components, each providing a single, atomic, simple mechanism, avoiding to mess up with policies (which are for the perusal of humans, not machines...). This is what KISS is all about. We have always needed and will always need more software. But never "more standard policies" or "unique and perfect solutions" for each problem. Those are quests for fools. HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Mon, 17 Apr 2017 10:09:02 +0200 exha...@posteo.net wrote: > in the same way that there's not a perfect graphical desktop for all > the people, Quoted for truth! You can't *download* or *install* the perfect GUI desktop, you must construct the GUI desktop that's best for you. And that requires interchangeable parts. I sense a theme here. SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 17/04/2017 at 03:57, Steve Litt wrote: [...] > There's no need to search for the perfect init or supervisor. Long ago > we got a bunch of them that are all good enough, and can be combined to > fill almost any need. This continuing search for a perfect init is just > wheel spinning, or perhaps an excuse for profit-driven complexification. I fully agree. Than you for having put down such a good manifesto. -- Alessandro Selli Tel. 3701355486 VOIP SIP: dhatarat...@ekiga.net Chiave PGP/GPG key: B7FD89FD ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Fri, 14 Apr 2017 at 12:06:44 +0200 (CEST) k...@aspodata.se wrote: > Enrico Weigelt: > ... > > Let's just take some example: libsrvmgt with funcs like that: > > > > * srvmgt_daemonize() > > --> detach from controlling terminal, etc > > Why do any monitor program need to know if the program has detached > or not, the only thing it needs to know is the pid and the state, > which would/could be provided by the *_state() functions, > or am I wrong ? All daemons detach from the control terminal. The monitor and the daemon must coordinate each other about who's doing the detach to avoid launch time failures. > > * srvmgt_droppriv(...) > > --> drop root privileges (if we are still root) > > --> several versions, eg. with fetching the target uid/gid from env > > Given the pid, it isn't that hard for a monitor to find the uid/gid of > the process, why has this have to go through the monitor ? I don't think here the monitor is used to extract process info (such as user/group it's running under), rather the environment is used to determine what user:group the daemon must be set to by the monitor. > Also, given that you can do the above without this lib opens gives the > program the option too cheat, say, if the monitor wishes to have a > tight control of the process. It depends by where is that environment set. It might be a config file. You're supposed to trust your config files. [...] > > * srvmgt_report_state(...) > --> report the service state to the supervisor > ... > > There could be something like *_a_would_like_to_be_monitored() and > *_I_wish_to_regain_my_free_will(). The monitor is supposed to implement local security/service continuity/resource allocation policies the daemon must not be able to bail itself out of. > > --> states could be eg. > > * SRVMGT_STATE_STARTUP -- still within the startup phase > > * SRVMGT_STATE_READY_LOCAL -- ready for local clients only > > * SRVMGT_STATE_READY_ALL -- ready for all clients > > * SRVMGT_STATE_BUSY-- too busy to process new requests > > * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing > > queued requests > > * SRVMGT_STATE_DEFERRED-- temporarily can't accept new > > requests (eg. overload) > > * SRVMGT_STATE_WAITING -- wait for resource (eg. printer > > needs paper or ink) > > * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some > > fatal error) > > Given the choises given, it seems that the target of the monitor is > network servers. Couldn't the monitor find out the ready_local, > ready_all and shutdown by itself by monitoring which ports are open ? The monitor is not supposed to always grant any daemon any port it wishes. Monitors are there also to prevent daemons to do just as they will. One thing is to trust the monitor, another thing is to trust the whole collection of deamons it manages. [...] > /// > > And, what if the monitor cannot trust the program it monitors ? > Given the lib suggestion, it seems that we are trusting the program. Well... what if you do not trust the monitor? What do you mean by what you wrote? How do you think you can teach a monitor if some program can be trusted? I don't think I will love a monitor that is not going to start, say, the sshd daemon because it thinks it cannot be trusted. I must know what it means by this (bad to system security? to the available resourses?), I want to know the specific reason it believes sshd is bad and I want this behaviour to be configurable. > But what if if we don't know if we can trust the walues provided by > the program, the we have to check what the program is really doing, > and then the lib have no value, isn't that so ? Could you please provide us with an example of untrusted behaviour/program provided data that the monitor is supposed to detect and handle accordingly? Alessandro ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
exha...@posteo.net wrote: > In my opinion, this is the best that has been written about init or > supervisor, +1 > (I'm absolutely agree with this) > > "There's no need to search for the perfect init or supervisor. Long ago > we got a bunch of them that are all good enough, and can be combined to > fill almost any need. This continuing search for a perfect init is just > wheel spinning, or perhaps an excuse for profit-driven complexification" Absolutely. And I confess it's something I fall prey to from time to time - seeking to "do it right" which sometimes results in not doing it at all. Lets just say there was "some disagreement" between myself and SWMBO as to what "decorate the spare bedroom" means - though at the end of it (a year per room, two rooms so far !) she's been happy with the results :-) ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Mon, Apr 17, 2017 at 12:33:06PM +0200, marc wrote: > the kernel thesedays > allows one to delegate the "default-child-collector" function of > init to other processes - that is what the container infrastructure uses. Which seems to be an effective rebuttal against the argument for systemd being a universal service supervisor as pid 1. -- hendrik ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sun, Apr 16, 2017 at 09:57:38PM -0400, Steve Litt wrote: > On Sun, 16 Apr 2017 19:22:36 -0400 > Hendrik Boom wrote: > > > On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote: > > > On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT > > > consult wrote: > > > > > By the way: maybe we should write an RFC draft for the whole > > > > issue ... > > > > > > Looked for a relevant RFC. But found only this: > > > https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html > > > [RFC] [PATCH] notify init daemon when children are reparented to it > > > > > > But this doesn't seem to be quite what we want, and I can't say I > > > have enough context to understand it. > > > > That so-calledRFC seems to be very much in the style of > > requiring the init process to be intimately involved in process > > supervision actions, which is a strong constraint on what init > > systems can be chosen. And, I suspect, an init system that is so > > involved is a potential tool to gradually take over the entire OS, as > > systemd has done. > > And, in fact, being written in 2008, that so called rfc might have been > either the excuse or the inspiration for systemd. > > The "rfc" suffers from a perfectionism related to bikeshedding. Heaven > forbid there's ever a zombie or a process with a 'T' in the ps status > field! We simply MUST make a perfect init or supervisor system: Nothing > less will do. And while we're indulging our perfection, it's important > to remember that simplicity not be a priority at all. We will remain > oblivious to the fact that complexity and lack of encapsulation > creates flaws far worse than the flaws we're moving heaven and earth > to get rid of. And for gosh sakes, let's forget the facts: > > THE FACTS > > * sysvinit is perfectly workable for the vast majority of users, or at > least it was until the systemd people influenced the "upstreams" to > build in code to fail with sysvinit. > > * sysvinit plus daemontools is perfectly workable for almost all users > who are capable of writing a short run file and creating a symlink. > Daemontools suffers from none of the problems hand-wrung by the > "rfc", and indeed in a daemontools world (or at least as a runit > world which I assume is the same), the only process whose parent is > PID1 is runsvdir (equivalent of daemontools svscanboot). On my > computer, the executables reparented to PID1 are all stuff spawned by > dmenu or UMENU, as well as the gvim executable, which doubleforks > itself automatically. More on this later... > > * sysvinit plus OpenRC is perfectly workable for most users who don't > want daemon respawning. > > * sysvinit plus OpenRC plus daemontools is perfectly workable for users > who want some daemons respawnable. > > * A PID1 consisting of an rc file that somehow respawns the virtual > terminals, background-runs any daemons necessary, and then ends in a > loop that listens for and handles signals to PID1, is perfectly > workable for a person operating a small, special purpose computer. If > I knew how to respawn virtual terminals I'd do just that as a > demonstration, but respawning gettys is incredibly difficult: It > often kills the process that tried to respawn it. > > * The 83 line Suckless Init spawning an rc file is perfectly workable > for someone wanting simplicity and the ultimately simple PID1. > > Bottom line, all this bikeshedding perfectionism is unnecessary unless > you're a big commercial company trying to turn Free Software into a > cash cow by complexifying it. All the problems were solved long ago, or > are easily solvable by simple, user space addons needing no > modification to the likes of sysvinit, daemontools, runit, s6, Epoch, > OpenRC and the like. > > For instance, I have a real problem with zombies and T status processes > created by my use of dmenu and UMENU. It wouldn't be particularly > difficult for me to build a userspace daemontools analog, where one > process parented to PID1 (my analog of svscanboot) would spin around > listening for commands and/or scripts to run in such a way that IT was > the parent, or perhaps via analogs of svscan that don't respawn. I > could then modify dmenu and UMENU to message my daemon in order to run > commands. Notice the idea in this paragraph requires not one > modification to whatever "init system" you're using. And it's simple > enough that even I could implement it, given the time. > > There's no need to search for the perfect init or supervisor. Long ago > we got a bunch of them that are all good enough, and can be combined to > fill almost any need. This continuing search for a perfect init is just > wheel spinning, or perhaps an excuse for profit-driven complexification. Thank you for this eloquent and extensive statement of principle. -- hendrik ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cg
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Le 14/04/2017 à 10:57, Enrico Weigelt, metux IT consult a écrit : * srvmgt_report_state(...) --> report the service state to the supervisor --> states could be eg. * SRVMGT_STATE_STARTUP -- still within the startup phase * SRVMGT_STATE_READY_LOCAL -- ready for local clients only * SRVMGT_STATE_READY_ALL -- ready for all clients * SRVMGT_STATE_BUSY-- too busy to process new requests * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing queued requests * SRVMGT_STATE_DEFERRED-- temporarily can't accept new requests (eg. overload) * SRVMGT_STATE_WAITING -- wait for resource (eg. printer needs paper or ink) * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some fatal error) Reporting state is usefull, but this could be achieved by writing the state in plain text to stdout. It is up to the supervisr to establish a pipe between itself and the daemon's stdout. If there isn't a supervisor, then writing to /dev/null doesn't harm. Or why not write to a file in /run? Anyway, I don't think it deserves to link to a library to perform such a simple thing. Didier ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
> > I wonder why that general task of daemonizing cant just be done in a > > generic separate program and left out of the individual daemons. > > So, everybody can decide on this own how to actually start/manage > > the daemons - some use a supervisor, some just call via a daemonizer > > program from init scripts, etc, etc. Only the program itself[*] knows when the transition between it being started and it being ready has happened. If the program daemonises itself, it can communicate this by having the parent exit at the correct point. [*]For a generic program to understand this, there would have to be some sort of convention (close stderr seems to be sanest I could think of). But no convention has been established yet and we have had some major boneheaded ideas (child processes stops itself when ready (WTF!?), write a magic sequence to stdout (fugly?)). Hiding this behind some library call makes it worse, as it ties things to a particular implementation. The daemontools advocates take the view that this notification when ready isn't strictly needed, as the dependent tasks should be smart enough to retry/persist. They have a point, but it can mask the actual source of failures and isn't workable for hard dependencies (fs not mounted). > > By the way: maybe we should write an RFC draft for the whole issue ... > > Looked for a relevant RFC. But found only this: > https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html > [RFC] [PATCH] notify init daemon when children are reparented to it > > But this doesn't seem to be quite what we want, and I can't say I have > enough context to understand it. It isn't what we want. Sadly some of the statements are outdated or something between misleading or incorrect. For example: *> *but* due to a quirk of POSIX, if it were to be made to open() a tty *> device, it would end up owning it! FAIL. POSIX anticipates this problem and provides a O_NOCTTY to the open call to prevent this problem happening. So NO FAIL. *> If there's two apache2 daemons running (in different chroots, or for *> different IPs or ports?), it [init] doesn't know which of the two died *> because the PID that died is unknown to it. This isn't the case. With a sigaction hander init can retrieve the pid and uid of the exited process. Also outdated because the kernel thesedays allows one to delegate the "default-child-collector" function of init to other processes - that is what the container infrastructure uses. > It seems to follow the practice of creating two processes for a > daemon, a parent and a child, and then orphaning the child. > This is not what we seem to be discussing here, and leads to > init having nontrivial ongoing work. Hmm... it is a bit of a tradeoff. This approach makes it possible to indicate readiness and thus order the system startup properly. It is also needed when random users need to daemonise their tasks (eg screen). On the other hand it makes automated respawning tricky. However it is an open question if well written daemons need to ever respawn. Respawning a crashed daemon favours availability over all other considerations. And uncontrolled respawn is a fork bomb, an oracle for an attacker, a log file filler, a database corruptor, etc, etc. > And although it says [RFC] in the title, it doesn't seem to be an > official RFC. Anybody can make a Request For Comments :-) (The RFCs of the IETF are different, they don't really relate to OS internals) Anyway here is my request for comments: If you want to signal that your daemon is ready close stderr, even if you don't do a fork(). Then the proposed generic wrapper can notice that. regards marc ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
> OK, I completely understand. It makes sense. It's pretty slick, too. I > never would have thought of it. > > Being a daemontools enthusiast, I still don't see fork_parent() as an > asset in late boot. But I have a feeling it, or something similar, > could be an asset in different contexts. As a matter of fact, in my > UMENU program, I might change my backgrounding code to incorporate > that, and perhaps in doing so eliminate some sleep kludges. Yay. I am glad that the explanation was helpful. regards marc ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
In my opinion, this is the best that has been written about init or supervisor, (I'm absolutely agree with this) "There's no need to search for the perfect init or supervisor. Long ago we got a bunch of them that are all good enough, and can be combined to fill almost any need. This continuing search for a perfect init is just wheel spinning, or perhaps an excuse for profit-driven complexification" the search for the perfect dictator is not the great idea, at least that's what I think in the same way that there's not a perfect graphical desktop for all the people, is sure that there's not a perfect init system for all the situations in which you can use a computer the init system must be something that the people can change in an easy way, and so, choose between different options for different situations sometimes I think that the people that is focused in develop software quickly, is the people that search for a perfect init system, so they can develop fast software but fast software is like fast food, I think that freedom is better that fast software José María Ávila On 17.04.2017 03:57, Steve Litt wrote: On Sun, 16 Apr 2017 19:22:36 -0400 Hendrik Boom wrote: On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote: > On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT > consult wrote: > > By the way: maybe we should write an RFC draft for the whole > > issue ... > > Looked for a relevant RFC. But found only this: > https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html > [RFC] [PATCH] notify init daemon when children are reparented to it > > But this doesn't seem to be quite what we want, and I can't say I > have enough context to understand it. That so-calledRFC seems to be very much in the style of requiring the init process to be intimately involved in process supervision actions, which is a strong constraint on what init systems can be chosen. And, I suspect, an init system that is so involved is a potential tool to gradually take over the entire OS, as systemd has done. And, in fact, being written in 2008, that so called rfc might have been either the excuse or the inspiration for systemd. The "rfc" suffers from a perfectionism related to bikeshedding. Heaven forbid there's ever a zombie or a process with a 'T' in the ps status field! We simply MUST make a perfect init or supervisor system: Nothing less will do. And while we're indulging our perfection, it's important to remember that simplicity not be a priority at all. We will remain oblivious to the fact that complexity and lack of encapsulation creates flaws far worse than the flaws we're moving heaven and earth to get rid of. And for gosh sakes, let's forget the facts: THE FACTS * sysvinit is perfectly workable for the vast majority of users, or at least it was until the systemd people influenced the "upstreams" to build in code to fail with sysvinit. * sysvinit plus daemontools is perfectly workable for almost all users who are capable of writing a short run file and creating a symlink. Daemontools suffers from none of the problems hand-wrung by the "rfc", and indeed in a daemontools world (or at least as a runit world which I assume is the same), the only process whose parent is PID1 is runsvdir (equivalent of daemontools svscanboot). On my computer, the executables reparented to PID1 are all stuff spawned by dmenu or UMENU, as well as the gvim executable, which doubleforks itself automatically. More on this later... * sysvinit plus OpenRC is perfectly workable for most users who don't want daemon respawning. * sysvinit plus OpenRC plus daemontools is perfectly workable for users who want some daemons respawnable. * A PID1 consisting of an rc file that somehow respawns the virtual terminals, background-runs any daemons necessary, and then ends in a loop that listens for and handles signals to PID1, is perfectly workable for a person operating a small, special purpose computer. If I knew how to respawn virtual terminals I'd do just that as a demonstration, but respawning gettys is incredibly difficult: It often kills the process that tried to respawn it. * The 83 line Suckless Init spawning an rc file is perfectly workable for someone wanting simplicity and the ultimately simple PID1. Bottom line, all this bikeshedding perfectionism is unnecessary unless you're a big commercial company trying to turn Free Software into a cash cow by complexifying it. All the problems were solved long ago, or are easily solvable by simple, user space addons needing no modification to the likes of sysvinit, daemontools, runit, s6, Epoch, OpenRC and the like. For instance, I have a real problem with zombies and T status processes created by my use of dmenu and UMENU. It wouldn't be particularly difficult for me to build a userspace daemontools analog, where one process parented to PID1 (my analog of svscanboot) would spin around listening for c
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sun, 16 Apr 2017 19:22:36 -0400 Hendrik Boom wrote: > On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote: > > On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT > > consult wrote: > > > By the way: maybe we should write an RFC draft for the whole > > > issue ... > > > > Looked for a relevant RFC. But found only this: > > https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html > > [RFC] [PATCH] notify init daemon when children are reparented to it > > > > But this doesn't seem to be quite what we want, and I can't say I > > have enough context to understand it. > > That so-calledRFC seems to be very much in the style of > requiring the init process to be intimately involved in process > supervision actions, which is a strong constraint on what init > systems can be chosen. And, I suspect, an init system that is so > involved is a potential tool to gradually take over the entire OS, as > systemd has done. And, in fact, being written in 2008, that so called rfc might have been either the excuse or the inspiration for systemd. The "rfc" suffers from a perfectionism related to bikeshedding. Heaven forbid there's ever a zombie or a process with a 'T' in the ps status field! We simply MUST make a perfect init or supervisor system: Nothing less will do. And while we're indulging our perfection, it's important to remember that simplicity not be a priority at all. We will remain oblivious to the fact that complexity and lack of encapsulation creates flaws far worse than the flaws we're moving heaven and earth to get rid of. And for gosh sakes, let's forget the facts: THE FACTS * sysvinit is perfectly workable for the vast majority of users, or at least it was until the systemd people influenced the "upstreams" to build in code to fail with sysvinit. * sysvinit plus daemontools is perfectly workable for almost all users who are capable of writing a short run file and creating a symlink. Daemontools suffers from none of the problems hand-wrung by the "rfc", and indeed in a daemontools world (or at least as a runit world which I assume is the same), the only process whose parent is PID1 is runsvdir (equivalent of daemontools svscanboot). On my computer, the executables reparented to PID1 are all stuff spawned by dmenu or UMENU, as well as the gvim executable, which doubleforks itself automatically. More on this later... * sysvinit plus OpenRC is perfectly workable for most users who don't want daemon respawning. * sysvinit plus OpenRC plus daemontools is perfectly workable for users who want some daemons respawnable. * A PID1 consisting of an rc file that somehow respawns the virtual terminals, background-runs any daemons necessary, and then ends in a loop that listens for and handles signals to PID1, is perfectly workable for a person operating a small, special purpose computer. If I knew how to respawn virtual terminals I'd do just that as a demonstration, but respawning gettys is incredibly difficult: It often kills the process that tried to respawn it. * The 83 line Suckless Init spawning an rc file is perfectly workable for someone wanting simplicity and the ultimately simple PID1. Bottom line, all this bikeshedding perfectionism is unnecessary unless you're a big commercial company trying to turn Free Software into a cash cow by complexifying it. All the problems were solved long ago, or are easily solvable by simple, user space addons needing no modification to the likes of sysvinit, daemontools, runit, s6, Epoch, OpenRC and the like. For instance, I have a real problem with zombies and T status processes created by my use of dmenu and UMENU. It wouldn't be particularly difficult for me to build a userspace daemontools analog, where one process parented to PID1 (my analog of svscanboot) would spin around listening for commands and/or scripts to run in such a way that IT was the parent, or perhaps via analogs of svscan that don't respawn. I could then modify dmenu and UMENU to message my daemon in order to run commands. Notice the idea in this paragraph requires not one modification to whatever "init system" you're using. And it's simple enough that even I could implement it, given the time. There's no need to search for the perfect init or supervisor. Long ago we got a bunch of them that are all good enough, and can be combined to fill almost any need. This continuing search for a perfect init is just wheel spinning, or perhaps an excuse for profit-driven complexification. SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sun, Apr 16, 2017 at 05:04:18PM -0400, Hendrik Boom wrote: > On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT consult > wrote: > > On 15.04.2017 19:50, Steve Litt wrote: > > > > > About my characterizations: "Baroque" is a relative thing. What I wrote > > > was based on "why would you not simply use a process supervisor like > > > systemd?" If a person has a reason not to use such a supervisor, and in > > > fact the whole OpenRC init system seems to be based on such objections > > > to supervisors, then the six system call solution you outline > > > transitions from "a whole bunch of needless stuff" to "hey, that's a > > > pretty darn kool solution." So your solution is baroque only so far as > > > one enjoys using daemontools or similar. > > > > If one doesn't want a supervisor, why not just using something like > > start-stop-daemon ? IIRC, it should handle that quite well. > > > > I wonder why that general task of daemonizing cant just be done in a > > generic separate program and left out of the individual daemons. > > So, everybody can decide on this own how to actually start/manage > > the daemons - some use a supervisor, some just call via a daemonizer > > program from init scripts, etc, etc. > > > > By the way: maybe we should write an RFC draft for the whole issue ... > > Looked for a relevant RFC. But found only this: > https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html > [RFC] [PATCH] notify init daemon when children are reparented to it > > But this doesn't seem to be quite what we want, and I can't say I have > enough context to understand it. That so-calledRFC seems to be very much in the style of requiring the init process to be intimately involved in process supervision actions, which is a strong constraint on what init systems can be chosen. And, I suspect, an init system that is so involved is a potential tool to gradually take over the entire OS, as systemd has done. Of course, even with such an init system, it's possible to use the other mechanisms we have been discussing here to enable specific daemons to ignore the init system. -- hendrik > > It seemms to follow the practice of creating two processes for a > daemon, a parent and a child, and then orphaning the child. > This is not what we seem to be discusison here, and leads to > init having nontrivial ongoing work. > > And although it says [RFC] in the title, it doesn't seem to be an > official RFC. > > -- hendrik > > > > > > > --mtx > > ___ > > Dng mailing list > > Dng@lists.dyne.org > > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng > ___ > Dng mailing list > Dng@lists.dyne.org > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sun, Apr 16, 2017 at 09:59:36PM +0200, Enrico Weigelt, metux IT consult wrote: > On 15.04.2017 19:50, Steve Litt wrote: > > > About my characterizations: "Baroque" is a relative thing. What I wrote > > was based on "why would you not simply use a process supervisor like > > systemd?" If a person has a reason not to use such a supervisor, and in > > fact the whole OpenRC init system seems to be based on such objections > > to supervisors, then the six system call solution you outline > > transitions from "a whole bunch of needless stuff" to "hey, that's a > > pretty darn kool solution." So your solution is baroque only so far as > > one enjoys using daemontools or similar. > > If one doesn't want a supervisor, why not just using something like > start-stop-daemon ? IIRC, it should handle that quite well. > > I wonder why that general task of daemonizing cant just be done in a > generic separate program and left out of the individual daemons. > So, everybody can decide on this own how to actually start/manage > the daemons - some use a supervisor, some just call via a daemonizer > program from init scripts, etc, etc. > > By the way: maybe we should write an RFC draft for the whole issue ... Looked for a relevant RFC. But found only this: https://lists.ubuntu.com/archives/kernel-team/2008-December/003628.html [RFC] [PATCH] notify init daemon when children are reparented to it But this doesn't seem to be quite what we want, and I can't say I have enough context to understand it. It seemms to follow the practice of creating two processes for a daemon, a parent and a child, and then orphaning the child. This is not what we seem to be discusison here, and leads to init having nontrivial ongoing work. And although it says [RFC] in the title, it doesn't seem to be an official RFC. -- hendrik > > > --mtx > ___ > Dng mailing list > Dng@lists.dyne.org > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Enrico Weigelt: ... > If one doesn't want a supervisor, why not just using something like > start-stop-daemon ? IIRC, it should handle that quite well. Why not just start the program and kill it when not needed anymore ? You know, you don't have to have a supervisor. > I wonder why that general task of daemonizing cant just be done in a > generic separate program and left out of the individual daemons. > So, everybody can decide on this own how to actually start/manage > the daemons - some use a supervisor, some just call via a daemonizer > program from init scripts, etc, etc. ... http://software.clapper.org/daemonize/daemonize.html BTW, -f and using daemon() is darn easy to implement. We did have a kind of supervisor in inetd (mostly to save memory and reduce load), but it's use has fallen out of favour. So why do we have to invent new ways of starting and killing programs ? Regards, /Karl Hammar --- Aspö Data Lilla Aspö 148 S-742 94 Östhammar Sweden +46 173 140 57 ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 15.04.2017 19:50, Steve Litt wrote: > About my characterizations: "Baroque" is a relative thing. What I wrote > was based on "why would you not simply use a process supervisor like > systemd?" If a person has a reason not to use such a supervisor, and in > fact the whole OpenRC init system seems to be based on such objections > to supervisors, then the six system call solution you outline > transitions from "a whole bunch of needless stuff" to "hey, that's a > pretty darn kool solution." So your solution is baroque only so far as > one enjoys using daemontools or similar. If one doesn't want a supervisor, why not just using something like start-stop-daemon ? IIRC, it should handle that quite well. I wonder why that general task of daemonizing cant just be done in a generic separate program and left out of the individual daemons. So, everybody can decide on this own how to actually start/manage the daemons - some use a supervisor, some just call via a daemonizer program from init scripts, etc, etc. By the way: maybe we should write an RFC draft for the whole issue ... --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sun, 16 Apr 2017 10:01:02 +0200 marc wrote in response to Steve Litt: > > And therefore I was wondering > > why fork_parent() didn't take a function address as an argument, and > > call that callback function's address where you have the elipses. > > No function pointers needed. After a fork() system call you have two > almost identical copies of the same process. The one which sees a > return code greater than 0 is the parent. The parent instance however > never makes it out of the fork_parent() library call. > > > Yes. It makes sense. My question was where you put the code > > represented by your elipses above. > > The code in the elipses runs outside the fork_parent() call. It could > run in the main() function or elsewhere. OK, I completely understand. It makes sense. It's pretty slick, too. I never would have thought of it. Being a daemontools enthusiast, I still don't see fork_parent() as an asset in late boot. But I have a feeling it, or something similar, could be an asset in different contexts. As a matter of fact, in my UMENU program, I might change my backgrounding code to incorportate that, and perhaps in doing so eliminate some sleep kludges. Thanks, SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Hello > Wait. Are you saying that the way fork_parent() is used is to modify > the source code of fork_parent() itself? No - you just call it, like daemon(). Then later, outside fork_parent() you call close(). > I was under the impression that fork-parent() was a library > call that you made to properly background your source code, which > exists outside of function fork_parent(). That is the correct impression. > And therefore I was wondering > why fork_parent() didn't take a function address as an argument, and > call that callback function's address where you have the elipses. No function pointers needed. After a fork() system call you have two almost identical copies of the same process. The one which sees a return code greater than 0 is the parent. The parent instance however never makes it out of the fork_parent() library call. > Yes. It makes sense. My question was where you put the code represented > by your elipses above. The code in the elipses runs outside the fork_parent() call. It could run in the main() function or elsewhere. > Is fork_parent() a template (in the general > sense, not the C++ sense) that you modify, No, not a template. > or is it a tool you use, as > is, to background code you put in a separate function? It is a library call. regards marc ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sat, 15 Apr 2017 22:50:20 +0200
marc wrote:
> > Concerning the fork-parent.c and fork-parent.h revealed at
> > http://welz.org.za/notes/fork-parent.tar.gz , I can't envision how
> > you'd bind the fork-parent() function to the stuff your daemon
> > actually does, in order to have your daemon properly fork and then
> > terminate the parent when ready for business.
> >
> > I look at the code in fork-parent.c, and it looks to me like all the
> > actual tasks of the daemon should be placed immediately above the
> > child's closing p[1], so p[1] is closed when the daemon is ready.
> > But in fact, fork-parent() has no interface to the real work of the
> > daemon, as far as I can see.
>
> So fork_parent() can be seen as a minor extension of the
> daemon(3) library call: In both cases after you have called it
> your function is now running in the child process. The parent
> process stays "stuck" in the fork_parent() or daemon()
> functions.
>
> Note that there are "extra" close() calls in the fork_parent()
> code, as after the fork() both ends of the pipe are visible
> in both processes. Without closing the "other" end in each
> process a read would not return EOF and the parent would stay
> up even after the child has closed its end.
>
> Here pseudocode, commented differently:
>
> pipe() /* done in fork_parent */
> if(fork() > 0){ /* in fork_parent */
> close() /* in fork_parent, not the stderr of the child
> */ read()/* in fork_parent */
> exit()/* in fork_parent */
> /* parent no longer exists beyond this point */
> /* this point in code, not in time */
> }
>
> /* here we are in the child, the parent is still running */
> /* here do all the complicated setup routes, eg */
> /* check databases, bind ports */
> ...
Wait. Are you saying that the way fork_parent() is used is to modify
the source code of fork_parent() itself? If that's the case, I
understand. I was under the impression that fork-parent() was a library
call that you made to properly background your source code, which
exists outside of function fork_parent(). And therefore I was wondering
why fork_parent() didn't take a function address as an argument, and
call that callback function's address where you have the elipses.
> /* when everything is ready, you do: */
>
> close(STDERR_FILENO);
Yes. It makes sense. My question was where you put the code represented
by your elipses above. Is fork_parent() a template (in the general
sense, not the C++ sense) that you modify, or is it a tool you use, as
is, to background code you put in a separate function?
SteveT
Steve Litt
April 2017 featured book: Troubleshooting Techniques
of the Successful Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
> Concerning the fork-parent.c and fork-parent.h revealed at
> http://welz.org.za/notes/fork-parent.tar.gz , I can't envision how
> you'd bind the fork-parent() function to the stuff your daemon actually
> does, in order to have your daemon properly fork and then terminate the
> parent when ready for business.
>
> I look at the code in fork-parent.c, and it looks to me like all the
> actual tasks of the daemon should be placed immediately above the
> child's closing p[1], so p[1] is closed when the daemon is ready. But
> in fact, fork-parent() has no interface to the real work of the daemon,
> as far as I can see.
So fork_parent() can be seen as a minor extension of the
daemon(3) library call: In both cases after you have called it
your function is now running in the child process. The parent
process stays "stuck" in the fork_parent() or daemon()
functions.
Note that there are "extra" close() calls in the fork_parent()
code, as after the fork() both ends of the pipe are visible
in both processes. Without closing the "other" end in each
process a read would not return EOF and the parent would stay
up even after the child has closed its end.
Here pseudocode, commented differently:
pipe() /* done in fork_parent */
if(fork() > 0){ /* in fork_parent */
close() /* in fork_parent, not the stderr of the child */
read()/* in fork_parent */
exit()/* in fork_parent */
/* parent no longer exists beyond this point */
/* this point in code, not in time */
}
/* here we are in the child, the parent is still running */
/* here do all the complicated setup routes, eg */
/* check databases, bind ports */
...
/* when everything is ready, you do: */
close(STDERR_FILENO);
/* at this point the parent exits (see above) */
/* and the rcS scripts know to start the dependencies */
It turns out that the same basic effect can be achieved with a
pause() and kill(getppid()), but that makes it tricky to relay
errors like "-x requires a parameter" to stderr, where usage
errors should be reported.
regards
marc
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sat, 15 Apr 2017 08:14:48 +0200 marc wrote: > I have written up the details at > http://welz.org.za/notes/on-starting-daemons.html Concerning the fork-parent.c and fork-parent.h revealed at http://welz.org.za/notes/fork-parent.tar.gz , I can't envision how you'd bind the fork-parent() function to the stuff your daemon actually does, in order to have your daemon properly fork and then terminate the parent when ready for business. I look at the code in fork-parent.c, and it looks to me like all the actual tasks of the daemon should be placed immediately above the child's closing p[1], so p[1] is closed when the daemon is ready. But in fact, fork-parent() has no interface to the real work of the daemon, as far as I can see. So how does one bolt fork-parent() on to his or her daemon code to enable it with proper backgrounding and timing? Thanks, SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sat, 15 Apr 2017 13:50:22 -0400 Steve Litt wrote: Oops. Concerning my statement: > What I > wrote was based on "why would you not simply use a process supervisor > like systemd?" s/systemd/daemontools/ SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sat, 15 Apr 2017 19:12:31 +0200 marc wrote: > I appreciate that Steve likes the daemontools approach where the > server process stays in the foreground, and I have no quarrel with > the advice of including a -f foreground option... The daemontools > approach makes a completely different set of tradeoffs - some > people really like them, others consider them to be the lunatic > fringe. To each their own. > > But for a classic sysvinit startup path, the above logic > will do a decent job of signalling that a process is ready to > serve requests. And has been an appropriate mechanism for > decades. No need for yet another API. I'll admit this: For the person not wanting to use either a daemontools-inspired process supervisor, nor systemd (which I understand has the same capability as daemontools to background a foreground process), the fork-early, parent blocks, kid does the work, kid messages (and in this case the message is just closing the pipe), parent unblocks and exits is an excellent dependency tool, assuming all daemons do it, do it right, and do it accurately. About my characterizations: "Baroque" is a relative thing. What I wrote was based on "why would you not simply use a process supervisor like systemd?" If a person has a reason not to use such a supervisor, and in fact the whole OpenRC init system seems to be based on such objections to supervisors, then the six system call solution you outline transitions from "a whole bunch of needless stuff" to "hey, that's a pretty darn kool solution." So your solution is baroque only so far as one enjoys using daemontools or similar. LOL, as far as subtle, I'm just not seeing subtle in your solution. The implementation is elegant, but is the solution subtle? Not in my opinion. Then again, I don't think any software I ever wrote was subtle. I have some technical questions about your software, but that's for another email. SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Hello again Steve
> > The TLDR is that "A good daemon should fork soon, but have the
> > parent process exit only once the child is ready to service requests"
> >
> > Sadly it seems too subtle for many people and the concept goes
> > un-noticed ...
>
> It's as subtle as a 10 megaton nuclear device, and goes un-noticed
> because most of us who know of it hope it will be forgotten, so we
> don't speak of it.
>
> Read http://welz.org.za/notes/on-starting-daemons.html. According to
> this document, quite apart from the daemon's real function, the daemon
> must fork itself early, form a pipeline between parent and child, and
> when the child believes it's ready to function completely, it sends a
> message to the parent, who upon receipt of the message exits. That's
> not subtle: It's quite baroque. Your idea of canning it into a neat
> little tool is quite elegant, but it's an elegant implementation of
> something baroque.
Baroque, 10 megaton nuclear device, eh ? It is (error checking/reporting
elided) an effort of 6 system calls:
pipe()
if(fork() > 0){ /* if we are in parent */
close() /* close the child end of the pipe */
read()/* wait for child */
exit()/* exit the parent process */
}
... /* do whatever is need to bring up the daemon */
close()/dup2() /* no message required, Steve misunderstood that */
If these system calls make you hide in a nuclear fallout shelter
with your powdered white wig, one worries what will happen if somebody
tells you what gethostbyname() does.
Or even worse have you straced python recently ? It does 800+ system
calls before it even gets going ... and it only goes downhill from
there.
I appreciate that Steve likes the daemontools approach where the
server process stays in the foreground, and I have no quarrel with
the advice of including a -f foreground option... The daemontools
approach makes a completely different set of tradeoffs - some
people really like them, others consider them to be the lunatic
fringe. To each their own.
But for a classic sysvinit startup path, the above logic
will do a decent job of signalling that a process is ready to
serve requests. And has been an appropriate mechanism for
decades. No need for yet another API.
regards
marc
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 15.04.2017 00:04, Daniel Abrecht wrote: > I still think this isn't a problem the service manager should attempt to > solve. This is a situation where the database is temporary unavailable, > for which there are many possible reasons. The services which relay on > the database should be able to handle such a situation gracefully, or > they are just not yet stable enough. In theory, you're right. But the actual practise tends to be very different. Probably, databases aren't the prime usecase for that. There could be other scenarios, where a service has to do some preparations for others to work (maybe certain filesystems mounted, etc). I remember, eg., problems w/ nfs, when one of the dozens servers isn't up yet, so they needed to be started in specific order. Years went by since I last touched that, no idea that's still the case. --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 15.04.2017 03:17, Steve Litt wrote: > Sounds like a good idea, but it's not, for the following reasons: > > 1) There will be endless arguments about HOW each and every daemon will >let its init system know "I'm now ready for business". The number of >different ways will make the number of incompatible Unixes of 1985 >seem trivial to reconcile. Exactly the reason why I'd put it an separate library - so it can be easily replaced (even on binary level), w/o ever having to touch the application itself. It will be the operator's choice, which one shall be used. The whole problem arised w/ upstreams wanting such an mechanism, and nobody there who gagged up Lennart quick enough ... So, what do we make out of that now ? a) patch behind all the upstreams all that thoughless upstreams for all eternity b) maintain an dummy drop-in replacement (and keep up w/ everyting that the Lennartists come up) for all eternity c) provide a simple and stable API for exactly the root problem, (and convince upstreams to use that instead of Lennartware) and everybody can go its own way in piece. > 1.5) The resource daemon's opinion of being "ready for business" might > be erroneous, or not reflect how the user daemon will use the > resource daemon. Such things must be defined (on per service level) anyways - we'll never get around that. All we can do is try to standardize certain scenarios. For start, I'd define "READY" as fully operational. If some daemon provides several services, which be up/down independently (which I'd consider bad design), we could introduce sub-services (eg. apache might serve http as well as ftp) - would just add another const char* parameter. > 2) The connected and powerful will unilaterally declare one method of >"phoning home" is the one true way, thereby driving compatability >wedges between various software, and facilitating the defacto >banning of some daemons from a distro. That something we do not want. Actually the (de)potterization is one aspect of that. > 3) If the "one true way" is complex, for instance, if it has to >intimately mesh with dbus (sy what?), small daemon projects >will be ostricized out of the community. There doesn't need to be "one true way". It's only for a specific range of scenario: let the daemon just shout out it's overall status (what's required for startup ordering, and similar stuff, eg. some *trivial* monitoring) to anybody who's interested in it - the exact definition of that "anybody" would be confined to the specific implementation of the library - the application has no idea about that, and shouldn't even care whether somebody's listening. So, again, in the trivial case that would just be an dummy. If the library also maintains pidfiles (which I'd recommend, so we now have that stuff confined in one place), the lib of course has to implement that. But whether anybody actually reacts on the service notifies should be completely optional. As an application developer, I can just call that library, and dont have to care what happens then (okay, I'll have to consider that it might a few cycles in there). > 4) The problem this backward communication purports to solve was solved >around the time the Western Roman Empire was conquored by the >Barbarians. You mean, back then when the fascists just killed everybody who didn't submit to their rule and destroyed freedom (we still suffer from the roman law) ? ... >Regardless of your init system, run your daemons from >daemontools, and have your daemontools run scripts refuse to start >unless a test of the needed resource indicates it's working. >Whatever init system you're using, it's easy to get Daemontools, or >preferably Daemontools-Encore, running.[1] How exactly does it get that notification from the application ? Or does it go the other way round, having - per application - monitoring programs, that try to figure out whether the service is up and signals that to the rest ? Again: application developers want an interface for sending such notifies to the init system / service monitor. Whether you, or me, or anybody else, is actually using that, isn't in their scope - they just want that interface, therefore they're using the only thing they know, which happens to be Lenntart's proprietary one. So, let's just give them what they want, in a way that doesn't cause us any pain. > A word about the source of this thread: A guy, whom I /dev/nulled over a > year ago for making very disturbing political posts in response to > technical discussions, magically woke up a few days ago and started > resurrecting all sorts of dead threads. You're not talking about me, don't you ? (you wouldn't reply my mails then, right ?) > My /dev/null log tells me he's > sent 20 or 30 emails. I don't begin to know what causes this person to > act this way, but he's not a friend of our community, and we might want > to think twice about respond
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 15.04.2017 11:42, Steve Litt wrote: > Once upon a time, daemon self-backgrounding was necessary, and this nohup (or something similar) was not sufficient ? I don't know when start-stop-daemon was incarnated (related to daemontools ?), but IMHO it seems to handle the daemonizing quite well. IMHO, a daemon should not daemonize itself, but always leave that to the some supervisor or startup tool. And if it can handle one connection per process, it should even leave socket listening to something inted-alike. Even, for multi-connection clients, opening the listener socket should be left to it. (people can still optionally provide wrappers that glue everything together to an completely-standalone server machinery, for convenience). --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Sat, 15 Apr 2017 08:14:48 +0200 marc wrote: > > > Go on down your path, but I suspect not many people would cheer > > > at you in this camp... > > > > I can see the merit - on two points, technical and political. > > > > > On the technical side, I can see the usefulness of a system > > wide standardised service status reporting - making it easy > > for one process to see if a service it depends on is actually > > running and ready (as opposed to, "has started"). I have a > > customer system I've inherited where it regularly fails to > > startup properly because Asterisk starts before MySQL has > > finished starting up. > > So it turns out that there exists an subtle yet elegant mechanism > for a process to report that it is ready. It has been in use > for decades - daemons as old as sysklog from the previous century > use it. > > I have written up the details at > http://welz.org.za/notes/on-starting-daemons.html Danger Will Robinson! If you write a daemon that backgrounds itself, whether early or late, whether parent and child communicate via signals or pipes or, well, what the heck, network sockets, be ABSOLUTELY sure to include a command line option to tell your daemon NOT to background itself, but to stay in the foreground all the time. If your daemon cannot stay in the foreground, your daemon is incompatible with runit, s6, perp and daemontools. And this foreground option should be separate from any "debugging" options, because nobody wants production logs to have debugging output. > > The TLDR is that "A good daemon should fork soon, but have the > parent process exit only once the child is ready to service requests" > > Sadly it seems too subtle for many people and the concept goes > un-noticed ... It's as subtle as a 10 megaton nuclear device, and goes un-noticed because most of us who know of it hope it will be forgotten, so we don't speak of it. Read http://welz.org.za/notes/on-starting-daemons.html. According to this document, quite apart from the daemon's real function, the daemon must fork itself early, form a pipeline between parent and child, and when the child believes it's ready to function completely, it sends a message to the parent, who upon receipt of the message exits. That's not subtle: It's quite baroque. Your idea of canning it into a neat little tool is quite elegant, but it's an elegant implementation of something baroque. It's also not a good solution. It won't work unless: 1) ALL daemons act this way 2) All daemons accurately time parent destruction to full readiness 3) Nobody uses daemontools, runit, s6, perp, etc. Once upon a time, daemon self-backgrounding was necessary, and this fork-early, parent terminate on readiness was state of the art and the Cadillac of the industry. But then, in 2001, daemontools was invented, making self-backgrounding unnecessary. From 2001 on, a daemon author could write a daemon as a normal foreground program, confident that daemontools, or one of its descendents (runit, s6, perp, daemontools-encore) would handle all aspects of getting it into the background while still supervising it, without the need of a PID file. Because daemontools, daemontools-encore, runit, s6 (and I would guess) perp can run as mere process supervisors on any arbitrary init system, the daemontools method of supervising foreground daemons is available to anyone willing to install daemontools, and installing daemontools is trivial. Anyway, by far the most important thing I said in this email is this: If you insist on making your daemon self-background, for gosh sakes give it a command line option to stay in the foreground, and make that command line option quite apart from any option that increases debugging output. SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Technologist http://www.troubleshooters.com/techniques ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
> > Go on down your path, but I suspect not many people would cheer at you > > in this camp... > > I can see the merit - on two points, technical and political. > > > On the technical side, I can see the usefulness of a system > wide standardised service status reporting - making it easy > for one process to see if a service it depends on is actually > running and ready (as opposed to, "has started"). I have a > customer system I've inherited where it regularly fails to > startup properly because Asterisk starts before MySQL has > finished starting up. So it turns out that there exists an subtle yet elegant mechanism for a process to report that it is ready. It has been in use for decades - daemons as old as sysklog from the previous century use it. I have written up the details at http://welz.org.za/notes/on-starting-daemons.html The TLDR is that "A good daemon should fork soon, but have the parent process exit only once the child is ready to service requests" Sadly it seems too subtle for many people and the concept goes un-noticed ... regards marc ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Fri, 14 Apr 2017 21:16:18 +0100 Simon Hobson wrote: > "Enrico Weigelt, metux IT consult" wrote: > > >> For those of us who put consistency above boot speed, simply > >> changing the init script so MySQL doesn't flag as "started" until > >> the daemon is up and ready to accept requests would fix it; > > > > But then you'll have kind of daemon who watches mysql until it's > > really ready and then signals back to the service monitor, so it > > can proceed with the other services. In the long run, sounds like a > > maintenance hell to me. > > Which is why I said I can see some value in a simple > communication/notification system. Ideally you'd have MySQL itself > make status calls along the lines of "I'm starting but not ready" and > hopefully end with "I'm now ready for business". Sounds like a good idea, but it's not, for the following reasons: 1) There will be endless arguments about HOW each and every daemon will let its init system know "I'm now ready for business". The number of different ways will make the number of incompatible Unixes of 1985 seem trivial to reconcile. 1.5) The resource daemon's opinion of being "ready for business" might be erroneous, or not reflect how the user daemon will use the resource daemon. 2) The connected and powerful will unilaterally declare one method of "phoning home" is the one true way, thereby driving compatability wedges between various software, and facilitating the defacto banning of some daemons from a distro. 3) If the "one true way" is complex, for instance, if it has to intimately mesh with dbus (sy what?), small daemon projects will be ostricized out of the community. 4) The problem this backward communication purports to solve was solved around the time the Western Roman Empire was conquored by the Barbarians. Regardless of your init system, run your daemons from daemontools, and have your daemontools run scripts refuse to start unless a test of the needed resource indicates it's working. Whatever init system you're using, it's easy to get Daemontools, or preferably Daemontools-Encore, running.[1] A word about the source of this thread: A guy, whom I /dev/nulled over a year ago for making very disturbing political posts in response to technical discussions, magically woke up a few days ago and started resurrecting all sorts of dead threads. My /dev/null log tells me he's sent 20 or 30 emails. I don't begin to know what causes this person to act this way, but he's not a friend of our community, and we might want to think twice about responding to him, even in the few cases where he has a good idea. [1] To run daemontools-encore, download the lastest tarball, untar it, modify its conf-* files to reflect correct directories and compile commands (usually unnecessary if you're using gcc on Linux), make, become root, make install, make the repository for daemontools daemons (I used /etc/daemontools/service), make the symlink directory (I used /var/daemontools/service), then edit the svscanboot file to reflect the actual directories that are being used, run svscanboot, and then make run files under the repository and symlink them to the symlink directory to install daemons. This way, no matter what init system you're using, you have a complete supervisor for daemons you believe need to be supervised. And pay no attention to the guy on Debian-User who tells you not to use the svscanboot file to instantiate your daemontools system.[2] [2] Recently a guy posted on Debian-User that you shouldn't use the svscanboot shellscript to start up daemontools. He states a few edge-case situations where svscanboot has bugs, most of which have nothing to do with Linux or normal Intel/AMD hardware. He links to a few people who have run daemontools by doing various init-foo instead of just spawning svscanboot. He just loves to hear himself hold forth on all things technical, but what he forgets to tell his audience is: 1) svscanboot is by far the simplest way to run daemontools. 2) djb created svscanboot, and although this guy holding forth is very smart (IMHO in a Poettering sort of way), if you're going to criticize djb on technical grounds, your name had better be Einstein or Hawking. 3) Differently named copies of svscanboot, each with its own directories, can implement some degree of startup order in daemontools, even though daemontools is widely regarded as having indeterminate start order. 4) If one were to run into one of those edge cases where a daemontools system has bugs if started with svscanboot, they might be fixable by changing the svscanboot shellscript, and that would be infinitely preferable than all sorts of init-foo. SteveT Steve Litt April 2017 featured book: Troubleshooting Techniques of the Successful Techno
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 2017-04-14 20:16, Simon Hobson wrote: > "Enrico Weigelt, metux IT consult" wrote: > >>> For those of us who put consistency above boot speed, simply changing >>> the init script so MySQL doesn't flag as "started" until the daemon >>> is up and ready to accept requests would fix it; >> But then you'll have kind of daemon who watches mysql until it's really >> ready and then signals back to the service monitor, so it can proceed >> with the other services. In the long run, sounds like a maintenance >> hell to me. > Which is why I said I can see some value in a simple > communication/notification system. Ideally you'd have MySQL itself make > status calls along the lines of "I'm starting but not ready" and hopefully > end with "I'm now ready for business". The latter would then allow other > processes that depend on a working database to continue. > At present, IIRC the script starts the MySQL process, then loops round > waiting for to appear - at which point it assumes MySQL is ready > for action and prints OK on the console. > > > That way, there's a standard way for processes to communicate their status, > and a simple and standard way for dependent processes to determine if > something they need is ready yet. I still think this isn't a problem the service manager should attempt to solve. This is a situation where the database is temporary unavailable, for which there are many possible reasons. The services which relay on the database should be able to handle such a situation gracefully, or they are just not yet stable enough. I would expect a service which has to handle a task for which it needs the database to return an error in this situation, but to continue working. Once the database is available, and a new task arrives which requires the database, the service should work as expected. If such a service needs to do some initialization using the database, it can just re-queue that until the first successful database connection. This way, there isn't even a need to start the database first, the service would be rock solid, and there wouldn't be any need for the service manager to handling such edge cases. Just keep it simple. signature.asc Description: OpenPGP digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
"Enrico Weigelt, metux IT consult" wrote: >> For those of us who put consistency above boot speed, simply changing >> the init script so MySQL doesn't flag as "started" until the daemon >> is up and ready to accept requests would fix it; > > But then you'll have kind of daemon who watches mysql until it's really > ready and then signals back to the service monitor, so it can proceed > with the other services. In the long run, sounds like a maintenance > hell to me. Which is why I said I can see some value in a simple communication/notification system. Ideally you'd have MySQL itself make status calls along the lines of "I'm starting but not ready" and hopefully end with "I'm now ready for business". The latter would then allow other processes that depend on a working database to continue. At present, IIRC the script starts the MySQL process, then loops round waiting for to appear - at which point it assumes MySQL is ready for action and prints OK on the console. That way, there's a standard way for processes to communicate their status, and a simple and standard way for dependent processes to determine if something they need is ready yet. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Enrico Weigelt: > On 14.04.2017 12:06, k...@aspodata.se wrote: > > Enrico Weigelt: > > ... > >> Let's just take some example: libsrvmgt with funcs like that: > >> > >> * srvmgt_daemonize() > >> --> detach from controlling terminal, etc > > > > Why do any monitor program need to know if the program has detached > > or not, the only thing it needs to know is the pid and the state, > > which would/could be provided by the *_state() functions, > > or am I wrong ? > > That function should also do the daemonizing, so it doesn't need to do > be implemented individually anymore. What you are requesting is that developers should learn a new api for the same end as the old, I won't support that. What is wrong with daemon(3) ? > BTW: I'd add pidfile handling here, so it's just done once and for all. > Reduces repeated code between individual packages and gives dists a > central hook to do whatever tweaks they might wanna do. Might also come > handy when somebody wants to have per-user services. I'd say that the individual program has no need nor any interest of any pid file, it will be running just as fine without it. A pid file is only in the interest of a monitoring program, so why not let the monitor create it for it's own use, why clutter all idividual programs with it. And, you didn't answer the question, is there anything besides the pid and the state that a monitoring program needs to know ? > >> * srvmgt_droppriv(...) > >> --> drop root privileges (if we are still root) > >> --> several versions, eg. with fetching the target uid/gid from env > > > > Given the pid, it isn't that hard for a monitor to find the uid/gid of > > the process, why has this have to go through the monitor ? > > As above. It's not just notifying, it's doing that. If somebody wants > an explicit notification for whatever reason (maybe just as simple as > a syslog message), he can do it there, instead of within the > individal application. If you "do it there", you are actually doing it within the individual program, and if you are so fond of replacing standard libc calls, why don't you go the fakeroot route: fakeroot works by replacing the file manipulation library functions (chmod(2), stat(2) etc.) by ones that simulate the effect the real library functions would have had, had the user really been root. These wrapper functions are in a shared library /usr/lib/libfakeroot.so* which is loaded through the LD_PRELOAD mechanism of the dynamic loader. (See ld.so(8)) Then you won't need to change any program, your monitor just provides a libc wrapper containing the wrappers for whatever functions it fancy monitoring. Why do I have to modify "my" source code just to please a monitor ? ... > > Given the choises given, it seems that the target of the monitor is > > network servers. Couldn't the monitor find out the ready_local, > > ready_all and shutdown by itself by monitoring which ports are open ? > > That's not easy - the monitor would need a lot of application specific > handling. For example sendmail doesn't close the inbound socket if it > just cant accept new mails temporarily - instead it just tells that > within the smtp handshake. Well there you have your service notification, it is already implemented. Why don't you use it ? And if you really care about the mail server, you look in its logs. And, in parenting a child, you always need to know something about the child. > My goal here is to have an simple notification for simple scenarios, > where you don't have a full-blown network monitoring (eg. nagios). > The whole thing could also be useful for per-user services that might > not even be accessible from the outside. So what are thoose scenarios, if you really want monitoring you could use simple setups of mon or nagios. At the other end is: just start the program, if it dies, you investigate and complain to upstream. > > What's the difference between busy and deferred, seems to be the same > > thing. > > A little bit: busy means waiting for some external resource (eg. > printer ran out of paper), while deferred is due some load limit > (too high load, disk full, ...). Maybe we'd have to find better names. > > > And, what if the monitor cannot trust the program it monitors ? > > It's meant to be trusted (regarding security, etc) - it's just for > easier notification, so that user/operator has at least an idea what's > going on, and allows the monitor to defer startup of other depending > services, etc. So, this is only about staggered startup at root ? Why don't other programs just sleep() and retry till the problem is gone. You have the same problem if your db is on one box, you dns server is on another, etc. Or do you propose that the monitor should run multihost ? > > But what if if we don't know if we can trust the walues provided by > > the program, the we have to check what the program is really doing, > > and then the lib have no value, isn't that so ? > > That would be sc
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Hi From my point of view, systemd always tries to keep services running, no matter how hard they fail, and to mask possible problems when starting a service, so the service maintainers don't have to fix their service, which is really unfortunate. In case of those service state notifications with sd_notify, I think they are usually used to signal when a service is starting, but not ready yet. This may seam reasonable at the beginning, but I think it fixes the problem at the wrong place; When a service needs another service, but it's temporary unavailable, it should cause an error or warning to be returned and logged, but it should never be a fatal error which causes the service to stop. When a service needs to handle this situation on it's own, it will become more stable over time. If this task is taken away from it, it will become fragile, and relay on the service manager. I think this would be really dangerous, because it could lead to cascades of failing services once one service fails. At some point, a service manager couldn't possibly handle that anymore and there would just be waves of restarting and failing processes. Daniel Abrecht signature.asc Description: OpenPGP digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 14.04.2017 12:17, Simon Hobson wrote: > For those of us who put consistency above boot speed, simply changing > the init script so MySQL doesn't flag as "started" until the daemon > is up and ready to accept requests would fix it; But then you'll have kind of daemon who watches mysql until it's really ready and then signals back to the service monitor, so it can proceed with the other services. In the long run, sounds like a maintenance hell to me. Having an simple status reporting, an application like mysql could at least tell "hey, I'm not ready yet - give a coffee break". Whether to actually use it, should be the decision of the operator (and distro should give a proper default) --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 14.04.2017 12:29, Simon Hobson wrote: > But I also recognise that others aren't happy and if they want to > use something else then that's OK by me - as long as what they > propose isn't something that "infects" stuff I want to run under > SysVInit. Exactly my goal. sysvinit users would probably just have the minimal version installed, that doesn't do more than a tiny bit syslog()'ing, others may install more complex versions fitting the service monitor of their choice. --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 14.04.2017 12:06, k...@aspodata.se wrote: > Enrico Weigelt: > ... >> Let's just take some example: libsrvmgt with funcs like that: >> >> * srvmgt_daemonize() >> --> detach from controlling terminal, etc > > Why do any monitor program need to know if the program has detached > or not, the only thing it needs to know is the pid and the state, > which would/could be provided by the *_state() functions, > or am I wrong ? That function should also do the daemonizing, so it doesn't need to do be implemented individually anymore. BTW: I'd add pidfile handling here, so it's just done once and for all. Reduces repeated code between individual packages and gives dists a central hook to do whatever tweaks they might wanna do. Might also come handy when somebody wants to have per-user services. >> * srvmgt_droppriv(...) >> --> drop root privileges (if we are still root) >> --> several versions, eg. with fetching the target uid/gid from env > > Given the pid, it isn't that hard for a monitor to find the uid/gid of > the process, why has this have to go through the monitor ? As above. It's not just notifying, it's doing that. If somebody wants an explicit notification for whatever reason (maybe just as simple as a syslog message), he can do it there, instead of within the individal application. >> --> states could be eg. >> * SRVMGT_STATE_STARTUP -- still within the startup phase >> * SRVMGT_STATE_READY_LOCAL -- ready for local clients only >> * SRVMGT_STATE_READY_ALL -- ready for all clients >> * SRVMGT_STATE_BUSY-- too busy to process new requests >> * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing >> queued requests >> * SRVMGT_STATE_DEFERRED-- temporarily can't accept new >> requests (eg. overload) >> * SRVMGT_STATE_WAITING -- wait for resource (eg. printer >> needs paper or ink) >> * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some >> fatal error) > > Given the choises given, it seems that the target of the monitor is > network servers. Couldn't the monitor find out the ready_local, > ready_all and shutdown by itself by monitoring which ports are open ? That's not easy - the monitor would need a lot of application specific handling. For example sendmail doesn't close the inbound socket if it just cant accept new mails temporarily - instead it just tells that within the smtp handshake. My goal here is to have an simple notification for simple scenarios, where you don't have a full-blown network monitoring (eg. nagios). The whole thing could also be useful for per-user services that might not even be accessible from the outside. > What's the difference between busy and deferred, seems to be the same > thing. A little bit: busy means waiting for some external resource (eg. printer ran out of paper), while deferred is due some load limit (too high load, disk full, ...). Maybe we'd have to find better names. > And, what if the monitor cannot trust the program it monitors ? It's meant to be trusted (regarding security, etc) - it's just for easier notification, so that user/operator has at least an idea what's going on, and allows the monitor to defer startup of other depending services, etc. > But what if if we don't know if we can trust the walues provided by > the program, the we have to check what the program is really doing, > and then the lib have no value, isn't that so ? That would be scenarios where any kind of notification to the service monitor would be useless. My point is that there are indeed some scenarios where notification can be helpful, and that's only what that lib is for. --mtx ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
k...@aspodata.se wrote: > Given the choises given, it seems that the target of the monitor is > network servers. Couldn't the monitor find out the ready_local, > ready_all and shutdown by itself by monitoring which ports are open ? ... > And, what if the monitor cannot trust the program it monitors ? > > Given the lib suggestion, it seems that we are trusting the program. > But what if if we don't know if we can trust the walues provided by > the program, the we have to check what the program is really doing, > and then the lib have no value, isn't that so ? Well there isn't really any easy way round the "don't trust my programs" problem. Your suggestion of monitoring open ports doesn't get round it - what if a program opens up port 80 but just "does nothing" with any connections it gets ? One way you see that "port 80 is open" and assume the service is up, the other way you get a notification that the service is up and assume that it's up. I really don't see how you can work around that at the supervisor level. Also, if you are going to detect a service as up just because it's opened one or more ports, that means the supervisor needs to know in detail what each program does. That goes against all good practice in terms of decoupling functions and leads to the systemd problem of massively interlinking stuff that doesn't need to be linked. Not to mention that there may well be services where a port (or socket) is opened before the service is actually ready to accept requests. So then you are into the supervisor having to know how to talk to each service so as to be able to make a request and see if it gets a sensible answer. But then I'm one of those generally happy with SysVInit style booting. It's easy to understand, it's generally reliable, it's fairly easy to debug if something isn't right. But I also recognise that others aren't happy and if they want to use something else then that's OK by me - as long as what they propose isn't something that "infects" stuff I want to run under SysVInit. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
KatolaZ wrote: >> For start, we'd just write a small library, that logs to syslog, >> perhaps maintains some pidfiles (maybe even a *compile-time* option >> to route directly to libsystemd), then patch up packages that currently >> use libsystemd to use our new one. >> > > I personally don't see why one would like to redo libsystemd0 from > scratch, as you seem so kee of doing. > > Go on down your path, but I suspect not many people would cheer at you > in this camp... I can see the merit - on two points, technical and political. On the technical side, I can see the usefulness of a system wide standardised service status reporting - making it easy for one process to see if a service it depends on is actually running and ready (as opposed to, "has started"). I have a customer system I've inherited where it regularly fails to startup properly because Asterisk starts before MySQL has finished starting up. For those of us who put consistency above boot speed, simply changing the init script so MySQL doesn't flag as "started" until the daemon is up and ready to accept requests would fix it; I can see how many would love to be able to have each init script just start and do a "wait for service X to be ready" step, so that init could just fire up all the scripts in parallel and they'd start as each dependency reached the available state. On the political side, when the Poetteristas say how horrible startup scripts are, and how they don't support parallel/dependency startup - there's be the response that "actually, you don't need all that furball of crap called SystemD because this simple, clean, minimally interlinked, API does it". ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
Enrico Weigelt: ... > Let's just take some example: libsrvmgt with funcs like that: > > * srvmgt_daemonize() > --> detach from controlling terminal, etc Why do any monitor program need to know if the program has detached or not, the only thing it needs to know is the pid and the state, which would/could be provided by the *_state() functions, or am I wrong ? > * srvmgt_droppriv(...) > --> drop root privileges (if we are still root) > --> several versions, eg. with fetching the target uid/gid from env Given the pid, it isn't that hard for a monitor to find the uid/gid of the process, why has this have to go through the monitor ? Also, given that you can do the above without this lib opens gives the program the option too cheat, say, if the monitor wishes to have a tight control of the process. A reliable (well, we have to trust the kernel) way to get the pid and uid of a sender is through signals, are there any other ways ? > * srvmgt_report_state(...) --> report the service state to the supervisor ... There could be something like *_a_would_like_to_be_monitored() and *_I_wish_to_regain_my_free_will(). > --> states could be eg. > * SRVMGT_STATE_STARTUP -- still within the startup phase > * SRVMGT_STATE_READY_LOCAL -- ready for local clients only > * SRVMGT_STATE_READY_ALL -- ready for all clients > * SRVMGT_STATE_BUSY-- too busy to process new requests > * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing > queued requests > * SRVMGT_STATE_DEFERRED-- temporarily can't accept new > requests (eg. overload) > * SRVMGT_STATE_WAITING -- wait for resource (eg. printer > needs paper or ink) > * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some > fatal error) Given the choises given, it seems that the target of the monitor is network servers. Couldn't the monitor find out the ready_local, ready_all and shutdown by itself by monitoring which ports are open ? What's the difference between busy and deferred, seems to be the same thing. /// And, what if the monitor cannot trust the program it monitors ? Given the lib suggestion, it seems that we are trusting the program. But what if if we don't know if we can trust the walues provided by the program, the we have to check what the program is really doing, and then the lib have no value, isn't that so ? Regards, /Karl Hammar --- Aspö Data Lilla Aspö 148 S-742 94 Östhammar Sweden +46 173 140 57 ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On 14-04-17 11:20, KatolaZ wrote: On Fri, Apr 14, 2017 at 10:57:01AM +0200, Enrico Weigelt, metux IT consult wrote: [cut] * srvmgt_daemonize() --> detach from controlling terminal, etc * srvmgt_droppriv(...) --> drop root privileges (if we are still root) --> several versions, eg. with fetching the target uid/gid from env * srvmgt_report_state(...) --> report the service state to the supervisor --> states could be eg. * SRVMGT_STATE_STARTUP -- still within the startup phase * SRVMGT_STATE_READY_LOCAL -- ready for local clients only * SRVMGT_STATE_READY_ALL -- ready for all clients * SRVMGT_STATE_BUSY-- too busy to process new requests * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing queued requests * SRVMGT_STATE_DEFERRED-- temporarily can't accept new requests (eg. overload) * SRVMGT_STATE_WAITING -- wait for resource (eg. printer needs paper or ink) * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some fatal error) For start, we'd just write a small library, that logs to syslog, perhaps maintains some pidfiles (maybe even a *compile-time* option to route directly to libsystemd), then patch up packages that currently use libsystemd to use our new one. I personally don't see why one would like to redo libsystemd0 from scratch, as you seem so kee of doing. Go on down your path, but I suspect not many people would cheer at you in this camp... HND KatolaZ I am too do not see a reason to redo libsystemd0 but we can think about maintaining a Devuan version of libsystemd0. To ease the maintenance of packages depending on those basic api requests. We simply cannot trust the debian version but a Devuan version should be trustworthy. Grtz, Nick ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] tiny service state api [WAS: Fwd: init system agnosticism]
On Fri, Apr 14, 2017 at 10:57:01AM +0200, Enrico Weigelt, metux IT consult wrote: [cut] > * srvmgt_daemonize() > --> detach from controlling terminal, etc > * srvmgt_droppriv(...) > --> drop root privileges (if we are still root) > --> several versions, eg. with fetching the target uid/gid from env > * srvmgt_report_state(...) > --> report the service state to the supervisor > --> states could be eg. > * SRVMGT_STATE_STARTUP -- still within the startup phase > * SRVMGT_STATE_READY_LOCAL -- ready for local clients only > * SRVMGT_STATE_READY_ALL -- ready for all clients > * SRVMGT_STATE_BUSY-- too busy to process new requests > * SRVMGT_STATE_SHUTDOWN-- shutting down, still finishing > queued requests > * SRVMGT_STATE_DEFERRED-- temporarily can't accept new > requests (eg. overload) > * SRVMGT_STATE_WAITING -- wait for resource (eg. printer > needs paper or ink) > * SRVMGT_STATE_OFFLINE -- completely offline (eg. due some > fatal error) > > For start, we'd just write a small library, that logs to syslog, > perhaps maintains some pidfiles (maybe even a *compile-time* option > to route directly to libsystemd), then patch up packages that currently > use libsystemd to use our new one. > I personally don't see why one would like to redo libsystemd0 from scratch, as you seem so kee of doing. Go on down your path, but I suspect not many people would cheer at you in this camp... HND KatolaZ -- [ ~.,_ Enzo Nicosia aka KatolaZ - GLUGCT -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
