Re: Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Roberto C . Sánchez]

On Sat, Nov 17, 2018 at 08:27:45PM +, Jonathan de Boyne Pollard wrote:
> to...@tuxteam.de:
> 
> > Inittab was the original way of doing things for AT&T, BSD copied that,
> > and SysV grafted /etc/init.d onto it. Slowly other unices followed
> > (alas!).
> > 
> 
> Some history seems in order.
> 
I love reading this stuff :-)

-- 
Roberto C. Sánchez

Re: Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Jonathan de Boyne Pollard]

to...@tuxteam.de:

Inittab was the original way of doing things for AT&T, BSD copied 
that, and SysV grafted /etc/init.d onto it. Slowly other unices 
followed (alas!).




Some history seems in order.

There wasn't really an original way of doing things, as in First Edition 
Unix this stuff, including starting up mel's da program, was simply 
hard-coded into the init program itself.  One can enjoy seeing 
/usr/mel/da and the rest in the resurrected 1st Edition source.


By the time of Seventh Edition and 4BSD, init was running /etc/rc and 
terminals were listed in /etc/ttys.  The BSDs stuck with this for 2 to 3 
decades more (depending from the BSD).  AT&T on the other hand changed 
to an inittab system with run-levels in System 3 at the start of the 1980s.


In the AT&T world, inittab was soon augmented and superseded, within a 
decade.  By the time of System 5 Release 4 in 1988 handling of terminal 
login had been moved out of inittab to the Service Access Facility, 
specifically to ttymon.  IBM AIX likewise gained the System Resource 
Controller in version 3.1 two years later, and reduced the number of 
run-levels actually used to just 1 (everything in inittab being marked 
as run level "2").  This left comparatively very little in inittab on 
those systems and in practice did away with the run-level system on Unix 
in the AT&T side of the universe.


By 1990.

Yes, run-levels and a lot of /etc/inittab were obsolete before Linux was 
even invented.


* http://jdebp.eu./FGA/run-levels-are-history.html

Two years after that Miquel van Smoorenburg cloned the old System 5 
init+rc system for Minix, but did not clone the more recent Service 
Access Facility or anything like the SRC; meaning that terminal login 
was back to the old inittab way of operating, inittab was back to being 
full of stuff, and run-levels were back.  Most Linux operating systems 
adopted it.  van Smoorenburg init had several different rcs to accompany 
it.  By the late 1990s there were so-called sysv-rc, the van Smoorenburg 
one, and file-rc developed from Winfried Truemper's r2d2.  van 
Smoorenburg init itself did not stay static, splitting the old single 
user mode into emergency and rescue modes in 1995, although it took 
about 5 years for the names to fully catch on.


* http://jdebp.eu./FGA/emergency-and-rescue-mode-bootstrap.html

In the BSD world, things were quite turbulent as well, as the 
edit-this-single-monolithic-shell-script system was proving troublesome 
in real world use.  One paper from the 1991 LISA conference published by 
USENIX talked about how /etc/rc.local had accrued all of the very same 
problems of /etc/rc that had caused /etc/rc.local to be split out in the 
first place.  FreeBSD re-engineered /etc/rc.local into an 
/etc/rc.local.d/ system with drop-in files in 1995,  and outright got 
rid of /etc/rc.local in 1998.  (The commit that deleted it was made by 
one Matthew Dillon, after encouragement from Jordan Hubbard.)  There was 
a big discussion at the end of the decade, which culminated in Mewburn 
rc replacing the old /etc/rc in NetBSD 1.5 in 2000, and in FreeBSD in 
2002.  Mewburn rc did not have run levels, and learned from experience 
of van Smoorenburg rc, organizing things so that rc scripts were (as 
intended and often, but alas far from always) a few lines of variable 
assignments and just two commands.  Gentoo's OpenRC went down the same 
route years later in 2007.  Debian's variant of van Smoorenburg rc 
started going down the same route almost as long again after that, in 2014.


OpenBSD was the lone holdout, hanging on to the old 4BSD way until 2011 
when it switched to a system similar to, but not the same as, Mewburn rc.


* http://jdebp.eu./FGA/rc.local-is-history.html

In the Linux world, there were then simpleinit, jinit, minit (once 
packaged and part of Debian), runit-init (still packaged and part of 
Debian) depinit, GNU dmd, pinit, initng, eINIT, upstart, finit, systemd, 
nosh system-manager, s6-rc, and Epoch.  Over the same period, Solaris 
switched to the SMF and MacOS switched to launchd.


* http://jdebp.eu./FGA/inittab-is-history.html

* https://blog.darknedgy.net/technology/2015/09/05/0/

Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[tomas]

On Tue, Oct 09, 2018 at 08:47:50AM -0400, Greg Wooledge wrote:

[...]

> You're actually thinking of inittab here [...]

I know, I know. I've been around this for longer than I care.
I saw HP-UX's transition from one to another (9 -> 10). I've
had GNU/Linux distros spawning PostgreSQL off inittab (bad
idea to put respawn there without a sleep...)

Inittab was the original way of doing things for AT&T, BSD
copied that, and SysV grafted /etc/init.d onto it. Slowly
other unices followed (alas!).

I think inittab's ailments could have been cured differently,
but history is like that.

Cheers
-- t


signature.asc
Description: Digital signature

Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Greg Wooledge]

On Tue, Oct 09, 2018 at 10:31:41AM +0100, Jonathan Dowland wrote:
> If you have a program that does fork, but you don't want it to, there's
> nothing you can do to stop it.

There's DJB's fghack program, but as the name implies, it's a hack, and
not guaranteed to work.

https://cr.yp.to/daemontools/fghack.html

Fortunately, upstream maintainers of daemons and other services have
come to understand that self-backgrounding is no longer considered
appropriate mandatory behavior for a daemon.  All the recent releases
of major daemons have options to STOP self-backgrounding, or they no
longer do it by default, and offer an option to do it if desired.

Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Greg Wooledge]

On Tue, Oct 09, 2018 at 11:04:21AM +0200, to...@tuxteam.de wrote:
> Systemd's stance is that it wants to supervise the processes it creates. And
> the most effective way to do that is to keep them as children; that's what
> old BSD init did, and what other process supervision programs (e.g. runit)
> do. They can watch closely when one of their children die and take action.

You're actually thinking of inittab here.  This is what System V's inittab
was intended to do.  It's a monolithic file that controls the starting
and restarting of all services on the system.  You (root) edit the file
using your favorite text editor to add or remove a service, and then
run "init q" (or "telinit q" in later Linux implementations) to have
the running init process re-read the file.

For some reason, this model never caught on.  People didn't like having
a monolithic file that had to be human-edited and which could break the
entire boot process if munged.

So, they built another layer on top of inittab.

> The whole PID thing came up with SysV init and is a crutch: many things
> can happen: process dies, leaving behind the PID file, so you have to
> check explicitly whether process is still running (in itself a race
> condition), PIDs "wrap around", so if process died a while ago, the
> PID may be active again for a totally unrelated process, etc. etc.

This is the additional layer that was added later.  It's sometimes called
"sysv-rc".

In this model, the inittab file is told to run a single script which then
runs a whole bunch of other scripts.  Each of these other scripts is
of the form /etc/init.d/foobar and takes a single argument, like "start"
or "stop".  The objective of the /etc/init.d/foobar script is to start
its designated service as some kind of background program, and capture
its PID and store it in a file, so that later, it can terminate this
process.

The storing of PIDs in files is a catastrophic hack, for the reasons
you've already explained.  The only sane way to make sure you're
controlling the actual service that you started is to stay running,
as the parent of the service.  As the parent process, you have unique
authority over the child-service.  You can be informed immediately
when it terminates, you can receive its exit status, and perhaps most
importantly of all, the child is left in the process table as a zombie
until you, the parent, have asked for its exit status.

Under the sysv-rc model, the child-service processes are orphaned.
There's no responsible parent hanging around waiting for that phone
call from the kernel telling it that Little Johnny has died.  The sysv-rc
services are wards of the state.  They become children of init (PID 1),
and init does not care about their exit status, or about notifying the
next of kin, or any of that.

When an orphaned process dies, the state (init) reaps and discards its
exit status, and the zombie is gone.  The PID is free to be reused by
some other process.  The sysv-rc shell scripts have no way of knowing
whether this has occurred.  So, people tried to layer more and more and
more hacks (Debian's start-stop-daemon is one of these) on top of the
problem.  But going from 99% working to 99.9% working is still not a
good solution.

The only 100% working solution is to keep the parent alive.  MANY people
wrote service management systems to do this.  I won't attempt to list
them all here.  Systemd is just one of them.

For a more detailed analysis of the history of sysv-rc and its failings,
see  among many other
pages.

Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[tomas]

On Tue, Oct 09, 2018 at 10:34:23AM +0100, Jonathan Dowland wrote:
> On Tue, Oct 09, 2018 at 11:04:21AM +0200, to...@tuxteam.de wrote:
> >Now systemd is very opinionated [1] software, and "there is just one
> >way to do it", so the daemons serviced by it have to behave in this
> >one way (stay in the foreground).
> 
> It certainly is opinionated, but daemons serviced by it do not *have* to
> behave in this one way: the version on my system (232-25+deb9u4)
> supports "simple" (this no-fork mode that we are discussing), "forking",
> "oneshot", "dbus", "notify" and "idle" (via man systemd.service(5)).

Thanks for the clarification

Cheers
-- t


signature.asc
Description: Digital signature

Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Jonathan Dowland]

On Tue, Oct 09, 2018 at 11:04:21AM +0200, to...@tuxteam.de wrote:

Now systemd is very opinionated [1] software, and "there is just one
way to do it", so the daemons serviced by it have to behave in this
one way (stay in the foreground).


It certainly is opinionated, but daemons serviced by it do not *have* to
behave in this one way: the version on my system (232-25+deb9u4)
supports "simple" (this no-fork mode that we are discussing), "forking",
"oneshot", "dbus", "notify" and "idle" (via man systemd.service(5)).


--

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Jonathan Dowland]

On Tue, Oct 09, 2018 at 12:13:31PM +0530, Didar Hossain wrote:

Please explain as to why "forking" is "broken" behaviour so that I can make
informed decisions whether I should use forking or not in my own scripts.

Again this is not a flame bait, I just want to understand Pros and Cons.


If you a program that doesn't fork, but you want to run it in the
background, you can: either by invoking it with '&' in an interactive
shell, or suspending it in an interactive shell and issuing the "bg"
command. There are similar ways in non-interactive environments, and
a myriad other ways (execute via cron, or at, etc.)

If you have a program that does fork, but you don't want it to, there's
nothing you can do to stop it.

Modern init systems (like systemd) can manage programs that do not fork,
so there's no need to do when being used in conjunction with such
systems.

So not forking gives the user more control. It's also the simpler
implementation, and not necessary with modern init systems.



--

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: [OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[tomas]

On Tue, Oct 09, 2018 at 12:13:31PM +0530, Didar Hossain wrote:

[...]

> Please explain as to why "forking" is "broken" behaviour so that I can make
> informed decisions whether I should use forking or not in my own scripts.

Systemd's stance is that it wants to supervise the processes it creates. And
the most effective way to do that is to keep them as children; that's what
old BSD init did, and what other process supervision programs (e.g. runit)
do. They can watch closely when one of their children die and take action.

The whole PID thing came up with SysV init and is a crutch: many things
can happen: process dies, leaving behind the PID file, so you have to
check explicitly whether process is still running (in itself a race
condition), PIDs "wrap around", so if process died a while ago, the
PID may be active again for a totally unrelated process, etc. etc.

Not pretty.

Now systemd is very opinionated [1] software, and "there is just one
way to do it", so the daemons serviced by it have to behave in this
one way (stay in the foreground).

> Again this is not a flame bait, I just want to understand Pros and Cons.

No worries: flame baits tend to have a strong smell :-)

Cheers
[1] That's why some of us don't like systemd: personally, I think it's
   OK for people to have (sometimes strong) opinions, but I don't
   really like software to have opinions, especially if they try to
   override mine.

-- tomás


signature.asc
Description: Digital signature

[OT] Systemd and forking programs (was systemd can't start a dæmon and doesn't give any error either)

[Didar Hossain]

** Apologies for hijacking the thread, I really wanted to start a new one
   but I need the perspective of Greg on this **

On Mon, Oct 08, 2018 at 09:02:18AM -0400, Greg Wooledge wrote:
> On Mon, Oct 08, 2018 at 06:17:52PM +0800, Weiwu Zhang wrote:
> 
> You've used an option named "-daemon".  I am guessing that this causes
> the process to "daemonize" itself (auto-background, self-background).
> 
> This is what you want if you're on a BSD system in 1989.
> 
> This is NOT what you want if you're on a systemd system in 2018.

I am interested in understanding the above points. I am not a hater and do not
intend to start a flame war here about systemd. I am an old dog who is trying to
understand the new way of doing things.


> You did not specify what TYPE of service this is.  Since you didn't specify
> a Type= option, systemd uses the default, which is "simple", which means
> that it expects a single process to run in the foreground.
> 
> So: either omit that "-daemon" option, or whatever it takes to make the
> process run in the foreground like a good process should.
> 
> -OR-
> 
> Tell systemd that your process uses ancient, broken behavior
> (self-backgrounding) by specifying "Type=forking".

> I know which one I would pick.  Assuming that dropping -daemon does what

Please explain as to why "forking" is "broken" behaviour so that I can make
informed decisions whether I should use forking or not in my own scripts.

Again this is not a flame bait, I just want to understand Pros and Cons.

Regards,
Didar

-- 
 that's *IT*.  I'm never fucking attempting to install redhat
again.
 this is like the 10th fucking machine on which the installer has
imploded immediately after I went through the hell of their
package selection process.
 Sammy: just use debian and never look back
 timball: debian iso's are being written at this very moment.