Re: [Dng] Why daemontools is so cool
Le 30/03/2015 19:24, Steve Litt a écrit : Hi Didier, If your post says what I think it says, you're saying that modern init systems should always start services concurrently, not consecutively. Maybe this is a little too extremist and is going farther than what I meant. Let me moderate it a little. Typically, the early init phase happens in initramfs and finishes with switch_root or pivot_root to the final rootfs. At this time, when /sbin/init starts, /, /dev, /proc, /sys are already mounted and /dev is at least partly populated (totally in the case of a static /dev). Which part of the dependencies is best managed by init and which by the daemons is a matter of expertise. Starting things sequentially is certainly a too extreme way to manage the dependencies. But my point was rather on the daemons. I'm not saying that init should be a fork-bomb (I discovered this expression recently and enjoy it :-) ). My point is that services should be able to wait for their resources by themselves, not only in the initialization phase, and be able to react to changes in these resources. For a daemon we had two design options proposed: 1) resource allocation, then orphanning, then service, then death if resource lost, 2) statefull program providing service when resource is available. The first is the traditional fashion; it is fragile and unsafe to supervise; it was devised long ago, far before the advent of hotplugging. The second is a survivor, easy to supervise and fits with hotplugging. Didier ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
Le 29/03/2015 11:15, marc a écrit : No need to mix doubleforking and PID tracking on your program. That should be the duty of whatever daemonizes and manages your program. You know, like Daemontools or s6. So there is a very good reason for a deamon to handle its own backgrounding: The sensible convention is it that it should only background at the instant where it is ready to service requests: If there is a long initialisation phase it should stay in the foreground - so that things that depend on it in turn do not get started too soon. A more detailed description of this problem I wrote up a while ago at welz.org.za/notes/on-starting-daemons.html. More fundamentally: If an application has problems calling the a daemonize() or fork_parent() function or the handful of system calls that make up this, then maybe this a limitation of the development environment or language - if calling these this is regarded to be hard then one wonders how reliable the rest of the program is. Dear Marc, This makes sense if we consider the very old way a Linux system was run: boot; then mount filesystems, them start syslog; then initialize network; then start nfs and ssh ... then stop daemons in the reverse order, unmount filesystems and shutdown. A more modern way of starting daemons is to have fine-grained dependencies. Instead of waiting until syslogd is started, why not wait until the socket /var/log is created, or create it even before starting syslogd. And furthermore, is it really necessary to wait for anything before starting the daemons; why wait until the network is configured to start ssh and nfs? Things are not going the linear way anymore. The network cable, as an example, can be disconnected and reconnected, and the network interface de-configured and re-configured, and the ssh daemon will survive, and NFS as well. Even you can reboot an nfs server and clients having their rootfs nfs-mounted come back to life seemlessly. Daemons should be prepared to wait until the needed ressource is available; they should even be prepared to see the ressource they need disapear and to wait until it shows up again. You make a very good point - often forgotten , including by me - about daemon's current directory. But about orphanning the daemons, I disagree. I used to do like you for my private daemons, and to manage pid-files, but it was because I repeated the same scheme by pure lazyness. I am decided to now use a supervisor. Didier ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Mon, 30 Mar 2015 11:44:29 +0200 Didier Kryn k...@in2p3.fr wrote: Le 29/03/2015 11:15, marc a écrit : No need to mix doubleforking and PID tracking on your program. That should be the duty of whatever daemonizes and manages your program. You know, like Daemontools or s6. So there is a very good reason for a deamon to handle its own backgrounding: The sensible convention is it that it should only background at the instant where it is ready to service requests: If there is a long initialisation phase it should stay in the foreground - so that things that depend on it in turn do not get started too soon. A more detailed description of this problem I wrote up a while ago at welz.org.za/notes/on-starting-daemons.html. More fundamentally: If an application has problems calling the a daemonize() or fork_parent() function or the handful of system calls that make up this, then maybe this a limitation of the development environment or language - if calling these this is regarded to be hard then one wonders how reliable the rest of the program is. Dear Marc, This makes sense if we consider the very old way a Linux system was run: boot; then mount filesystems, them start syslog; then initialize network; then start nfs and ssh ... then stop daemons in the reverse order, unmount filesystems and shutdown. A more modern way of starting daemons is to have fine-grained dependencies. Instead of waiting until syslogd is started, why not wait until the socket /var/log is created, or create it even before starting syslogd. And furthermore, is it really necessary to wait for anything before starting the daemons; why wait until the network is configured to start ssh and nfs? Things are not going the linear way anymore. The network cable, as an example, can be disconnected and reconnected, and the network interface de-configured and re-configured, and the ssh daemon will survive, and NFS as well. Even you can reboot an nfs server and clients having their rootfs nfs-mounted come back to life seemlessly. Daemons should be prepared to wait until the needed ressource is available; they should even be prepared to see the ressource they need disapear and to wait until it shows up again. Hi Didier, If your post says what I think it says, you're saying that modern init systems should always start services concurrently, not consecutively. Certainly that's a good thing, and we're working toward it, but it's important to keep some perspective on the matter and do a cost/benefit analysis on the alternatives. On my experimental Manjaro machine, systemd, which most would agree is very concurrent, booted in 4 seconds. Epoch, which has absolutely no concurrency at all and boots completely consecutively, booted in 8 seconds. How much complexity, how much indeterminacy, are we willing to put up with to get A) 4 more seconds in our life every time we reboot, and B) do it the more modern way? The preceding question has no one right answer. As has been pointed out in many pro-systemd assertions, if you're contracted to provide 0.99 uptime, that four seconds means everything. If there's some reason you need to reboot six times a day, the four second difference could become meaningful. If you're starting 40 services, it could be a difference between 1 minute and 2 minutes, which might be somewhat meaningful if you shut down every night and boot up every morning. And most of all, if you or your distro is careless with order on a completely consecutive boot, it could make all the difference in the world. I've had 5 minute boots, of which 3 minutes was, IIRC, NFS timing out instead of running instantly, because of no reverse DNS. Even today, if you put wicd-cli in your bootup, it takes 20 seconds or so to do the wifi negotiations. But note that all wifi-equipped systemd systems I've seen simply delay wifi-negotiation out of init and into X startup. Looking at the use cases in the preceding two paragraphs, I'd say that in all other cases I can think of, the 4 second plus modern-man feelgood benefit you get from concurrent startup during init doesn't begin to pay for the increased complexity and decreased determinacy of concurrent service startup. By the way, one excellent thing about the Epoch init system is that, because it's completely consecutive, you can get a close look at which services are taking too long to start, troubleshoot them to find the bottleneck, and fix them, so that they'll start efficiently in your concurrent init. The quicker everything starts in a concurrent init, the less chance for race conditions. SteveT Steve Litt* http://www.troubleshooters.com/ Troubleshooting Training * Human Performance ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
Hi Steve, On Mon, Mar 30, 2015 at 7:24 PM, Steve Litt - sl...@troubleshooters.com devuan.kn.3c178e07a2.slitt#troubleshooters@ob.0sg.net wrote: Hi Didier, If your post says what I think it says, you're saying that modern init systems should always start services concurrently, not consecutively. Certainly that's a good thing, and we're working toward it, but it's important to keep some perspective on the matter and do a cost/benefit analysis on the alternatives. On my experimental Manjaro machine, systemd, which most would agree is very concurrent, booted in 4 seconds. Epoch, which has absolutely no concurrency at all and boots completely consecutively, booted in 8 seconds. How much complexity, how much indeterminacy, are we willing to put up with to get A) 4 more seconds in our life every time we reboot, and B) do it the more modern way? How fast is epoch when you throw it at a generic piece of hardware that you did not hand-tune it for? The problem there is that a consecutive boot system needs to probe for hardware and give that hardware time to show up, blocking the boot process in the meantime. If you have lots of hardware you need to probe for (as e.g. Devuan would need to out-of-the-box), then those wait-times can sum up really fast. And some hardware can take a very long time to register, so you need to be generous with those time-outs. On the other hand anybody without the hardware will stuck for the entire timeout, so you need to keep them as short as possible. A parallel boot system can just start all hardware probes at the same time, be very generic with the timeouts and just continue as soon as the necessary hardware showed up. It is also impossible to *generically* support all possible of combinations of real hardware, devicemapper, LVM, software raid, fsck and cryptsetup in a consecutive boot setup. Debian had long-standing bugs to that regard. Note that you can of course configure anything with any system manually, but any Linux distribution should strive to support as wide a range of setups as possible out-of-the-box. Well, arch and gentoo obviously do not need to, but they already fill their niches quite nicely:-) And most of all, if you or your distro is careless with order on a completely consecutive boot, it could make all the difference in the world. I've had 5 minute boots, of which 3 minutes was, IIRC, NFS timing out instead of running instantly, because of no reverse DNS. Even today, if you put wicd-cli in your bootup, it takes 20 seconds or so to do the wifi negotiations. But note that all wifi-equipped systemd systems I've seen simply delay wifi-negotiation out of init and into X startup. If your distro screws up, then you are screwed (until you fix it;-). That is pretty much true independent of the topic. I seriously doubt that systemd was pushing wifi-negotiation into X startup though. With any non-concurrent system X may be up and waiting for your login long before init is done with the initial setup of your system. So it may appear like wifi negotiation happens only after X login. Or you might have been ended up using network-manager, which has a tendency to only start the network after somebody logged in:-/ Looking at the use cases in the preceding two paragraphs, I'd say that in all other cases I can think of, the 4 second plus modern-man feelgood benefit you get from concurrent startup during init doesn't begin to pay for the increased complexity and decreased determinacy of concurrent service startup. The killer argument for parallel startup with dependency handling is robustness, not speed. Maybe it is my tendency to mess around with cryptsetup and co. that gets me into trouble, but I did have unbootable systems with sysv-init due to unexpected setup problems. Nothing I could not fix, but still an annoyance that I would be happy to get rid of. This is a statement about the concept of parallel init systems with dependencies, not about any specific implementation. By the way, one excellent thing about the Epoch init system is that, because it's completely consecutive, you can get a close look at which services are taking too long to start, troubleshoot them to find the bottleneck, and fix them, so that they'll start efficiently in your concurrent init. The quicker everything starts in a concurrent init, the less chance for race conditions. Yes and no. What happens if your filesystem has a slow day (e.g. due to some f*ing RAID controller deciding that it needs to do some extra sanity checks)? That will lead to the consecutive boot system panicking since its root device is not there (after some timeout), which in turn will lead to some poor admin having to investigate and nudge the server to try once more. The whole consecutive boot thing hinges on timeouts and that is neither generic nor robust. I admit that it is simple. And I also think that epoch can make a great init system for a specific system, but there are
Re: [Dng] Why daemontools is so cool
The problem there is that a consecutive boot system needs to probe for hardware and give that hardware time to show up, blocking the boot process in the meantime. The only hardware detection during boot that *needs* to block is mounting the root device. Boot cannot proceed without that. To do so, the kernel needs to have a driver for the device--either loaded into it by the initramfs (the loading and execution of which takes time), or compiled in directly. The remaining driver-loading does not run as part of the boot sequence. Only the modules needed to bring the system up get loaded (i.e. the modules in /etc/modprobe.d). Once root and the other filesystems are mounted, you should be able to start programs up in whatever order regardless of the state of the hardware. Individual programs like ntpd should be smart enough to wait until the required hardware is available and configured, such as network interfaces (either on their own, or with the help of a supervisor). If your boot sequence is taking too long because it's loading unnecessary drivers, then your boot sequence is misconfigured. And some hardware can take a very long time to register, so you need to be generous with those time-outs. On the other hand anybody without the hardware will stuck for the entire timeout, so you need to keep them as short as possible. If your hardware is taking a long time to spin up, it's due to bug in either the hardware or its driver, not the boot sequence. Again, if you're talking about waiting for the device with your root filesystem, the boot process is *supposed* to block until it's ready. Boot cannot proceed until the root device is found and mounted; otherwise you obviously cannot load programs. This process cannot be sped up through parallelization (Amdahl's Law and all that). Same goes for programs that cannot be started until /usr is mounted (if you have a separate /usr). A parallel boot system can just start all hardware probes at the same time, be very generic with the timeouts and just continue as soon as the necessary hardware showed up. Hardware detection happens in the kernel, and the kernel does detect hardware in parallel. That's one of the reasons why you get to deal with disk and interface names changing between boots. Just continuing as soon as the necessary hardware shows up is already what happens, like I said above. Also, it's useless to run more instances of insmod(8) than you have CPU cores. It can be dangerous to run even two insmod(8)'s in parallel, since last I checked [1], setting up a kernel module is not an atomic operation. Run multiple insmod(8)'s at your own risk--you may expose race conditions that crash your system or render your hardware unusable without a reboot. The killer argument for parallel startup with dependency handling is robustness, not speed. No, the opposite is true. Programs with multiple instances of execution (processes, threads, coroutines) in practice tend to be much more error-prone, because they are much harder to reason about. This is because the number of states such a program can be in increases with the *factorial* of the number of instances of execution it has. This is such a problem that determinism is often a design requirement for mission-critical software whose failure will result in huge costs and/or loss of life. Maybe it is my tendency to mess around with cryptsetup and co. that gets me into trouble, but I did have unbootable systems with sysv-init due to unexpected setup problems. Nothing I could not fix, but still an annoyance that I would be happy to get rid of. Parallel boot won't fix misconfigurations you introduced by messing around with it. The whole consecutive boot thing hinges on timeouts and that is neither generic nor robust. The boot sequence does *not* hinge on timeouts. If anything, timeouts are a fallback mechanism for working around other programs not making forward progress (i.e. due to bugs, a down network, or faulty hardware). If your boot sequence is encountering timeouts, then something's wrong with your boot sequence. -Jude [1] http://lkml.iu.edu/hypermail/linux/kernel/1502.2/01852.html On Mon, Mar 30, 2015 at 2:54 PM, devuan...@spamgourmet.net wrote: Hi Steve, On Mon, Mar 30, 2015 at 7:24 PM, Steve Litt - sl...@troubleshooters.com devuan.kn.3c178e07a2.slitt#troubleshooters@ob.0sg.net wrote: Hi Didier, If your post says what I think it says, you're saying that modern init systems should always start services concurrently, not consecutively. Certainly that's a good thing, and we're working toward it, but it's important to keep some perspective on the matter and do a cost/benefit analysis on the alternatives. On my experimental Manjaro machine, systemd, which most would agree is very concurrent, booted in 4 seconds. Epoch, which has absolutely no concurrency at all and boots completely consecutively, booted in 8 seconds. How much complexity,
Re: [Dng] Why daemontools is so cool
If you need to generically support a wide range of setups and include the fun stuff listed above into the mix, then your init system will need to do a lot more than what is necessary to bring up one individual system. Debian did run all of the above during its boot sequence (if the package was installed), just in case somebody actually went ahead and used it. Except that it doesn't. Debian will only do the extra fun stuff if it's installed and usable. Go read the init scripts in /etc/rcS.d if you don't believe me. Dependencies enable you to have a robust boot sequence even with hardware like this. No. No they do not. Let me say it again: if the device with your root filesystem is unavailable, the boot process *will* wait for it to become available. There is no alternative, besides panicking. There is no parallelization before root gets mounted, because nothing else *can* start. If you want to support a wide range of combinations of all the goodies Linux comes with, incl. software raid, logical volume management, disk encryption and any combination thereof, then you can not do so in a generic way using a strictly sequential init system. You can configure all of these for each individual system, but you can not have generic support for all of them in any valid combination in your distribution. First, the extra packages are not used unless you install them, and will only get invoked if they are needed. Second, Debian has initialized them all sequentially, in the right order, without fail for the better part of its existence. In fact, if you go look at Debian bug tracker, the introduction of init script parallelization strategies have lead to hard-to-reproduce boot bugs due to race conditions. You will also need to introduce timeouts (which are either too short or too long, depending on the system) into each and every step along the way. What are you talking about? root@t510:/home/jude# runlevel N 2 root@t510:/home/jude# fgrep -r sleep /etc/rc2.d /etc/rcS.d root@t510:/home/jude# You do *not* need timeouts to boot the system sequentially. My point is A init system should be robust to work with any valid configuration I can put the system in and I understand your reply to be Complexity makes programs less robust. I agree with both statement and see no contradiction whatsoever. Um, no. That has *never*, *EVER* been the case. If you break something, it stays broken until you fix it. The OS does exactly what you tell it to do, as it should. It's not the OS's responsibility to clean up your mistakes if you put it into a state where it can't boot. A strictly sequential boot is not able to do that in a generic way. You can configure *your* system to do that by tweaking the sequence, but it is impossible for a distribution to ship a sequential init system that can robustly set up any filesystem that you can mount manually. Then how, pray tell, has Debian been able to boot sequentially all these years, with all these features? Okay. I hate that it's come to this, but at this point in reading your reply I can no longer tell whether or not you're a troll. You talk about the low-level details of userspace like you think you understand it, but it's pretty clear that you do not (or, you do a very bad job at communicating it). Either way, this thread has gone wildly off-topic, so I will no longer reply to it. -Jude On Mon, Mar 30, 2015 at 7:06 PM, devuan...@spamgourmet.net wrote: Hi Jude, hello Isaac, I did express myself poorly when I spoke of hardware detection. You both are fully correct to call me out on that. Yes, hardware detection happens in the kernel, and indeed some of it is done in parallel there. I was thinking about all the user-space tools that scan drives and create new devices based on their findings when writing this mail. These little things like btrfs scan, lvm --forgot-the-parameter, cryptsetup open, and the others that allow you to do cool things with your filesystems. Sorry for the confusion this caused and my poor choice of words. On Tue, Mar 31, 2015 at 12:05 AM, Jude Nelson - jud...@gmail.com devuan.kn.ae5676beef.judecn#gmail@ob.0sg.net wrote: snip If your boot sequence is taking too long because it's loading unnecessary drivers, then your boot sequence is misconfigured. If you need to generically support a wide range of setups and include the fun stuff listed above into the mix, then your init system will need to do a lot more than what is necessary to bring up one individual system. Debian did run all of the above during its boot sequence (if the package was installed), just in case somebody actually went ahead and used it. And some hardware can take a very long time to register, so you need to be generous with those time-outs. On the other hand anybody without the hardware will stuck for the entire timeout, so you need to keep them as short as possible. If your hardware is taking a long time
Re: [Dng] Why daemontools is so cool
Okay, one technical correction (on myself): What are you talking about? root@t510:/home/jude# runlevel N 2 root@t510:/home/jude# fgrep -r sleep /etc/rc2.d /etc/rcS.d root@t510:/home/jude# You do *not* need timeouts to boot the system sequentially. I should have typed fgrep -r sleep /etc/rc2.d/* /etc/rcS.d/*, since it slipped my mind that fgrep does not follow symlinks unless specified on the command-line. While this command yields many instances of sleep, a cursory inspection of when they occur indicates that the init scripts almost always execute sleep as part a restart, reload, upgrade, or failsafe. They do not sleep on the start/stop critical paths. Thus, I stand by my earlier point that timeouts aren't an expected part of the boot process. -Jude On Mon, Mar 30, 2015 at 10:55 PM, Jude Nelson jud...@gmail.com wrote: If you need to generically support a wide range of setups and include the fun stuff listed above into the mix, then your init system will need to do a lot more than what is necessary to bring up one individual system. Debian did run all of the above during its boot sequence (if the package was installed), just in case somebody actually went ahead and used it. Except that it doesn't. Debian will only do the extra fun stuff if it's installed and usable. Go read the init scripts in /etc/rcS.d if you don't believe me. Dependencies enable you to have a robust boot sequence even with hardware like this. No. No they do not. Let me say it again: if the device with your root filesystem is unavailable, the boot process *will* wait for it to become available. There is no alternative, besides panicking. There is no parallelization before root gets mounted, because nothing else *can* start. If you want to support a wide range of combinations of all the goodies Linux comes with, incl. software raid, logical volume management, disk encryption and any combination thereof, then you can not do so in a generic way using a strictly sequential init system. You can configure all of these for each individual system, but you can not have generic support for all of them in any valid combination in your distribution. First, the extra packages are not used unless you install them, and will only get invoked if they are needed. Second, Debian has initialized them all sequentially, in the right order, without fail for the better part of its existence. In fact, if you go look at Debian bug tracker, the introduction of init script parallelization strategies have lead to hard-to-reproduce boot bugs due to race conditions. You will also need to introduce timeouts (which are either too short or too long, depending on the system) into each and every step along the way. What are you talking about? root@t510:/home/jude# runlevel N 2 root@t510:/home/jude# fgrep -r sleep /etc/rc2.d /etc/rcS.d root@t510:/home/jude# You do *not* need timeouts to boot the system sequentially. My point is A init system should be robust to work with any valid configuration I can put the system in and I understand your reply to be Complexity makes programs less robust. I agree with both statement and see no contradiction whatsoever. Um, no. That has *never*, *EVER* been the case. If you break something, it stays broken until you fix it. The OS does exactly what you tell it to do, as it should. It's not the OS's responsibility to clean up your mistakes if you put it into a state where it can't boot. A strictly sequential boot is not able to do that in a generic way. You can configure *your* system to do that by tweaking the sequence, but it is impossible for a distribution to ship a sequential init system that can robustly set up any filesystem that you can mount manually. Then how, pray tell, has Debian been able to boot sequentially all these years, with all these features? Okay. I hate that it's come to this, but at this point in reading your reply I can no longer tell whether or not you're a troll. You talk about the low-level details of userspace like you think you understand it, but it's pretty clear that you do not (or, you do a very bad job at communicating it). Either way, this thread has gone wildly off-topic, so I will no longer reply to it. -Jude On Mon, Mar 30, 2015 at 7:06 PM, devuan...@spamgourmet.net wrote: Hi Jude, hello Isaac, I did express myself poorly when I spoke of hardware detection. You both are fully correct to call me out on that. Yes, hardware detection happens in the kernel, and indeed some of it is done in parallel there. I was thinking about all the user-space tools that scan drives and create new devices based on their findings when writing this mail. These little things like btrfs scan, lvm --forgot-the-parameter, cryptsetup open, and the others that allow you to do cool things with your filesystems. Sorry for the confusion this caused and my poor choice of words. On Tue,
Re: [Dng] Why daemontools is so cool
No need to mix doubleforking and PID tracking on your program. That should be the duty of whatever daemonizes and manages your program. You know, like Daemontools or s6. So there is a very good reason for a deamon to handle its own backgrounding: The sensible convention is it that it should only background at the instant where it is ready to service requests: If there is a long initialisation phase it should stay in the foreground - so that things that depend on it in turn do not get started too soon. A more detailed description of this problem I wrote up a while ago at welz.org.za/notes/on-starting-daemons.html. More fundamentally: If an application has problems calling the a daemonize() or fork_parent() function or the handful of system calls that make up this, then maybe this a limitation of the development environment or language - if calling these this is regarded to be hard then one wonders how reliable the rest of the program is. regards marc ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, 28 Mar 2015 17:04:39 +0100 poitr pogo lepo...@gmail.com wrote: s6 advertises itself as init replacement. I'm not using s6 or daemontools. I do not need them. Just learned about them and I'm bit surprised thy tend to replace init instead of being a good helpers for those who need them. IMHO they came into existence cause more and more applications were missing deamon part, to make sysadmin/programmer life easier. s6 doe not support those which daemonize on its own, anyway :) Do one thing and do it right. You create a daemon: Perhaps a server, perhaps something else. Above all else, it's a **program**. It has a function. At least for debugging purposes, it darn well better be able to run in the foreground (sorry Apache2 and nginx). Do one thing and do it right: No need to mix doubleforking and PID tracking on your program. That should be the duty of whatever daemonizes and manages your program. You know, like Daemontools or s6. I got tired of cron on my Wheezy box (long story), so I created my own cron system in Python. It reads its (almost compatible) crontab file, checks the current time, and does the called-for commands, then sleeps for 31 seconds and does it all over again. I can run it in a terminal and watch everything message through on the screen. Once it was working properly, I just gave it a daemontools run script, and bang, it was a genuine daemon. No doublefork in my program. No PID tracking in my program. Nothing but cron. That's how life should be. My program takes care of its business and nothing more, and my program's instance management is up to daemontools, which was designed from the ground up to do that management. SteveT Steve Litt* http://www.troubleshooters.com/ Troubleshooting Training * Human Performance ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, 28 Mar 2015 22:24:51 +0800 Robert Storey robert.sto...@gmail.com wrote: Thanks for telling us about this. I hadn't heard of daemontools before. I did an apt-cache show daemontools and interestingly it suggests runit. I presume that Devuan will at least include daemontools and possibly runit, which should be a fascinating combination to play with. Runit is studly. I personally replaced systemd with runit on a systemd encumbered Manjaro, and on Centos :-) Runit works as advertised. Most people (not I, but most people) would use either runit or daemontools, but not both. Runit is like a daemontools superset with PID1 functionality and orderly shutdown/reboot. Can't wait to get Devuan installed. Yeah, just put daemontools or daemontools-encore in your /etc/initd, and slowly switch management of your daemons from sysvinit to daemontools. SteveT Steve Litt* http://www.troubleshooters.com/ Troubleshooting Training * Human Performance ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, 28 Mar 2015 12:31:04 + Nuno Magalhães nunomagalh...@eu.ipp.pt wrote: On Sat, Mar 28, 2015 at 12:15 PM, poitr pogo lepo...@gmail.com wrote: S6 even forces such a program must not handle daemon mode itself. That process must not background itself: being run by a supervision tree already makes it a background task. I don't know its innards, but i find it awkward that a program would force, thus prevent another from daemonizing. Can you explain? Cheers, Nuno Yes. First, see http://en.wikipedia.org/wiki/Daemon_%28computing%29 for the definition of daemon. Here's a quote from the page: = In multitasking computer operating systems, a daemon (/ˈdiːmən/ or /ˈdeɪmən/)[1] is a computer program that runs as a background process, rather than being under the direct control of an interactive user. = It's a program running in the background, in no way attached to an interactive terminal. There are many, many ways to achieve this, including these common tricks: myprogram nohup myprogram Double-fork I haven't read all the daemontools code, but from ps axjf, it appears to me that the daemontools svscan program runs an instance of the daemontools supervise for each directory symlinked to /service, and each of those supervise programs runs its directory's run script, probably using popen to capture the run script's stdout. Why capture stdout? Because, for each /service/whateverd directory, it also runs supervise on /service/whateverd/log , and passes the stdout to the log. Because supervise runs its runscript *in the foreground*, it has access to its stdout, and has access to its current PID. No more playing parse ps ax and guess or doing detective work in /proc. In fact, svstat /service/whateverd gives you whateverd's PID. Daemontools wouldn't be able to do this stuff if supervise chose to run its service in the background, relative to itself. With the daemontools method you get much better control. Now meanwhile, any daemon worth its salt gives you a way to run it in the foreground: * ntpd -d * sshd -D * smbd -F * nmbd -F So, in the daemontools run script for the program, you just run it with the don't detach option. The fact that the daemon is run, by supervise, in the foreground gives you much more control, a lot more debugging options, and means if you want to write your own daemon, you don't need to worry about any doublefork nonsense, nor write a file with its PID, and all that other hassle. SteveT Steve Litt* http://www.troubleshooters.com/ Troubleshooting Training * Human Performance ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, 28 Mar 2015 14:52:47 +0100 poitr pogo lepo...@gmail.com wrote: Cannot. Cannot what? This is from the s6 docs. I'm not using s6. Just heard about it today. I sure can't accuse you of procrastinating in forming an opinion. Another s6 limitation - finish script must end in 3 seconds. or will be killed. I'm pretty sure that 3 seconds is for each daemon's finish script, not for the whole of shutdown. What would be a better figure than 3 seconds? Have you ever had one of these computers that took a half hour to shut down? Nice for desktop, not a server. Because??? It used to be that an application is responsible for proper start up/shutdown procedure, as application knows the proper way how to do it. Yes. It's in the man page. Init script is just a place where to hook a startup/shutdown script to be executed when system needs to start/stop application in some automatic way. Now there are tools which help deamonize things. Depending how portable they are it make the price one has to pay when decides to use them instead of handling this yourself in the application. Daemontools, daemontools-encore, runit, and s6 daemonize any program that can run in the foreground. No need to mess around in the application. SteveT Steve Litt* http://www.troubleshooters.com/ Troubleshooting Training * Human Performance ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, Mar 28, 2015 at 12:15 PM, poitr pogo lepo...@gmail.com wrote: S6 even forces such a program must not handle daemon mode itself. That process must not background itself: being run by a supervision tree already makes it a background task. I don't know its innards, but i find it awkward that a program would force, thus prevent another from daemonizing. Can you explain? Cheers, Nuno ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
Cannot. This is from the s6 docs. I'm not using s6. Just heard about it today. Another s6 limitation - finish script must end in 3 seconds. or will be killed. Nice for desktop, not a server. It used to be that an application is responsible for proper start up/shutdown procedure, as application knows the proper way how to do it. Init script is just a place where to hook a startup/shutdown script to be executed when system needs to start/stop application in some automatic way. Now there are tools which help deamonize things. Depending how portable they are it make the price one has to pay when decides to use them instead of handling this yourself in the application. Regards piotr ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
Thanks for telling us about this. I hadn't heard of daemontools before. I did an apt-cache show daemontools and interestingly it suggests runit. I presume that Devuan will at least include daemontools and possibly runit, which should be a fascinating combination to play with. Can't wait to get Devuan installed. cheers, Robert Quote of the day for systemd fans... No society has to be dragged kicking and screaming down the slope of decline and fall; one and all, they take that slope at a run, yelling in triumph, utterly convinced that the road to imminent ruin will lead them to paradise on Earth. - John Michael Greer ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, Mar 28, 2015 at 4:15 PM, Isaac Dunham ibid...@gmail.com wrote: (...) If you want to *manage* a daemon (ie, do something when it stops, like restarting it), it's very tricky to do it reliably without having a fixed It might be tricky if application does not support startup/shutdown procedure. That's why I would expect application to provide a script to start/stop it. I expect application to be tested on a system it is to be running on, so developing a stop/start script is not a big deal comparing to the whole test procedure. PID; daemonizing means close stdin/stdout/stderr, then if (fork()) exit(0); or some such thing (in other words, create a new process and exit the old one), so you do not have a fixed PID. On Linux, BSD, and older UNIX versions, this can be done thus: daemon(0,0); How this can be done in java? ;) Detaching from terminal, binding to reserved ports, dropping privileges, etc. Another s6 limitation - finish script must end in 3 seconds. or will be killed. Nice for desktop, not a server. sysvinit /etc/init.d/sendsigs actually only allows 1 second. I see it is skipping some processes (which have their own upstart scripts, and others which are recorded in special file not to be killed by sendsigs). So it (sendsigs) is for all the rest of processes which do not care about how they should be stopped. Not for that which provide its own start/stop scripts. In case of s6 it is the limit for the finish script for particular application. (...) My understanding is that s6 and some of the equivalent packages, such as daemontools, are fairly portable. They are generally not used as a way to offload daemonizing, but as a way to ensure that a service stays started. If you don't need that, you don't need them. s6 advertises itself as init replacement. I'm not using s6 or daemontools. I do not need them. Just learned about them and I'm bit surprised thy tend to replace init instead of being a good helpers for those who need them. IMHO they came into existence cause more and more applications were missing deamon part, to make sysadmin/programmer life easier. s6 doe not support those which daemonize on its own, anyway :) init has inittab for monitoring application which need continuous restart by design like getty. If a daemon needs to be monitored and restarted when it dies, bug should be fixed. I expect a deamon to work from start to stop command. No self stopping. As workaround nohup, screen, while `true`,etc are enough to keep system running until bug is fixed. I do not like buggy software, so maybe that's why i do not need s6/daemontools. just IMHO. Regards piotr ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, Mar 28, 2015 at 02:52:47PM +0100, poitr pogo wrote: (in response to the question of why s6 prevents daemonizing) Cannot. This is from the s6 docs. I'm not using s6. Just heard about it today. If you want to *manage* a daemon (ie, do something when it stops, like restarting it), it's very tricky to do it reliably without having a fixed PID; daemonizing means close stdin/stdout/stderr, then if (fork()) exit(0); or some such thing (in other words, create a new process and exit the old one), so you do not have a fixed PID. On Linux, BSD, and older UNIX versions, this can be done thus: daemon(0,0); Another s6 limitation - finish script must end in 3 seconds. or will be killed. Nice for desktop, not a server. sysvinit /etc/init.d/sendsigs actually only allows 1 second. It used to be that an application is responsible for proper start up/shutdown procedure, as application knows the proper way how to do it. Init script is just a place where to hook a startup/shutdown script to be executed when system needs to start/stop application in some automatic way. Now there are tools which help deamonize things. Depending how portable they are it make the price one has to pay when decides to use them instead of handling this yourself in the application. My understanding is that s6 and some of the equivalent packages, such as daemontools, are fairly portable. They are generally not used as a way to offload daemonizing, but as a way to ensure that a service stays started. If you don't need that, you don't need them. Regards piotr Thanks, Isaac Dunham ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
Hey Poitr, s6 advertises itself as init replacement. No, S6, runit, daemontools, etc advertise as supervisor tools. Some of these, like S6, are also a sane init replacement, should you want to go this way. I'm not using s6 or daemontools. I do not need them. Some people do need them. We do a lot of work with large scale (1000's) containers. Supervisors are mighty handy for that use-case, as well as a fair few others. Just learned about them and I'm bit surprised thy tend to replace init instead of being a good helpers for those who need them. They can be both. IMHO they came into existence cause more and more applications were missing deamon part, to make sysadmin/programmer life easier. s6 doe not support those which daemonize on its own, anyway :) They came into existence because some jobs cannot afford to fail. If they do, they must be restarted, and if they keep failing, a human has to be alerted. Supervisors have been around since the dawn of modern computing. init has inittab for monitoring application which need continuous restart by design like getty. If a daemon needs to be monitored and restarted when it dies, bug should be fixed. Yeah, because the only reason jobs die is because bugs. Even so, fixing bugs takes time. when an ecommerce site dies, you might put up a holding page saying PLEASE WAIT WE ARE FIXING BUGS BECAUSE OUR PRINCIPLES ABOUT HOW SOFTWARE SHOULD BE OPERATED ARE MORE IMPORTANT THAN SERVING CUSTOMERS. We will chose to log the fault, restart the process, and keep on trucking. I expect a deamon to work from start to stop command. No self stopping. How did that work out for you in real life? As workaround nohup, screen, while `true`,etc are enough to keep system running until bug is fixed. Seriously? You are simply describing a crude, primitive and low-functional supervisor routine. Essentially what you are saying is Supervisors should never be needed, because I write my own I do not like buggy software, so maybe that's why i do not need s6/daemontools. I do not like disappointing customers, and that's why we use supervisors. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
IMHO s6/daemontools and similar tools are just a helpers. That is *exactly* what they are. They help to run as a unix daemon a program which is not written to be a unix daemon. No, sorry, you are wrong. They are supervisors, and as such they ensure any long-running job keeps running when it is supposed to. Now they pretend to the role of init, which they are not. they can be, if you want them to. Some can do this better than others. And should not be. lol, why not? The program should be modified to be able to switch to background itself. lol, why? S6 even forces such a program must not handle daemon mode itself. That process must not background itself: being run by a supervision tree already makes it a background task. You say this like it is a bad thing. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
Le 28/03/2015 17:04, poitr pogo a écrit : On Sat, Mar 28, 2015 at 4:15 PM, Isaac Dunham ibid...@gmail.com wrote: (...) If you want to *manage* a daemon (ie, do something when it stops, like restarting it), it's very tricky to do it reliably without having a fixed It might be tricky if application does not support startup/shutdown procedure. That's why I would expect application to provide a script to start/stop it. I expect application to be tested on a system it is to be running on, so developing a stop/start script is not a big deal comparing to the whole test procedure. It's exactly that script which is tricky; you know, the trick with pid-files to be sure to not have two or more instances running and be able to send it signals. Usually this startup script does exactly what you don't like in daemontool, s6 and other supervisors: give an option to the daemon to tell it to not orphan itself, so that the script can use the pid to write the pid-file. Some daemons are smart enough to create the pid-file themselves; this adds yet more complexity in every daemon. Why not have an application which does that for every daemon: create pid-file and then exec the daemon? The start/stop scrpits in Debian (before Jessie) invoke start-stop-daemon to do something like that. PID; daemonizing means close stdin/stdout/stderr, then if (fork()) exit(0); And the daemon gets orphanned... is there any reason to do that? or some such thing (in other words, create a new process and exit the old one), so you do not have a fixed PID. On Linux, BSD, and older UNIX versions, this can be done thus: daemon(0,0); How this can be done in java? ;) Detaching from terminal, binding to reserved ports, dropping privileges, etc. Another s6 limitation - finish script must end in 3 seconds. or will be killed. Nice for desktop, not a server. sysvinit /etc/init.d/sendsigs actually only allows 1 second. I see it is skipping some processes (which have their own upstart scripts, and others which are recorded in special file not to be killed by sendsigs). So it (sendsigs) is for all the rest of processes which do not care about how they should be stopped. Not for that which provide its own start/stop scripts. In case of s6 it is the limit for the finish script for particular application. (...) My understanding is that s6 and some of the equivalent packages, such as daemontools, are fairly portable. They are generally not used as a way to offload daemonizing, but as a way to ensure that a service stays started. If you don't need that, you don't need them. s6 advertises itself as init replacement. I'm not using s6 or daemontools. I do not need them. Just learned about them and I'm bit surprised thy tend to replace init instead of being a good helpers for those who need them. IMHO they came into existence cause more and more applications were missing deamon part, to make sysadmin/programmer life easier. s6 doe not support those which daemonize on its own, anyway :) init has inittab for monitoring application which need continuous restart by design like getty. If a daemon needs to be monitored and restarted when it dies, bug should be fixed. I expect a deamon to work from start to stop command. No self stopping. As workaround nohup, screen, while `true`,etc are enough to keep system running until bug is fixed. I often experience rsyslog stopping for unknown reason, on Debian-Wheezy servers where it's used to log messages from remote hosts. I don't intend to debug rsyslog; I would be happy if it was restarted automatically. I do not like buggy software, so maybe that's why i do not need s6/daemontools. just IMHO. I share your dislike. However I see only one way to not run buggy software: turn off your computer... as long as software is written by human beings :-) A small and do-almost-nothing supervisor is a sensible way to monitor the daemons. But nobody forces you... Didier ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
On Sat, Mar 28, 2015 at 6:59 PM, Martijn Dekkers devuan-li...@dekkers.org.uk wrote: (...) They help to run as a unix daemon a program which is not written to be a unix daemon. No, sorry, you are wrong. They are supervisors, and as such they ensure any long-running job keeps running when it is supposed to. No. They ensure the job will be started again if it stops for some reason. :) demons vs long running jobs. programs designed and written to run in background vs those logging to stdout/stderr, but customer/business requires them to run in endless loop if we had time/money/knowledge we would make them proper daemons but we did not have. The product is not best quality but let's wrap it in a beautiful,colorful,professionally looking piece of paper :) S6 even forces such a program must not handle daemon mode itself. That process must not background itself: being run by a supervision tree already makes it a background task. You say this like it is a bad thing. naming myself an init replacement i am not supporting all old standard unix daemons :) it's not bad, it's funny ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [Dng] Why daemontools is so cool
Have a look at this excellent comparison for supervisor / init tools: http://skarnet.org/software/s6/why.html S6 is the bees' knees On 28 March 2015 at 01:32, Steve Litt sl...@troubleshooters.com wrote: Hi all, I'm managing more and more of my daemons with daemontools. This is currently happening on my sysvinit-initted Wheezy, but I think I'll do it forever. By the way, anything good I say about daemontools goes even more so for Bruce Gunter's daemontools-encore. I could have managed my daemons with sysvinit, using the respawn, and a lot of people ask me what's the point of the extra software (daemontools) when it's already doable from your init system? Here's why: * Daemontools has a rich set of tools to start, stop, and restart your daemons. * Daemontools has built-in logging that captures everything from the daemon's stdout (stderr too if you want). * Daemontools' svstat command gives you the daemon's PID: No guessing, no .pid files, no parsing ps ax. * Daemontools is ultra-reliable. * Daemontools is dead bang simple, constructed purely from Unix constructs. It's completely understandable. * Daemontools can be made to run at the end of a boot, thereby removing complexity from the init itself. * If you can devise a test to show that a required service is running correctly, you can implement service dependencies reliably with Daemontools. * Daemontools can take any old shellscript or Python/Perl/Ruby/Lua program you write and turn it into an always starts, reliably managed daemon, with logging. Daemontools is the easy way to deploy DIY daemons. * A deep understanding of daemontools, which is not hard to achieve, gives you a big head start on understanding actual init programs. * Daemontools gives you much better ways to declare that daemons run or don't run than lame, number-in-comments symlinks to five function init scripts. * Daemontools init scripts are pretty much all the same, are short and easy to understand. Contrast this to sysvinit, systemd, OpenRC or Upstart. The following is my daemontools run script for ntpd. Note that it enforces a dependency that the network be up before it starts: == #!/bin/sh DON'T START ntpd UNTIL THE NETWORK'S UP ### pingaddr=8.8.8.8 #pingaddr=8.8.8.86 # for testing unpingable IP address echo Starting Litts ntpd daemon echo Litts ntpd daemon checking network if ! ping -q -c1 $pingaddr /dev/null; then echo failed ping test on $pingaddr sleep 3 echo Bailing til next time exit 1 fi ### RUN THE DAEMON ### echo ping test succeeded, starting ntpd exec 21 echo About to start ntpd exec /usr/sbin/ntpd -d -s == For the guy who has memorized all of systemd's init keywords, this is a little bit more complicated than the equivalent systemd unit file. Foor the guy who hasn't memorized systemd's unit file keywords, this is much easier and more intuitive. Anyone with a bit of Unix knowledge and the understanding that a daemontools run script that terminates gets restarted by supervise in 1 to 5 seconds instantly understands what this script is doing, and can write it from his/her knowledge. In my opinion, no matter what init system you're running, daemontools or daemontools-encore is a great addition to make your system more understandable and maintainable. SteveT Steve Litt* http://www.troubleshooters.com/ Troubleshooting Training * Human Performance ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
