Re: Bug#126750: klogd should optionally be started from init(8)
On Sat, 29 Dec 2001, Henrique de Moraes Holschuh wrote: This is bogus, anything can die in an OOM situation. Are you going to put all daemons into inittab? True, true. However, sysklogd and klogd are logging daemons. They deserve some special treatment IMHO. Actually, I am pondering doing such a thing to sshd on my remote systems, too. I think it would be far better to have the logging daemons and ssh checked by the watchdog daemon. IMO it is not a good idea to restart the daemons when we are OOM anyway, since this might do real damage, a clean reboot would be better. You should be trying to avoid OOM situations in the first place. That is not always possible, and sometimes a kernel VM screwup will cause it, no? Hrm, I have 768 megs of swap, and watchdog is set up to reboot before OOM happens, which never did, so it is possible. In the case of a VM screwup, I'd like my box to reboot and send me a mail rather than let mysql die in the middle of a transaction and serve database connection refused pages to everyone until apache also dies. Simon -- GPG public key available from http://phobos.fs.tum.de/pgp/Simon.Richter.asc Fingerprint: DC26 EB8D 1F35 4F44 2934 7583 DBB6 F98D 9198 3292 Hi! I'm a .signature virus! Copy me into your ~/.signature to help me spread!
Re: Bug#126750: klogd should optionally be started from init(8)
Thanks a lot folks, you provided good arguments with these two bug reports. I've considered the issue on my own as well and came to a different implementation. Instead of making syslogd/klogd controlled by init they will now be restarted by regular cron scripts if they got lost in the meantime. This requires a running cron, of course. However, after an OOM situation you are most probably beaten by *something* anyway. I hope this meets your needs as well. Regards, Joey -- Computers are not intelligent. They only think they are. Please always Cc to me when replying to me on the lists.
Re: Bug#126750: klogd should optionally be started from init(8)
On Sun, 30 Dec 2001, Dominik Kubla wrote: On Sat, Dec 29, 2001 at 11:02:39PM -0200, Henrique de Moraes Holschuh wrote: Such a table should not (and needs not) to benefit processes running by someone else than root, unless you wanted to do such a thing on purpose and coded it like that. Root processes already have a much lesser probability of being killed. If I want a table to fix the issue with the ordering on time of two or three processes, I would need to be quite stupid to do something that screwed up the OOM balancing so much that users would start naming their processes syslog or named to avoid them being killed. So you are proposing to run all kind of daemons as root to avoid them being hit by the OOM killer early? Did I say that? Please. I am not stupid. I NEVER said anything about all kind of daemons. I will end up getting insulted if you keep trying to read such things between my lines. I am very aware of the amount of danger on running stuff as root, especially when they do not need it. I want the LOGGING daemons (i.e. only syslog and klogd), which ALREADY run as root, to be restarted should they die. Due to OOM killer, due to segfaults. Whatever. Stop preaching to the choir. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh
Re: Bug#126750: klogd should optionally be started from init(8)
On Sun, 30 Dec 2001, Dominik Kubla wrote: On Sun, Dec 30, 2001 at 08:13:36PM -0200, Henrique de Moraes Holschuh wrote: I want the LOGGING daemons (i.e. only syslog and klogd), which ALREADY run as root, to be restarted should they die. Due to OOM killer, due to segfaults. Whatever. It could be argued that those need not be run as root. All they need are the necessary capabilities: Indeed. In which case just using init to keep them alive, and leaving the OOM killer alone might be best. The syscall aproach could still be used I suppose, but it gets a bit more complicated. daemons. If they fail, they fail. The system still works. But if they Well, the system still works, but logging stops. And sometimes that log information might have been very useful. are not killed by OOM because of an excemption the system might kill the nfs daemon or the sendmail process. So where do you draw the line? Good question. I would not bother with anything else other than ssh and logging (and the kernel threads, i suppose). And all that just to make them harder (not impossible) to get killed by the OOM killer. BTW, I consider tweaking with the OOM killer a very partial solution, and not a very good idea. Something that would restart a process the admin told it to (i.e. init, or something like it) is a better solution, as it does not touch the kernel, and it will restart services that got ill. Keep in mind that if the system is so resource starved that the OOM mechanism kicks in, the failing processes might not be able to log anything meaningful anyway. Yes, I know this quite well. I've been through an OOM killer rampage once or twice with an older 2.4.x kernel. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh
Re: Bug#126750: klogd should optionally be started from init(8)
Henrique de Moraes Holschuh [EMAIL PROTECTED] wrote:
True, true. However, sysklogd and klogd are logging daemons. They deserve
some special treatment IMHO.
Even so, starting it from inittab is too much of a kludge. For one thing,
it means that /etc/init.d/syslogd stop will either not work, or be an
ugly hack that fools around with inittab.
In any case, if the OOM killer has moved onto syslogd, then you've probably
lost control of the box anyway so restarting it is pretty pointless.
That is not always possible, and sometimes a kernel VM screwup will cause
it, no?
Why should we set up such ugly work arounds for kernel bugs or incompetent
admins?
--
Debian GNU/Linux 2.2 is out! ( http://www.debian.org/ )
Email: Herbert Xu ~{PmVHI~} [EMAIL PROTECTED]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: Bug#126750: klogd should optionally be started from init(8)
On Sat, Dec 29, 2001 at 02:40:41AM -0200, Henrique de Moraes Holschuh wrote: You should be trying to avoid OOM situations in the first place. That is not always possible, and sometimes a kernel VM screwup will cause it, no? Hmm.. OOM Killer should avoid killing long running root daemons, whats is speacial on your system that it affects the klogd so often? If we solve that, then having a perodic process checking for all daemons is probably better. On the other hand, having a generic interface to register (move) daemons from rc.d to inittab would be a nice thing. Because for example sshd is so critical to me I like to move it to inittab often, too. Greetings Bernd -- (OO) -- [EMAIL PROTECTED] -- ( .. ) [EMAIL PROTECTED],linux.de,debian.org} http://home.pages.de/~eckes/ o--o *plush* 2048/93600EFD [EMAIL PROTECTED] +497257930613 BE5-RIPE (OO) When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!
Re: Bug#126750: klogd should optionally be started from init(8)
On Sat, 29 Dec 2001, Herbert Xu wrote: Henrique de Moraes Holschuh [EMAIL PROTECTED] wrote: True, true. However, sysklogd and klogd are logging daemons. They deserve some special treatment IMHO. Even so, starting it from inittab is too much of a kludge. For one thing, It is far better than anything else I can think of. Fiddling with the OOM killer to avoid killing syslog and klog is worse, for example. Writing something else to do what init already does, and does well is pointless, too. This could be done in a nice, generic way so that we have a common interface to add stuff to inittab, sort of like update-rd.d. Then, it becomes less of a disgusting hack. it means that /etc/init.d/syslogd stop will either not work, or be an ugly hack that fools around with inittab. Yes, that part disgusts me. Writing a update-rc.d-like interface (or update-inetd-like, for that matter) to do it is actually a good idea, and probably the way to go. In any case, if the OOM killer has moved onto syslogd, then you've probably lost control of the box anyway so restarting it is pretty pointless. Not really. Forsenics are always useful. That is not always possible, and sometimes a kernel VM screwup will cause it, no? Why should we set up such ugly work arounds for kernel bugs or incompetent admins? I don't know, maybe because it is useful for real? And because the current kernel VM is too incompetent at doing its job? I agree we should not setup an ugly work-around, but we can design, write and setup a non-ugly inittab interface, and use that. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh
Re: Bug#126750: klogd should optionally be started from init(8)
Martin Schulze [EMAIL PROTECTED] writes:
Florian Weimer wrote:
The package installation scripts should offer to run klogd from
inittab, since klogd regularly dies in OOM situations and is not
restarted if the current mechanism is used.
IMHO the right solution is to slowly replace sysvinit's init.d
with something that can monitor whether the children are still
alive. For _everything_.
I know many of you hate DJB, but his daemontools is a good
idea (though I do dislike the implementation, atleast parts of
it).
--
[EMAIL PROTECTED],havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com}
double a,b=4,c;main(){for(;++a2e6;c-=(b=-b)/a++);printf(%f\n,c);}
Re: Bug#126750: klogd should optionally be started from init(8)
also sprach Tommi Virtanen [EMAIL PROTECTED] [2001.12.29.2035 +0100]:
IMHO the right solution is to slowly replace sysvinit's init.d
with something that can monitor whether the children are still
alive. For _everything_.
ntpdate??? for instance...
surely not everything, but everything that qualifies as a daemon...
I know many of you hate DJB, but his daemontools is a good
idea (though I do dislike the implementation, atleast parts of
it).
he is an asshole, but his software is good.
[EMAIL PROTECTED],havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com}
^
h! someone i can bug when full cluster decides to not work the way i
want... :)
--
martin; (greetings from the heart of the sun.)
\ echo mailto: !#^.*|tr * mailto:; [EMAIL PROTECTED]
(a)bort, (r)etry, (p)retend this never happened
pgpeBFHbUHaO5.pgp
Description: PGP signature
Re: Bug#126750: klogd should optionally be started from init(8)
On Sat, Dec 29, 2001 at 02:09:36PM -0200, Henrique de Moraes Holschuh wrote:
It is far better than anything else I can think of. Fiddling with the OOM
killer to avoid killing syslog and klog is worse, for example. Writing
Nope, that's exactly what the OOM killer was designed to do. Processes
like syslogd is meant to be the last ones to be killed.
--
Debian GNU/Linux 2.2 is out! ( http://www.debian.org/ )
Email: Herbert Xu ~{PmVHI~} [EMAIL PROTECTED]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: Bug#126750: klogd should optionally be started from init(8)
On Sun, 30 Dec 2001, Herbert Xu wrote: On Sat, Dec 29, 2001 at 02:09:36PM -0200, Henrique de Moraes Holschuh wrote: It is far better than anything else I can think of. Fiddling with the OOM killer to avoid killing syslog and klog is worse, for example. Writing Nope, that's exactly what the OOM killer was designed to do. Processes like syslogd is meant to be the last ones to be killed. Well, you're the authority on the kernel here. If you think it is a good idea to modify the OOM, I will not disagree. Such mod to the OOM would at the very least give us time to leisurely write an inittab control interface (generic, might be useful for a lot of stuff), and test and deploy it. It would be useful for the *getty packages, for example. Then it would be simple to make such changes locally, or based on debconf requests of user preference (or not at all) for any package that might benefit from it. I am not at ease to go poking on the OOM, though. Someone else better used to kernel programming should do it... -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh
Re: Bug#126750: klogd should optionally be started from init(8)
Henrique de Moraes Holschuh [EMAIL PROTECTED] wrote:
Nope, that's exactly what the OOM killer was designed to do. Processes
like syslogd is meant to be the last ones to be killed.
I am not at ease to go poking on the OOM, though. Someone else better used
to kernel programming should do it...
The OOM killer should already do this as it is, no modifications are
required...
--
Debian GNU/Linux 2.2 is out! ( http://www.debian.org/ )
Email: Herbert Xu ~{PmVHI~} [EMAIL PROTECTED]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: Bug#126750: klogd should optionally be started from init(8)
(not cc'ed to the bts) On Sun, 30 Dec 2001, Herbert Xu wrote: Henrique de Moraes Holschuh [EMAIL PROTECTED] wrote: Nope, that's exactly what the OOM killer was designed to do. Processes like syslogd is meant to be the last ones to be killed. I am not at ease to go poking on the OOM, though. Someone else better used to kernel programming should do it... The OOM killer should already do this as it is, no modifications are required... I've just read the OOM killer in 2.4.17. It has provisions that tend NOT to select klogd or syslogd in a system that has been running for a while, but it may end up killing them. It does this by assuming that such important stuff was started early, and stayed put (i.e. was not restarted)... at least that is what I understood from the code and comments. Now, if you just upgraded sysklogd, or otherwise caused an important process to restart, you are less safe against the OOM trying to kill what you did NOT want it to kill. Nowhere does it use the process name to lessen the chances of killing a process. IMHO it would be a nice idea to have such a whitelist just in case. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh
Re: Bug#126750: klogd should optionally be started from init(8)
On Sun, 30 Dec 2001 01:03, Dominik Kubla wrote: On Sat, Dec 29, 2001 at 09:47:27PM -0200, Henrique de Moraes Holschuh wrote: Nowhere does it use the process name to lessen the chances of killing a process. IMHO it would be a nice idea to have such a whitelist just in case. Extremely bad idea... All of a sudden every process somebody does not want to be killed is called syslog or named or what ever. True. Let's face it folks: if the OOM killer hits you constantly you have exhausted your resources. The solution is to buy new resources, not to invent convoluted schemes to make a bad situatation last longer. There are other options. You could have a root-only syscall which says don't kill me to go with the root-only syscall for don't page me out. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Bug#126750: klogd should optionally be started from init(8)
On Sun, 30 Dec 2001, Russell Coker wrote: On Sun, 30 Dec 2001 01:03, Dominik Kubla wrote: On Sat, Dec 29, 2001 at 09:47:27PM -0200, Henrique de Moraes Holschuh wrote: Nowhere does it use the process name to lessen the chances of killing a process. IMHO it would be a nice idea to have such a whitelist just in case. Extremely bad idea... All of a sudden every process somebody does not want to be killed is called syslog or named or what ever. True. Not just like that. Read the code you two before you start talking nonsense. Such a table should not (and needs not) to benefit processes running by someone else than root, unless you wanted to do such a thing on purpose and coded it like that. Root processes already have a much lesser probability of being killed. If I want a table to fix the issue with the ordering on time of two or three processes, I would need to be quite stupid to do something that screwed up the OOM balancing so much that users would start naming their processes syslog or named to avoid them being killed. As for root naming processes syslog to avoid them being killed, there is something to be said about stupidity. Let's face it folks: if the OOM killer hits you constantly you have exhausted your resources. The solution is to buy new resources, not to invent convoluted schemes to make a bad situatation last longer. Lets face it: it would be best if the logging stuff did not get killed without reason, and would restart itself in such cases. There IS a lot of advantage on resilience. Improving resilience is not futile, but it is not an substitute to good administration. Does that mean one should never bother to improve the tools to be more resilient, because good administration is more important? IMHO, no. I dislike important crap that dies too easily, and I think init is the natural choice for taking care of [the very few] processes that need to be shephered, because it already does just that, and does it very well. I fail to see why that is a convoluted solution, and why it is good to have frail syslog daemons, and nothing to kick their back into life should they die, but maybe I am myoptic... as for the OOM, it was not MY idea to fiddle with it. There are other options. You could have a root-only syscall which says don't kill me to go with the root-only syscall for don't page me out. Yes. This is another way to do it, and it is a good one too. Harder to implement, since you need to somehow keep that 'don't kill me' state stored somewhere, but it is a better solution than the table, I suppose. But do recall it must mean I am important, schedule me to die last, not don't kill me. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh
Re: Bug#126750: klogd should optionally be started from init(8)
What do people think? Please copy mails that you consider important in this context to [EMAIL PROTECTED] and [EMAIL PROTECTED] so they get recorded properly. Regards, Joey Florian Weimer wrote: Package: klogd Version: 1.4.1-8 Severity: wishlist Tags: security The package installation scripts should offer to run klogd from inittab, since klogd regularly dies in OOM situations and is not restarted if the current mechanism is used. -- System Information Debian Release: 3.0 Architecture: i386 Kernel: Linux CERT 2.4.14-xfs #1 SMP Fri Nov 23 21:34:33 CET 2001 i686 Locale: LANG=C, LC_CTYPE=en_US.ISO-8859-1 Versions of packages klogd depends on: ii libc6 2.2.4-7GNU C Library: Shared libraries an ii sysklogd 1.4.1-8System Logging Daemon ii sysklogd [system-log-daemon] 1.4.1-8System Logging Daemon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- GNU GPL: The source will be with you... always. Please always Cc to me when replying to me on the lists.
Re: Bug#126750: klogd should optionally be started from init(8)
Florian Weimer wrote:
Package: klogd
Version: 1.4.1-8
Severity: wishlist
Tags: security
The package installation scripts should offer to run klogd from
inittab, since klogd regularly dies in OOM situations and is not
restarted if the current mechanism is used.
This is bogus, anything can die in an OOM situation. Are you going to
put all daemons into inittab?
You should be trying to avoid OOM situations in the first place.
--
Debian GNU/Linux 2.2 is out! ( http://www.debian.org/ )
Email: Herbert Xu ~{PmVHI~} [EMAIL PROTECTED]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: Bug#126750: klogd should optionally be started from init(8)
On Sat, 29 Dec 2001, Herbert Xu wrote: This is bogus, anything can die in an OOM situation. Are you going to put all daemons into inittab? True, true. However, sysklogd and klogd are logging daemons. They deserve some special treatment IMHO. Actually, I am pondering doing such a thing to sshd on my remote systems, too. You should be trying to avoid OOM situations in the first place. That is not always possible, and sometimes a kernel VM screwup will cause it, no? -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh
