Re: zombies - solved
Thanks. Paul de Weerd wrote: ... Zombies are part of unix, you *need* them in cases. Leaving them dangling (for too long) is not good of course, clean-up is required. That's what's happening. I see that one work-around would be to have cron periodically send a kill signal to the parent. But it pains me to even mention such a lame and problematic hack. ...This is the job of the parent process so the 'generic way' to *solve* these issues is by fixing the parent process. Easier said than done given the original state of my concentration / coding skills and the subsequent deterioration from that state. ... You may want to investigate alternative options or fix the code if you can... Where is it that the problem most likely lies? Apache2, perl or the heinous 'apt-cacher' script called by Apache2? /usr/bin/perl /usr/sbin/apt-cacher -d -p /var/run/apt-cacher.pid Looking ahead, what is the timeline for moving to Apache2? Or what are the major reasons 4.3 is going to still use 1.3x? Regards, -Lars
Re: zombies - solved
On Wed, Mar 12, 2008 at 2:18 AM, Lars Noodin [EMAIL PROTECTED] wrote: Or what are the major reasons 4.3 is going to still use 1.3x? Licensing.
Re: zombies
On Wed, Mar 12, 2008 at 10:36:23AM +0200, Lars Nood??n wrote: How are zombies best dealt with, correctively? My OBSD 4.2 x86 machine is showing memory and CPU utilization are a negligable fraction of the total capacity. Yet, it is getting maxed out in regards to number of processes, apparently due to the zombies. kill -KILL seems to have no effect. zombie processes are already dead, you cannot kill them. Some interaction between Apache2 and perl is creating zombies. After several months, this number has crept up to close to a thousand and with kern.maxproc=1024, problems are starting For example, ps says : ... _apache297 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 19083 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 24147 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 30821 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 6995 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 26059 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 31087 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) ... So again, what corrective measures can be taken to rid the machine of zombie processes? And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 zombie state happend if a child process exits, but its parent did not execute a wait(2) system call (or one if its alternatives) for the process (yet). So this seem a bug in the handling of CGIs. -Otto Regards, -Lars
Re: zombies - solved
Looking ahead, what is the timeline for moving to Apache2? Likely never, unless they decide to change their license. Or what are the major reasons 4.3 is going to still use 1.3x? apache2 is not free enough.
Re: zombies - half solved
How are zombies best dealt with, correctively? Sorry to answer my own question. The solution was to find the parent process and kill it. But the second question still stands, is there a generic way to prevent the formation of zombies? The cause in this specific case is a perl-based CGI script called by apache2. Regards, -Lars
Re: zombies
On Wed, Mar 12, 2008 at 10:36:23AM +0200, Lars Nood??n wrote: | How are zombies best dealt with, correctively? By fixing the bugs in the parent. | My OBSD 4.2 x86 machine is showing memory and CPU utilization are a | negligable fraction of the total capacity. Yet, it is getting maxed out | in regards to number of processes, apparently due to the zombies. Zombies don't consume any (or, nearly any) resources apart form the one pid. | kill -KILL seems to have no effect. Nope. Read up on 'em to find out why. Hint : you can't kill what's already dead. | Some interaction between Apache2 and perl is creating zombies. After | several months, this number has crept up to close to a thousand and with | kern.maxproc=1024, problems are starting | | For example, ps says : | ... | _apache297 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 19083 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 24147 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 30821 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 6995 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 26059 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 31087 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | ... | | So again, what corrective measures can be taken to rid the machine of | zombie processes? Find the parent process and restart it. This process is buggy. You may want to investigate alternative options or fix the code if you can. | And, is there a generic way to prevent them? The cause is a perl CGI | called by apache2 Zombies are part of unix, you *need* them in cases. Leaving them dangling (for too long) is not good of course, clean-up is required. This is the job of the parent process so the 'generic way' to *solve* these issues is by fixing the parent process. Cheers, Paul 'WEiRD' de Weerd -- [++-]+++.+++[---].+++[+ +++-].++[-]+.--.[-] http://www.weirdnet.nl/
Re: zombies
On 12 March 2008, Lars NoodC)n [EMAIL PROTECTED] wrote: [...] And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 Depending on what you're doing, make the parent wait(2) for the processes or setsid(3). Regards, Liviu Daia -- Dr. Liviu Daia http://www.imar.ro/~daia
Re: zombies - solved
* Theo de Raadt [EMAIL PROTECTED] [2008-03-12 10:36]: Looking ahead, what is the timeline for moving to Apache2? Likely never, unless they decide to change their license. even then... I don't see any advatages in apache2, but lots of disadvantages and a gigantic design fault. No, not one, multiple. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg Amsterdam
Re: zombies
Hi! On Wed, Mar 12, 2008 at 12:05:29PM +0200, Liviu Daia wrote: On 12 March 2008, Lars NoodC)n [EMAIL PROTECTED] wrote: [...] And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 Depending on what you're doing, make the parent wait(2) for the processes or setsid(3). setsid(2) (yes, it's section 2 on OpenBSD) doesn't make the child lose the connection to the parent. See the source of daemon(3) for how to use setsid in connection with fork and exit (in fact _exit) to make a process disconnect from its parent and its controlling terminal etc. Kind regards, Hannah.
Re: zombies
On 12 March 2008, Hannah Schroeter [EMAIL PROTECTED] wrote: Hi! On Wed, Mar 12, 2008 at 12:05:29PM +0200, Liviu Daia wrote: On 12 March 2008, Lars NoodC)n [EMAIL PROTECTED] wrote: [...] And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 Depending on what you're doing, make the parent wait(2) for the processes or setsid(3). setsid(2) (yes, it's section 2 on OpenBSD) Yes, sorry. doesn't make the child lose the connection to the parent. No, it actually makes the calling process a session leader. See the source of daemon(3) for how to use setsid in connection with fork and exit (in fact _exit) to make a process disconnect from its parent and its controlling terminal etc. Actually, there's a bunch of other things to take care of, like signals and pipes. A more complete answer would be something like: read a book about UNIX process management; I was trying to provide a hint in the right direction, not abstract a book in a sentence. :) Regards, Liviu Daia -- Dr. Liviu Daia http://www.imar.ro/~daia
Re: zombies - solved
On 3/12/08, Lars NoodC)n [EMAIL PROTECTED] wrote: Looking ahead, what is the timeline for moving to Apache2? Or what are the major reasons 4.3 is going to still use 1.3x? Take a look at http://nginx.net/ BSD license, seems to work, but I don't know about its security profile. I'm sure it's not as secure as the OBSD Apache, but it might be ok compared with apache2.
Re: zombies
Otto == Otto Moerbeek [EMAIL PROTECTED] writes: Otto zombie state happend if a child process exits, but its parent did not Otto execute a wait(2) system call (or one if its alternatives) for the Otto process (yet). So this seem a bug in the handling of CGIs. Most likely a bug in a Perl script that forks but doesn't wait for its kid. I generally *don't* see zombies in well-written Perl programs. Was this FastCGI by any chance? I know there's unique problems related to that for naive code that creates a child, because the parent never goes away (since it's shared by the next series of CGI hits). But again, with proper care, even a FastCGI script can be written properly. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: zombies - solved
On Wed, Mar 12, 2008 at 08:39:07AM -0500, Gregg Reynolds wrote: On 3/12/08, Lars NoodC)n [EMAIL PROTECTED] wrote: Looking ahead, what is the timeline for moving to Apache2? Or what are the major reasons 4.3 is going to still use 1.3x? Take a look at http://nginx.net/ BSD license, seems to work, but I don't know about its security profile. I'm sure it's not as secure as the OBSD Apache, but it might be ok compared with apache2. There's also a port of nginx as of 4.2-current. THe port is of the stable version, not the development version. -- Darrin Chandler| Phoenix BSD User Group | MetaBUG [EMAIL PROTECTED] | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation
Re: zombies
On Wed, Mar 12, 2008 at 09:57:16AM +0100, Otto Moerbeek wrote: zombie state happend if a child process exits, but its parent did not execute a wait(2) system call (or one if its alternatives) for the process (yet). So this seem a bug in the handling of CGIs. I'd like to add a bit to the above and to Paul de Weerd's comments: Zombie processes are there to maintain a little info in case the parent process calls wait() later to retrieve it. Some program designs catch SIGCHILD or have a thread block on wait*, and in those cases the zombie lasts such a short time you'll probably never see it in top or ps. Other designs use non-blocking forms and zombies may stick around long enough to notice, but then disappear later when the parent makes a pass. If the parent dies before calling wait, then the zombie is inherited by init which will take care of it. So, zombies happen, but the only time they stay around for a long time is a negligent/misdesigned parent that is still alive but not calling wait* on the children. The OS can't make a badly written program into a well written program. So as admin you are stuck restarting the parent periodically, switching to something else, or bugging the developers to fix the problem. -- Darrin Chandler| Phoenix BSD User Group | MetaBUG [EMAIL PROTECTED] | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation
Re: zombies - solved
Theo de Raadt wrote: apache2 is not free enough. Ok. There were some additional reasons mentioned, but licensing is enough on its own. I found the old announcement now that I know what to look for: http://archives.neohapsis.com/archives/openbsd/2004-06/0448.html Apache 1.3.29 is decent enough and has the functionality, name brand recognition and familiarity needed. But without updates, it seems a dead end and not a good idea for new activities. I'm also not finding reference to IPv6 in the documentation for Apache 1.3.x either online or in the man pages and that was my main reason for even looking at Apache2. A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. Gregg proposed, nginx ( http://nginx.net/ ), which seems to be just getting started. It's under a 'BSD-like' license. It might work, but seems new. I see Lighttpd already in the 'packages' and it is under an appropriate license. In the last year, it has gained a lot in both visibility and user-base. In a lot of cases, perhaps most, new setups could be steered towards Lighttpd, if it were mentioned in the documentation here and there. I probably would have chosen it over grabbing Apache2 from the ports tree had it been mentioned. Apache2 and Lighttpd both required some adjustment and I would rather future-proof my activities, just in case they have to be supported that long. The mention of it can be small and does not need to affect how things are currently done. But as more use it, it will be easier later to drop Apache when (if) the time comes. Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. Regards, -Lars
Re: zombies
Randal L. Schwartz wrote: Most likely a bug in a Perl script that forks but doesn't wait for its kid. I generally *don't* see zombies in well-written Perl programs. ;) Was this FastCGI by any chance? No. I think it's the perl script, but now that gets added to my list of things to do. The hints about setsid(2) and wait(2) give an idea of what to look for. regards, -Lars
Re: zombies - solved
Ok. There were some additional reasons mentioned, but licensing is enough on its own. I found the old announcement now that I know what to look for: http://archives.neohapsis.com/archives/openbsd/2004-06/0448.html Apache 1.3.29 is decent enough and has the functionality, name brand recognition and familiarity needed. But without updates, it seems a dead end and not a good idea for new activities. That is 1 persons opinion, and I think you will find yourself isolated. It's just a bloody web server. It's easy. I'm also not finding reference to IPv6 in the documentation for Apache 1.3.x either online or in the man pages and that was my main reason for even looking at Apache2. There are diffs coming that add v6 support. There have been reasons not to add it in the past. A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. When we started work on OpenSSH, there were people just like you saying that it did not seem like a good return on investment. Investment. Who are you to tell us how we should spend our time, and what we should do? If you don't LIKE IT, then do whatever you want. Gregg proposed, nginx ( http://nginx.net/ ), which seems to be just getting started. It's under a 'BSD-like' license. It might work, but seems new. Huh? We've already GOT a completely working fixed one in our tree. It's fine. And we have zero interest in swapping to some other piece of shit when this piece of shit will do. Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. No.
Re: zombies - solved
If you want to serve http content via IPv6, then perhaps you can run httpd on your (IPv4) loopback interface, and have relayd listen on your public IPv6 interface, and forward requests over IPv4 to it ? /Pete On 12 Mar 2008, at 4:22 PM, Lars Noodin wrote: Theo de Raadt wrote: apache2 is not free enough. Ok. There were some additional reasons mentioned, but licensing is enough on its own. I found the old announcement now that I know what to look for: http://archives.neohapsis.com/archives/openbsd/2004-06/0448.html Apache 1.3.29 is decent enough and has the functionality, name brand recognition and familiarity needed. But without updates, it seems a dead end and not a good idea for new activities. I'm also not finding reference to IPv6 in the documentation for Apache 1.3.x either online or in the man pages and that was my main reason for even looking at Apache2. A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. Gregg proposed, nginx ( http://nginx.net/ ), which seems to be just getting started. It's under a 'BSD-like' license. It might work, but seems new. I see Lighttpd already in the 'packages' and it is under an appropriate license. In the last year, it has gained a lot in both visibility and user-base. In a lot of cases, perhaps most, new setups could be steered towards Lighttpd, if it were mentioned in the documentation here and there. I probably would have chosen it over grabbing Apache2 from the ports tree had it been mentioned. Apache2 and Lighttpd both required some adjustment and I would rather future-proof my activities, just in case they have to be supported that long. The mention of it can be small and does not need to affect how things are currently done. But as more use it, it will be easier later to drop Apache when (if) the time comes. Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. Regards, -Lars
Re: zombies - solved
Op Wed, 12 Mar 2008 17:05:01 +0100 schreef Pete Vickers [EMAIL PROTECTED]: If you want to serve http content via IPv6, then perhaps you can run httpd on your (IPv4) loopback interface, and have relayd listen on your public IPv6 interface, and forward requests over IPv4 to it ? And then what if the HTTP request reads something like GET [::1] ? -- Boudewijn Dijkstra Indes - IDS B.V. +31 345 545 535
Re: zombies - solved
Lars NoodC)n wrote: Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. Why do some people think Apache needs to be replaced? Moreover, if the developers are satisfied with Apache 1.3, why would the recommend another product in the documentation?
Re: zombies - solved
On Wed, Mar 12, 2008 at 11:58 AM, Theo de Raadt [EMAIL PROTECTED] wrote: A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. I'm just curious what is in 2.x that you need, that is unavailable in 1.3? When we started work on OpenSSH, there were people just like you saying that it did not seem like a good return on investment. Investment. Who are you to tell us how we should spend our time, and what we should do? If you don't LIKE IT, then do whatever you want. Well, obviously we want an upgrade to Apache 2, and an upgrade to Apache 3 when that comes out. If only you are not so selfish as to go on mountain climbing hikes, and satay eating binges, then you'll definitely have time to invest in upgrading to Apache v3! :) -- http://www.glumbert.com/media/shift http://www.youtube.com/watch?v=tGvHNNOLnCk This officer's men seem to follow him merely out of idle curiosity. -- Sandhurst officer cadet evaluation. Securing an environment of Windows platforms from abuse - external or internal - is akin to trying to install sprinklers in a fireworks factory where smoking on the job is permitted. -- Gene Spafford learn french: http://www.youtube.com/watch?v=j1G-3laJJP0feature=related
Re: zombies - solved
On 2008-03-12, Pete Vickers [EMAIL PROTECTED] wrote: If you want to serve http content via IPv6, then perhaps you can run httpd on your (IPv4) loopback interface, and have relayd listen on your public IPv6 interface, and forward requests over IPv4 to it ? Here's a better way: test the diffs at http://mini.vnode.ch/ and provide feedback.
Re: zombies - solved
bofh wrote: On Wed, Mar 12, 2008 at 11:58 AM, Theo de Raadt [EMAIL PROTECTED] wrote: A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. I'm just curious what is in 2.x that you need, that is unavailable in 1.3? mod_proxy_balancer Jonathan -- Jonathan Weiss http://blog.innerewut.de
Re: zombies - solved
Quoting Jonathan Weiss [EMAIL PROTECTED]: bofh wrote: On Wed, Mar 12, 2008 at 11:58 AM, Theo de Raadt [EMAIL PROTECTED] wrote: A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. I'm just curious what is in 2.x that you need, that is unavailable in 1.3? mod_proxy_balancer Ok, you have a need for Apache 2.x. That does not mean that the Apache server in the base install needs to be updated. http://www.openbsd.org/cgi-bin/cvsweb/ports/www/apache-httpd/ -- Tim Donahue This message was sent using IMP, the Internet Messaging Program.
IPv6 web servers (was Re: zombies - solved)
Markus Lude wrote: mbalmer@ posted a diff for IPv6 support for the base apache back last december: see http://mini.vnode.ch/ Excellent. What, in general, are the plans? (Any answer is fine.) Knowing more reduces the unnecessary questions, experiments and speculations that get in the way. My interest in this is having an IPv6 web server on OpenBSD so I can try an in-house IPv6 pilot sometime this year. The timeline for that is probably May or August. In general, it's preferable for me to follow -STABLE because the less I have to tweak the less likely I am to break something and the less custom documentation I need to provide to pass it on to others. I plan to rebuild for OBSD 4.3 in May, I'll try the IPv6 patch (or however it happens to be available) for 1.3 then. Going back to 1.3 will actually save me work and it's what I'm most familiar with already. Apache 2.2.x is in ports if you really need it. Some ports have an -ap2 flavor for that newer version. That's what I grabbed last autumn, but aside from IPv6 there's nothing needed that was/is specific to Apache 2. regards, -Lars
Re: zombies - solved
On Wed, Mar 12, 2008 at 12:19:18PM -0400, bofh wrote: | A fork does not seem like a good return on investment, so v 1.3.29 will | probably go away sooner than later once the Apache Foundation drops | maintenance on the 1.3 series. | | | I'm just curious what is in 2.x that you need, that is unavailable in 1.3? The only reason I run Apache 2 on my OpenBSD machine is IPv6. There's patches for 1.3, but for now we chose Apache 2. I know there's people working on integrating the v6 patches in OpenBSD and I hope those make it for 4.4, but we'll see how it goes. Paul 'WEiRD' de Weerd -- [++-]+++.+++[---].+++[+ +++-].++[-]+.--.[-] http://www.weirdnet.nl/
Re: zombies - half solved
Lars wrote:
But the second question still stands, is there a generic way to prevent
the formation of zombies? The cause in this specific case is a
perl-based CGI script called by apache2.
The easiest way might be to let perl auto-reap the children for you.
It's as simple as prepending this line within the block that spawns the
child processes:
local $SIG{CHLD} = 'IGNORE'; # straight outa perlipc(1)
One thing to watch out for though is that you may get weird side effects
if you set this and then use system() in the same scope. If so, just
use wait/waitpid instead (it's only a few more lines of code).
--
Stephen Takacs [EMAIL PROTECTED] http://perlguru.net/
4149 FD56 D078 C988 9027 1EB4 04CC F80F 72CB 09DA
