Re: Adding zombies to a pf table?
Hi Benny/et al,
On 2015-09-24 Thu 14:42 PM |, Benny Lofgren wrote:
> On 2015-09-24 11:37, Pantelis Roditis wrote:
> > On 09/24/2015 11:39 AM, Peter Hessler wrote:
> >> On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote:
> >> :Hello,
> >> :
> >> :Zombies are often attacking ports which don't have services running,
> >> :such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc.
> >>
> >
> > Hi,
> >
> > This is the exact reason why we created bofh-divert[1]. The idea is that
> > you pass those packets with PF to a divert socket opened by a daemon.
> > The daemon grabs the source IP and adds it to a predefined table.
>
> I've used one of the inetd "trivial services" (echo, discard, chargen,
> daytime or time) for this purpose, in combination with a couple of PF
> rules. Something like this:
>
> match in log on egress from any to tag honeypot
> pass in log tagged honeypot rdr-to 127.0.0.1 port echo keep state \
> (max-src-conn-rate 1/30, overload flush global)
>
Late reply (g.)
Here's what I'm using at the moment (not an expert.):
$ fgrep inetd /etc/rc.conf.local
inetd_flags=''
$ fgrep daytime /etc/inetd.conf
daytime stream tcp nowait rootinternal
#daytimestream tcp6nowait rootinternal
daytime dgram udp waitrootinternal
#daytimedgram udp6waitrootinternal
I noticed daytime is quicker than echo, probably because it terminates
the connection. The other end holding chargen open could use up
bandwidth fast. Same for discard if it keeps sending in junk. echo might
confuse the zombie the most. I don't use IPv6 yet.
/etc/pf.conf bits:
table persist counters file "/etc/pf/tables/dynamic/scanners.txt"
set skip on lo
set block-policy drop
match in all scrub ( random-id reassemble tcp )
#
# #
# Zombie tagging #
# #
#
match in log on $ext_if inet proto tcp \
to $ext_if:0 port { telnet, ms-sql-s, epmap, 3389:3391, auth, mysql,
8080, \
microsoft-ds, 9200, pptp, 5900:5905, 2220:2226, 4899, 502, 81,
91 } \
tag scanners
match in log on $ext_if inet proto udp \
to $ext_if:0 port { sip, 13270:13290, ssdp, 63875, netbios-ns, 3551,
sunrpc, \
l2tp, asf-rmcp, 5351, 47808 } \
tag scanners
match in log on $ext_if inet proto tcp \
to { nolisting.nuke-spam, highlisting.nuke-spam } port != smtp \
tag scanners
match in log on $ext_if inet proto udp \
to { nolisting.nuke-spam, highlisting.nuke-spam } \
tag scanners
###
### ###
### FILTERING ###
### ###
###
#
# #
# Blocking#
# #
#
block in all
block in log on $ext_if
block return in on $lan_if
block return out
#
# #
# Honeypot zombie trapping#
# #
#
pass in log from ! tagged scanners \
rdr-to localhost port daytime keep state \
( \
max-src-conn 1, max-src-conn-rate 1/1, \
sloppy, overload flush global \
)
# rules for legit services
#
# #
# the end:#
# #
#
block in log from
block in log from
block in log from
block in log on $ext_if inet proto tcp from any os "NMAP"
block in from $bogon
block return out to $bogon
block on ! lo inet6
antispoof for { lo0 $lan_if $ext_if } inet
78.33.153.148 is this machine's main external interface,
.157 & .158 are aliases & .159 is the broadcast address.
It's rather effective at trapping this sort of shite:
Oct 10 15:03:34.888900 40.118.131.81.5272 > 78.33.153.157.3389: tcp 0 (DF)
Oct 10 15:03:37.888052 40.118.131.81.5272 > 78.33.153.157.3389: tcp 0 (DF)
Oct 10 15:03:43.897752 40.118.131.81.5272 > 78.33.153.157.3389: tcp 0 (DF)
Oct 10 15:03:52.529205 40.118.131.81.
Re: Adding zombies to a pf table?
On 2015 Sep 24 (Thu) at 12:37:03 +0300 (+0300), Pantelis Roditis wrote:
:On 09/24/2015 11:39 AM, Peter Hessler wrote:
:>On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote:
:>:Hello,
:>:
:>:Zombies are often attacking ports which don't have services running,
:>:such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc.
:>
:
:Hi,
:
:This is the exact reason why we created bofh-divert[1]. The idea is that you
:pass those packets with PF to a divert socket opened by a daemon. The daemon
:grabs the source IP and adds it to a predefined table.
:
:The rules look something like this
:
:-- pf.conf snip --
:
:table persist counters
:
:block in log quick from
:
:pass in log quick on { egress } inet proto tcp from ! to port {
:3389, 5900, 6001, 8080, } divert-packet port 1100 no state
:
:-- pf.conf snip --
:
:We have used this on some of our firewalls for some time now without
:problems.
:
:>I've been playing with this, too. Overload won't work until the packet
:>is processed by a userland process.
:>
:>:Or is there something handy in ports to help?
:>:
:>
:>I don't know of any, but I have such a thing on my TODO.
:>
:
:The port[2] is under cleanup/testing and will be submitted for review soon.
:
:I hope this is close to what you guys were looking for.
:
:
:[1] https://github.com/echothrust/pf-diverters
:[2] https://github.com/echothrust/OpenBSD-ports-mystuff
:
Yes, this looks very close to what I had in mind.
Main comment: looks like no IPv6 support.
--
In Boston, it is illegal to hold frog-jumping contests in nightclubs.
Re: Adding zombies to a pf table?
On 09/24/2015 11:39 AM, Peter Hessler wrote:
On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote:
:Hello,
:
:Zombies are often attacking ports which don't have services running,
:such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc.
Hi,
This is the exact reason why we created bofh-divert[1]. The idea is that
you pass those packets with PF to a divert socket opened by a daemon.
The daemon grabs the source IP and adds it to a predefined table.
The rules look something like this
-- pf.conf snip --
table persist counters
block in log quick from
pass in log quick on { egress } inet proto tcp from ! to port {
3389, 5900, 6001, 8080, } divert-packet port 1100 no state
-- pf.conf snip --
We have used this on some of our firewalls for some time now without
problems.
I've been playing with this, too. Overload won't work until the packet
is processed by a userland process.
:Or is there something handy in ports to help?
:
I don't know of any, but I have such a thing on my TODO.
The port[2] is under cleanup/testing and will be submitted for review soon.
I hope this is close to what you guys were looking for.
[1] https://github.com/echothrust/pf-diverters
[2] https://github.com/echothrust/OpenBSD-ports-mystuff
Re: Adding zombies to a pf table?
On 09/24/2015 12:48 PM, Peter Hessler wrote: On 2015 Sep 24 (Thu) at 12:37:03 +0300 (+0300), Pantelis Roditis wrote: :On 09/24/2015 11:39 AM, Peter Hessler wrote: :>On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote: :>:Hello, :>: :>:Zombies are often attacking ports which don't have services running, :>:such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. :> [..] : :[1] https://github.com/echothrust/pf-diverters :[2] https://github.com/echothrust/OpenBSD-ports-mystuff : Yes, this looks very close to what I had in mind. Main comment: looks like no IPv6 support. I know, unfortunately my familiarity with anything IPv6 is close to 0. However it shouldn’t be too hard to add the support. If anyone is interested in taking the task I am happy to accept pull requests or patches.
Re: Adding zombies to a pf table?
On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote: :Hello, : :Zombies are often attacking ports which don't have services running, :such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. : :With a default pf block drop in on $ext_if, how can those source ips be :added to a table? Which all can be dropped & small queued. : :I've tried to overload a match statement, but that won't work. : I've been playing with this, too. Overload won't work until the packet is processed by a userland process. :Or is there something handy in ports to help? : I don't know of any, but I have such a thing on my TODO. Annoyingly, that TODO list is too long. If you beat me to it, please share :). :Thanks. :-- :By the time they had diminished from 50 to 8, :the other dwarves began to suspect "Hungry" ... :-- Gary Larson, "The Far Side" : -- Ed Sullivan will be around as long as someone else has talent. -- Fred Allen
Re: Adding zombies to a pf table?
Am Donnerstag, den 24.09.2015, 10:39 +0200 schrieb Peter Hessler: > On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote: > :Zombies are often attacking ports which don't have services running, > :such as telnet (most popular indeed), mysql, 3551, 8080, 13272, > etc. > : [..] > :I've tried to overload a match statement, but that won't work. > : > > I've been playing with this, too. Overload won't work until the > packet > is processed by a userland process. I remember to have done it once. But when I look into that old configuration, I am not sure whether the "synproxy state" or the "rdr-to 127.0.0.1 port 9" part of the rule did the trick. -- David Dahlberg Fraunhofer FKIE, Dept. Communication Systems (KOM) | Tel: +49-228-9435-845 Fraunhoferstr. 20, 53343 Wachtberg, Germany| Fax: +49-228-856277
Re: Adding zombies to a pf table?
On 2015-09-24 11:37, Pantelis Roditis wrote: > On 09/24/2015 11:39 AM, Peter Hessler wrote: >> On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote: >> :Hello, >> : >> :Zombies are often attacking ports which don't have services running, >> :such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. >> > > Hi, > > This is the exact reason why we created bofh-divert[1]. The idea is that > you pass those packets with PF to a divert socket opened by a daemon. > The daemon grabs the source IP and adds it to a predefined table. I've used one of the inetd "trivial services" (echo, discard, chargen, daytime or time) for this purpose, in combination with a couple of PF rules. Something like this: match in log on egress from any to tag honeypot pass in log tagged honeypot rdr-to 127.0.0.1 port echo keep state \ (max-src-conn-rate 1/30, overload flush global) Regards, /Benny PS. Who named unlistened-to ports "zombies" anyway? I've never heard that before. A zombie in a unix context have always been one thing and one thing only - a dead process that has yet to be wait()ed for by its parent.
Re: Adding zombies to a pf table?
On Thu, Sep 24, 2015 at 02:42:47PM +0200, Benny Lofgren wrote: > On 2015-09-24 11:37, Pantelis Roditis wrote: > > On 09/24/2015 11:39 AM, Peter Hessler wrote: > >> On 2015 Sep 23 (Wed) at 18:14:51 +0100 (+0100), Craig Skinner wrote: > >> :Hello, > >> : > >> :Zombies are often attacking ports which don't have services running, > >> :such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. > >> > > > > Hi, > > > > This is the exact reason why we created bofh-divert[1]. The idea is that > > you pass those packets with PF to a divert socket opened by a daemon. > > The daemon grabs the source IP and adds it to a predefined table. > > I've used one of the inetd "trivial services" (echo, discard, chargen, > daytime or time) for this purpose, in combination with a couple of PF > rules. Something like this: > > match in log on egress from any to tag honeypot > pass in log tagged honeypot rdr-to 127.0.0.1 port echo keep state \ > (max-src-conn-rate 1/30, overload flush global) > > > Regards, > /Benny > > > PS. Who named unlistened-to ports "zombies" anyway? I've never heard > that before. A zombie in a unix context have always been one thing and > one thing only - a dead process that has yet to be wait()ed for by its > parent. Zombie is also a pc taken over bij malware. -Otto
Re: Adding zombies to a pf table?
Thanks for all the helpful replies. On 2015-09-23 Wed 18:14 PM |, Craig Skinner wrote: > > Zombies are often attacking ports which don't have services running, > such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. > This was logged from Friday - Monday (zombies love the weekend)... Blocked logged connections from hosts in tables & count destination 12957 25 tcp 5396 23 tcp 3703 22 tcp 1578 1433 tcp 638 80 tcp 545 5060 udp 393 13282 udp 373 13272 udp 358 13281 udp 330 13283 udp 305 135 tcp 281 53 udp 269 3389 tcp 222 123 udp 210 443 tcp 208 113 tcp 194 3306 tcp 192 8080 tcp 124 445 tcp 124 1900 udp 118 9200 tcp 102 1723 tcp 93 63875 udp 82 137 udp 76 5902 tcp 75 tcp 74 5900 tcp 70 3551 udp 69 4899 tcp 67 19 udp 67 161 udp 64 53413 udp 61 5901 tcp 56 502 tcp 54 53 tcp 52 50571 udp 52 43022 udp 50 111 udp 48 2228 tcp 48 2223 tcp 47 110 tcp 40 81 tcp 40 3128 tcp 38 91 tcp 38 21320 tcp 38 1701 udp 34 520 udp 32 2226 tcp 32 2225 tcp 32 2224 tcp 31 8000 tcp 30 5351 udp 30 47808 udp 30 139 tcp 29 5093 udp 29 49153 udp 28 623 udp 27 441 tcp 26 27017 tcp 26 1434 udp 26 11211 tcp 24 30022 tcp 20 6379 tcp 19 17 udp 18 14435 tcp 18 1234 tcp 17 995 tcp 17 143 tcp 16 9443 tcp 16 5903 tcp 16 2227 tcp 16 22012 tcp 16 11911 tcp 15 8081 tcp 14 8800 tcp 14 4000 tcp 13 8443 tcp 13 5000 tcp 13 3443 tcp 12 tcp 12 5070 udp 12 5062 udp 12 5061 udp 12 33436 udp 11 5800 tcp 10 8123 tcp 10 8118 tcp 10 tcp 10 44818 udp 10 2022 tcp 9 tcp 9 80 udp 9 tcp 9 442 tcp 9 3444 tcp 9 21 tcp 9 2082 tcp 9 10444 tcp 8 9080 tcp 8 9000 tcp 8 843 tcp 8 8291 tcp 8 808 tcp 8 8022 tcp 8 8001 tcp 8 7003 tcp 8 6060 udp 8 5905 tcp 8 5904 tcp 8 5069 udp 8 5068 udp 8 5067 udp 8 5066 udp 8 5065 udp 8 5064 udp 8 5063 udp 8 5060 tcp 8 34352 tcp 8 27164 tcp 8 26600 tcp 8 25955 tcp 8 22122 tcp 8 2066 tcp 8 2055 tcp 8 2044 tcp 8 2033 tcp 8 1991 tcp 8 1218 tcp 8 tcp 8 10155 tcp 7 3 tcp 7 2323 tcp 7 2 tcp 7 1911 tcp 7 18000 tcp 7 1337 tcp 6 9797 tcp 6 9393 tcp 6 9090 tcp 6 9001 tcp 6 8140 tcp 6 8090 tcp 6 8089 tcp 6 8086 tcp 6 7808 tcp 6 7547 tcp 6 7004 tcp 6 tcp 6 63000 tcp 6 6006 tcp 6 5353 udp 6 37564 tcp 6 3 tcp 6 3130 tcp 6 3129 tcp 6 25967 tcp 6 2083 tcp 6 18186 tcp 6 14410 tcp 6 1080 tcp 5 9600 tcp 5 9051 tcp 5 5432 tcp 5 5007 tcp 5 1883 tcp 5 12345 tcp 5 11 tcp 4 9993 tcp 4 9987 udp 4 9527 tcp 4 9160 tcp 4 902 tcp 4 9010 tcp 4 9009 tcp 4 9008 tcp 4 9007 tcp 4 9006 tcp 4 9005 tcp 4 9004 tcp 4 9003 tcp 4 9002 tcp 4 9 tcp 4 udp 4 8810 tcp 4 8809 tcp 4 8808 tcp 4 8807 tcp 4 8806 tcp 4 8805 tcp 4 8804 tcp 4 8803 tcp 4 8802 tcp 4 8801 tcp 4 8686 tcp 4 8554 tcp 4 8145 tcp 4 8085 tcp 4 8010 tcp 4 8009 tcp 4 8008 tcp 4 8007 tcp 4 8006 tcp 4 8005 tcp 4 8004 tcp 4 8003 tcp 4 8002 tcp 4 8 tcp 4 7778 tcp 4 7443 tcp 4 7005 tcp 4 6443 tcp 4 6080 udp 4 6050 udp 4 6022 tcp 4 60022 tcp 4 6001 tcp 4 587 tcp 4 55313 tcp 4 5443 tcp 4 51132 tcp 4 5099 udp 4 5098 udp 4 5090 udp 4 5080 udp 4 5075 udp 4 5038 tcp 4 5022 tcp 4 5010 tcp 4 5009 tcp 4 5008 tcp 4 5006 tcp 4 5005 tcp 4 5004 tcp 4 5003 tcp 4 5002 tcp 4 5001 tcp 4 46536 tcp 4 44818 tcp 4 tcp 4 4443 tcp 4 4022 tcp 4 4 tcp 4 37191 tcp 4 3493 tcp 4 3264 tcp 4 3263 tcp 4 3262 tcp 4 3261 tcp 4 31337 tcp 4 3000 tcp 4 2701 tcp 4 25557 tcp 4 2443 tcp 4 22322 tcp 4 udp 4 2049 tcp 4 20288 tcp 4 2001 tcp 4 1755 tcp 4 17185 udp 4 15 tcp 4 1443 tcp 4 14226 tcp 4 14 tcp 4 13282 tcp 4 13281 tcp 4 13272 tcp 4 13 tcp 4 1283 tcp 4 12 tcp 4 1122 tcp 4 10 tcp 3 993 tcp 3 9151 tcp 3 82 tcp 3 64738 udp 3 500 udp 3 4500 udp 3 3780 tcp 3 3460 tcp 3 2480 tcp 3 2152 udp 3 21025 tcp 3 20547 tcp 3 19 tcp 3 1604 udp 3 1010 tcp 2 9798 tcp 2 8989 tcp 2 8834 tcp 2 88 udp 2 873 tcp 2 83 tcp 2 8060 tcp 2 7548 tcp 2 69 udp 2 6664 tcp 2 64436 tcp 2 63184 tcp 2 62484 tcp 2 6243 tcp 2 61049 tcp 2 60607 tcp 2 60333 tcp 2 59806 tcp 2 59395 tcp 2 57490 tcp 2 57358 tcp 2 5632 udp 2 56067 tcp 2 55650 tcp 2 5560 tcp 2 55107 tcp 2 5364 tcp 2 52072 tcp 2 51546 tcp 2 51483 tcp 2 5148 tcp 2 51065 tcp 2 50787 tcp 2 50009 tcp 2 4911 tcp 2 45925 tcp 2 44877 tcp 2 43501 tcp 2 4343 tcp 2 43192 tcp 2 42741 tcp 2 404
Re: Adding zombies to a pf table?
Hi Ted, On 2015-09-23 Wed 13:51 PM |, Ted Unangst wrote: > > > > Zombies are often attacking ports which don't have services running, > > such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. > > > > block log those ports, then process the log file? > Running tcpdump was my first thought too, via an rc.d started script, but I wasn't too keen on having that running all the time. Ta. -- An elephant is a mouse with an operating system.
Re: Adding zombies to a pf table?
Hi Pantelis, On 2015-09-24 Thu 12:37 PM |, Pantelis Roditis wrote: > > This is the exact reason why we created bofh-divert[1]. The idea is that you > pass those packets with PF to a divert socket opened by a daemon. The daemon > grabs the source IP and adds it to a predefined table. > Wow, that looks like the ticket. If nothing else, I was considering a fake inetd driven telnet daemon, which would just be a script to drive netcat, grab the remote ip & pfctl add it to a table. With pf re-directs to it for commonly attacked ports, finishing up with: block in log from Cheers. -- The only possible interpretation of any research whatever in the `social sciences' is: some do, some don't. -- Ernest Rutherford
Re: Adding zombies to a pf table?
On 2015-09-24 Thu 14:42 PM |, Benny Lofgren wrote: > > I've used one of the inetd "trivial services" (echo, discard, chargen, > daytime or time) for this purpose, in combination with a couple of PF > rules. Something like this: > > match in log on egress from any to tag honeypot > pass in log tagged honeypot rdr-to 127.0.0.1 port echo keep state \ > (max-src-conn-rate 1/30, overload flush global) > Ahhh! Cunning plan Benny. I shall play... > > PS. Who named unlistened-to ports "zombies" anyway? http://en.wikipedia.org/wiki/Zombie_computer Cool. -- It is only the great men who are truly obscene. If they had not dared to be obscene, they could never have dared to be great. -- Havelock Ellis
Adding zombies to a pf table?
Hello, Zombies are often attacking ports which don't have services running, such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. With a default pf block drop in on $ext_if, how can those source ips be added to a table? Which all can be dropped & small queued. I've tried to overload a match statement, but that won't work. Or is there something handy in ports to help? Thanks. -- By the time they had diminished from 50 to 8, the other dwarves began to suspect "Hungry" ... -- Gary Larson, "The Far Side"
Re: Adding zombies to a pf table?
Craig Skinner wrote: > Hello, > > Zombies are often attacking ports which don't have services running, > such as telnet (most popular indeed), mysql, 3551, 8080, 13272, etc. > > With a default pf block drop in on $ext_if, how can those source ips be > added to a table? Which all can be dropped & small queued. > > I've tried to overload a match statement, but that won't work. > > Or is there something handy in ports to help? block log those ports, then process the log file? block quick from block quick log in to port 8080 then you won't see them showing up in the log over and over.
zombies
How are zombies best dealt with, correctively? My OBSD 4.2 x86 machine is showing memory and CPU utilization are a negligable fraction of the total capacity. Yet, it is getting maxed out in regards to number of processes, apparently due to the zombies. kill -KILL seems to have no effect. Some interaction between Apache2 and perl is creating zombies. After several months, this number has crept up to close to a thousand and with kern.maxproc=1024, problems are starting For example, ps says : ... _apache297 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 19083 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 24147 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 30821 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 6995 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 26059 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 31087 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) ... So again, what corrective measures can be taken to rid the machine of zombie processes? And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 Regards, -Lars
Re: zombies - solved
Thanks. Paul de Weerd wrote: ... Zombies are part of unix, you *need* them in cases. Leaving them dangling (for too long) is not good of course, clean-up is required. That's what's happening. I see that one work-around would be to have cron periodically send a kill signal to the parent. But it pains me to even mention such a lame and problematic hack. ...This is the job of the parent process so the 'generic way' to *solve* these issues is by fixing the parent process. Easier said than done given the original state of my concentration / coding skills and the subsequent deterioration from that state. ... You may want to investigate alternative options or fix the code if you can... Where is it that the problem most likely lies? Apache2, perl or the heinous 'apt-cacher' script called by Apache2? /usr/bin/perl /usr/sbin/apt-cacher -d -p /var/run/apt-cacher.pid Looking ahead, what is the timeline for moving to Apache2? Or what are the major reasons 4.3 is going to still use 1.3x? Regards, -Lars
Re: zombies - solved
On Wed, Mar 12, 2008 at 2:18 AM, Lars Noodin [EMAIL PROTECTED] wrote: Or what are the major reasons 4.3 is going to still use 1.3x? Licensing.
Re: zombies
On Wed, Mar 12, 2008 at 10:36:23AM +0200, Lars Nood??n wrote: How are zombies best dealt with, correctively? My OBSD 4.2 x86 machine is showing memory and CPU utilization are a negligable fraction of the total capacity. Yet, it is getting maxed out in regards to number of processes, apparently due to the zombies. kill -KILL seems to have no effect. zombie processes are already dead, you cannot kill them. Some interaction between Apache2 and perl is creating zombies. After several months, this number has crept up to close to a thousand and with kern.maxproc=1024, problems are starting For example, ps says : ... _apache297 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 19083 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 24147 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 30821 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 6995 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 26059 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) _apache2 31087 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) ... So again, what corrective measures can be taken to rid the machine of zombie processes? And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 zombie state happend if a child process exits, but its parent did not execute a wait(2) system call (or one if its alternatives) for the process (yet). So this seem a bug in the handling of CGIs. -Otto Regards, -Lars
Re: zombies - solved
Looking ahead, what is the timeline for moving to Apache2? Likely never, unless they decide to change their license. Or what are the major reasons 4.3 is going to still use 1.3x? apache2 is not free enough.
Re: zombies - half solved
How are zombies best dealt with, correctively? Sorry to answer my own question. The solution was to find the parent process and kill it. But the second question still stands, is there a generic way to prevent the formation of zombies? The cause in this specific case is a perl-based CGI script called by apache2. Regards, -Lars
Re: zombies
On Wed, Mar 12, 2008 at 10:36:23AM +0200, Lars Nood??n wrote: | How are zombies best dealt with, correctively? By fixing the bugs in the parent. | My OBSD 4.2 x86 machine is showing memory and CPU utilization are a | negligable fraction of the total capacity. Yet, it is getting maxed out | in regards to number of processes, apparently due to the zombies. Zombies don't consume any (or, nearly any) resources apart form the one pid. | kill -KILL seems to have no effect. Nope. Read up on 'em to find out why. Hint : you can't kill what's already dead. | Some interaction between Apache2 and perl is creating zombies. After | several months, this number has crept up to close to a thousand and with | kern.maxproc=1024, problems are starting | | For example, ps says : | ... | _apache297 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 19083 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 24147 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 30821 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 6995 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 26059 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | _apache2 31087 0.0 0.0 0 0 ?? Z - 0:00.00 (perl) | ... | | So again, what corrective measures can be taken to rid the machine of | zombie processes? Find the parent process and restart it. This process is buggy. You may want to investigate alternative options or fix the code if you can. | And, is there a generic way to prevent them? The cause is a perl CGI | called by apache2 Zombies are part of unix, you *need* them in cases. Leaving them dangling (for too long) is not good of course, clean-up is required. This is the job of the parent process so the 'generic way' to *solve* these issues is by fixing the parent process. Cheers, Paul 'WEiRD' de Weerd -- [++-]+++.+++[---].+++[+ +++-].++[-]+.--.[-] http://www.weirdnet.nl/
Re: zombies
On 12 March 2008, Lars NoodC)n [EMAIL PROTECTED] wrote: [...] And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 Depending on what you're doing, make the parent wait(2) for the processes or setsid(3). Regards, Liviu Daia -- Dr. Liviu Daia http://www.imar.ro/~daia
Re: zombies - solved
* Theo de Raadt [EMAIL PROTECTED] [2008-03-12 10:36]: Looking ahead, what is the timeline for moving to Apache2? Likely never, unless they decide to change their license. even then... I don't see any advatages in apache2, but lots of disadvantages and a gigantic design fault. No, not one, multiple. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg Amsterdam
Re: zombies
Hi! On Wed, Mar 12, 2008 at 12:05:29PM +0200, Liviu Daia wrote: On 12 March 2008, Lars NoodC)n [EMAIL PROTECTED] wrote: [...] And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 Depending on what you're doing, make the parent wait(2) for the processes or setsid(3). setsid(2) (yes, it's section 2 on OpenBSD) doesn't make the child lose the connection to the parent. See the source of daemon(3) for how to use setsid in connection with fork and exit (in fact _exit) to make a process disconnect from its parent and its controlling terminal etc. Kind regards, Hannah.
Re: zombies
On 12 March 2008, Hannah Schroeter [EMAIL PROTECTED] wrote: Hi! On Wed, Mar 12, 2008 at 12:05:29PM +0200, Liviu Daia wrote: On 12 March 2008, Lars NoodC)n [EMAIL PROTECTED] wrote: [...] And, is there a generic way to prevent them? The cause is a perl CGI called by apache2 Depending on what you're doing, make the parent wait(2) for the processes or setsid(3). setsid(2) (yes, it's section 2 on OpenBSD) Yes, sorry. doesn't make the child lose the connection to the parent. No, it actually makes the calling process a session leader. See the source of daemon(3) for how to use setsid in connection with fork and exit (in fact _exit) to make a process disconnect from its parent and its controlling terminal etc. Actually, there's a bunch of other things to take care of, like signals and pipes. A more complete answer would be something like: read a book about UNIX process management; I was trying to provide a hint in the right direction, not abstract a book in a sentence. :) Regards, Liviu Daia -- Dr. Liviu Daia http://www.imar.ro/~daia
Re: zombies - solved
On 3/12/08, Lars NoodC)n [EMAIL PROTECTED] wrote: Looking ahead, what is the timeline for moving to Apache2? Or what are the major reasons 4.3 is going to still use 1.3x? Take a look at http://nginx.net/ BSD license, seems to work, but I don't know about its security profile. I'm sure it's not as secure as the OBSD Apache, but it might be ok compared with apache2.
Re: zombies
Otto == Otto Moerbeek [EMAIL PROTECTED] writes: Otto zombie state happend if a child process exits, but its parent did not Otto execute a wait(2) system call (or one if its alternatives) for the Otto process (yet). So this seem a bug in the handling of CGIs. Most likely a bug in a Perl script that forks but doesn't wait for its kid. I generally *don't* see zombies in well-written Perl programs. Was this FastCGI by any chance? I know there's unique problems related to that for naive code that creates a child, because the parent never goes away (since it's shared by the next series of CGI hits). But again, with proper care, even a FastCGI script can be written properly. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: zombies - solved
On Wed, Mar 12, 2008 at 08:39:07AM -0500, Gregg Reynolds wrote: On 3/12/08, Lars NoodC)n [EMAIL PROTECTED] wrote: Looking ahead, what is the timeline for moving to Apache2? Or what are the major reasons 4.3 is going to still use 1.3x? Take a look at http://nginx.net/ BSD license, seems to work, but I don't know about its security profile. I'm sure it's not as secure as the OBSD Apache, but it might be ok compared with apache2. There's also a port of nginx as of 4.2-current. THe port is of the stable version, not the development version. -- Darrin Chandler| Phoenix BSD User Group | MetaBUG [EMAIL PROTECTED] | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation
Re: zombies
On Wed, Mar 12, 2008 at 09:57:16AM +0100, Otto Moerbeek wrote: zombie state happend if a child process exits, but its parent did not execute a wait(2) system call (or one if its alternatives) for the process (yet). So this seem a bug in the handling of CGIs. I'd like to add a bit to the above and to Paul de Weerd's comments: Zombie processes are there to maintain a little info in case the parent process calls wait() later to retrieve it. Some program designs catch SIGCHILD or have a thread block on wait*, and in those cases the zombie lasts such a short time you'll probably never see it in top or ps. Other designs use non-blocking forms and zombies may stick around long enough to notice, but then disappear later when the parent makes a pass. If the parent dies before calling wait, then the zombie is inherited by init which will take care of it. So, zombies happen, but the only time they stay around for a long time is a negligent/misdesigned parent that is still alive but not calling wait* on the children. The OS can't make a badly written program into a well written program. So as admin you are stuck restarting the parent periodically, switching to something else, or bugging the developers to fix the problem. -- Darrin Chandler| Phoenix BSD User Group | MetaBUG [EMAIL PROTECTED] | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation
Re: zombies - solved
Theo de Raadt wrote: apache2 is not free enough. Ok. There were some additional reasons mentioned, but licensing is enough on its own. I found the old announcement now that I know what to look for: http://archives.neohapsis.com/archives/openbsd/2004-06/0448.html Apache 1.3.29 is decent enough and has the functionality, name brand recognition and familiarity needed. But without updates, it seems a dead end and not a good idea for new activities. I'm also not finding reference to IPv6 in the documentation for Apache 1.3.x either online or in the man pages and that was my main reason for even looking at Apache2. A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. Gregg proposed, nginx ( http://nginx.net/ ), which seems to be just getting started. It's under a 'BSD-like' license. It might work, but seems new. I see Lighttpd already in the 'packages' and it is under an appropriate license. In the last year, it has gained a lot in both visibility and user-base. In a lot of cases, perhaps most, new setups could be steered towards Lighttpd, if it were mentioned in the documentation here and there. I probably would have chosen it over grabbing Apache2 from the ports tree had it been mentioned. Apache2 and Lighttpd both required some adjustment and I would rather future-proof my activities, just in case they have to be supported that long. The mention of it can be small and does not need to affect how things are currently done. But as more use it, it will be easier later to drop Apache when (if) the time comes. Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. Regards, -Lars
Re: zombies
Randal L. Schwartz wrote: Most likely a bug in a Perl script that forks but doesn't wait for its kid. I generally *don't* see zombies in well-written Perl programs. ;) Was this FastCGI by any chance? No. I think it's the perl script, but now that gets added to my list of things to do. The hints about setsid(2) and wait(2) give an idea of what to look for. regards, -Lars
Re: zombies - solved
Ok. There were some additional reasons mentioned, but licensing is enough on its own. I found the old announcement now that I know what to look for: http://archives.neohapsis.com/archives/openbsd/2004-06/0448.html Apache 1.3.29 is decent enough and has the functionality, name brand recognition and familiarity needed. But without updates, it seems a dead end and not a good idea for new activities. That is 1 persons opinion, and I think you will find yourself isolated. It's just a bloody web server. It's easy. I'm also not finding reference to IPv6 in the documentation for Apache 1.3.x either online or in the man pages and that was my main reason for even looking at Apache2. There are diffs coming that add v6 support. There have been reasons not to add it in the past. A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. When we started work on OpenSSH, there were people just like you saying that it did not seem like a good return on investment. Investment. Who are you to tell us how we should spend our time, and what we should do? If you don't LIKE IT, then do whatever you want. Gregg proposed, nginx ( http://nginx.net/ ), which seems to be just getting started. It's under a 'BSD-like' license. It might work, but seems new. Huh? We've already GOT a completely working fixed one in our tree. It's fine. And we have zero interest in swapping to some other piece of shit when this piece of shit will do. Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. No.
Re: zombies - solved
If you want to serve http content via IPv6, then perhaps you can run httpd on your (IPv4) loopback interface, and have relayd listen on your public IPv6 interface, and forward requests over IPv4 to it ? /Pete On 12 Mar 2008, at 4:22 PM, Lars Noodin wrote: Theo de Raadt wrote: apache2 is not free enough. Ok. There were some additional reasons mentioned, but licensing is enough on its own. I found the old announcement now that I know what to look for: http://archives.neohapsis.com/archives/openbsd/2004-06/0448.html Apache 1.3.29 is decent enough and has the functionality, name brand recognition and familiarity needed. But without updates, it seems a dead end and not a good idea for new activities. I'm also not finding reference to IPv6 in the documentation for Apache 1.3.x either online or in the man pages and that was my main reason for even looking at Apache2. A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. Gregg proposed, nginx ( http://nginx.net/ ), which seems to be just getting started. It's under a 'BSD-like' license. It might work, but seems new. I see Lighttpd already in the 'packages' and it is under an appropriate license. In the last year, it has gained a lot in both visibility and user-base. In a lot of cases, perhaps most, new setups could be steered towards Lighttpd, if it were mentioned in the documentation here and there. I probably would have chosen it over grabbing Apache2 from the ports tree had it been mentioned. Apache2 and Lighttpd both required some adjustment and I would rather future-proof my activities, just in case they have to be supported that long. The mention of it can be small and does not need to affect how things are currently done. But as more use it, it will be easier later to drop Apache when (if) the time comes. Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. Regards, -Lars
Re: zombies - solved
Op Wed, 12 Mar 2008 17:05:01 +0100 schreef Pete Vickers [EMAIL PROTECTED]: If you want to serve http content via IPv6, then perhaps you can run httpd on your (IPv4) loopback interface, and have relayd listen on your public IPv6 interface, and forward requests over IPv4 to it ? And then what if the HTTP request reads something like GET [::1] ? -- Boudewijn Dijkstra Indes - IDS B.V. +31 345 545 535
Re: zombies - solved
Lars NoodC)n wrote: Would something like this be appropriate at the tail end of the httpd man page for v 1.3.29? Due to licensing changes, the version of Apache shipped with OpenBSD will stay at version 1.3.29. Bugfixes will be provided, but no further updates. Alternatively, Lighttpd is available via OpenBSD's packages. Why do some people think Apache needs to be replaced? Moreover, if the developers are satisfied with Apache 1.3, why would the recommend another product in the documentation?
Re: zombies - solved
On Wed, Mar 12, 2008 at 11:58 AM, Theo de Raadt [EMAIL PROTECTED] wrote: A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. I'm just curious what is in 2.x that you need, that is unavailable in 1.3? When we started work on OpenSSH, there were people just like you saying that it did not seem like a good return on investment. Investment. Who are you to tell us how we should spend our time, and what we should do? If you don't LIKE IT, then do whatever you want. Well, obviously we want an upgrade to Apache 2, and an upgrade to Apache 3 when that comes out. If only you are not so selfish as to go on mountain climbing hikes, and satay eating binges, then you'll definitely have time to invest in upgrading to Apache v3! :) -- http://www.glumbert.com/media/shift http://www.youtube.com/watch?v=tGvHNNOLnCk This officer's men seem to follow him merely out of idle curiosity. -- Sandhurst officer cadet evaluation. Securing an environment of Windows platforms from abuse - external or internal - is akin to trying to install sprinklers in a fireworks factory where smoking on the job is permitted. -- Gene Spafford learn french: http://www.youtube.com/watch?v=j1G-3laJJP0feature=related
Re: zombies - solved
On 2008-03-12, Pete Vickers [EMAIL PROTECTED] wrote: If you want to serve http content via IPv6, then perhaps you can run httpd on your (IPv4) loopback interface, and have relayd listen on your public IPv6 interface, and forward requests over IPv4 to it ? Here's a better way: test the diffs at http://mini.vnode.ch/ and provide feedback.
Re: zombies - solved
bofh wrote: On Wed, Mar 12, 2008 at 11:58 AM, Theo de Raadt [EMAIL PROTECTED] wrote: A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. I'm just curious what is in 2.x that you need, that is unavailable in 1.3? mod_proxy_balancer Jonathan -- Jonathan Weiss http://blog.innerewut.de
Re: zombies - solved
Quoting Jonathan Weiss [EMAIL PROTECTED]: bofh wrote: On Wed, Mar 12, 2008 at 11:58 AM, Theo de Raadt [EMAIL PROTECTED] wrote: A fork does not seem like a good return on investment, so v 1.3.29 will probably go away sooner than later once the Apache Foundation drops maintenance on the 1.3 series. I'm just curious what is in 2.x that you need, that is unavailable in 1.3? mod_proxy_balancer Ok, you have a need for Apache 2.x. That does not mean that the Apache server in the base install needs to be updated. http://www.openbsd.org/cgi-bin/cvsweb/ports/www/apache-httpd/ -- Tim Donahue This message was sent using IMP, the Internet Messaging Program.
IPv6 web servers (was Re: zombies - solved)
Markus Lude wrote: mbalmer@ posted a diff for IPv6 support for the base apache back last december: see http://mini.vnode.ch/ Excellent. What, in general, are the plans? (Any answer is fine.) Knowing more reduces the unnecessary questions, experiments and speculations that get in the way. My interest in this is having an IPv6 web server on OpenBSD so I can try an in-house IPv6 pilot sometime this year. The timeline for that is probably May or August. In general, it's preferable for me to follow -STABLE because the less I have to tweak the less likely I am to break something and the less custom documentation I need to provide to pass it on to others. I plan to rebuild for OBSD 4.3 in May, I'll try the IPv6 patch (or however it happens to be available) for 1.3 then. Going back to 1.3 will actually save me work and it's what I'm most familiar with already. Apache 2.2.x is in ports if you really need it. Some ports have an -ap2 flavor for that newer version. That's what I grabbed last autumn, but aside from IPv6 there's nothing needed that was/is specific to Apache 2. regards, -Lars
Re: zombies - solved
On Wed, Mar 12, 2008 at 12:19:18PM -0400, bofh wrote: | A fork does not seem like a good return on investment, so v 1.3.29 will | probably go away sooner than later once the Apache Foundation drops | maintenance on the 1.3 series. | | | I'm just curious what is in 2.x that you need, that is unavailable in 1.3? The only reason I run Apache 2 on my OpenBSD machine is IPv6. There's patches for 1.3, but for now we chose Apache 2. I know there's people working on integrating the v6 patches in OpenBSD and I hope those make it for 4.4, but we'll see how it goes. Paul 'WEiRD' de Weerd -- [++-]+++.+++[---].+++[+ +++-].++[-]+.--.[-] http://www.weirdnet.nl/
Re: zombies - half solved
Lars wrote:
But the second question still stands, is there a generic way to prevent
the formation of zombies? The cause in this specific case is a
perl-based CGI script called by apache2.
The easiest way might be to let perl auto-reap the children for you.
It's as simple as prepending this line within the block that spawns the
child processes:
local $SIG{CHLD} = 'IGNORE'; # straight outa perlipc(1)
One thing to watch out for though is that you may get weird side effects
if you set this and then use system() in the same scope. If so, just
use wait/waitpid instead (it's only a few more lines of code).
--
Stephen Takacs [EMAIL PROTECTED] http://perlguru.net/
4149 FD56 D078 C988 9027 1EB4 04CC F80F 72CB 09DA
