Re: shellscript as policy-service -- zombie/load
Hello,
Andre Hübner:
Hello,
for testing purposes i wrote a policy-service for postfix as a
shellscript.
My Script is working very well, iam happy with its functionality ;)
But unfortunately there is one problem when a lot of mails are incoming.
the
shellscript just does some grepping in small files etc. and is giving
back
a allowd result..
My Shellscript is spawned from master.cf like this:
policy-mycheck unix - n n - - spawn
user=nobody argv=nice -n 15 /usr/lib/postfix/mycheckscript.sh
When a lot of mails are incoming i got a high number of zombies. as a
consequence of this my system load gets really high.
Are there some general methods to avoid this?
Find out what is the parent process of the zombies. This parent
process is not cleaning up as it should.
Wietse
Unfortunately i cant bring it forward. i have no explanation for this.
Is it allowed to post some codelines? I will do it as short as possible,
showing just most significant lines.
I know, a bashscript as policy-service is not the best solution, but for my
purposes it should be enough.
May be the experts are seeing something and can tell a mistake i did.
This are my example scriptlines:
#!/bin/bash
while read line
do
# get some data from postfix input, just like this i do it for different
params like sasl_username, client_address, recipient (6 params att all)
sender_gefunden=`echo $line | tr -d '\n' | grep -i -v sasl_sender |
grep -i -c sender=`
if [ 1 -eq $sender_gefunden ];then
sender_domain=`echo $line | tr -d '\n' | gawk -F '@' {'print $2'}`
sender_adresse=`echo $line | tr -d '\n' | gawk -F '=' {'print $2'}`
fi
#empty line, starting the interpretation
if [ -z $line ];then
# i do some tests if senderdomain is in content of a file and some
comparisons
senderdomain_is_localedomain=`cat /path/path/file | grep -i
$sender_domain | wc -l`
# after all i create one of 4 answers to postfix, which is sent bevor
done, but within the last if which checks that we are in last empty line of
postfix
postfixantwort=action=dunno\n\n
printf $postfixantwort
#if i do an exit 0 here postfix writes in log warning: premature
end-of-input on private/policy-sasl while reading input attribute name
fi
done
# even if i jump out of the loop after getting and result and sending this
result as last line of my script there were zombies created :(
#end script
In processstatus this leads to al lot of zombieprocesses while runing
mycheckscript.sh
14953 nobody20 0 000 Z0 0.0 0:00.00
mycheckscript.sh defunct
14956 nobody20 0 000 Z0 0.0 0:00.00
mycheckscript.sh defunct
14957 nobody20 0 000 Z0 0.0 0:00.00
mycheckscript.sh defunct
14958 nobody20 0 000 Z0 0.0 0:00.00 tr defunct
14960 nobody20 0 000 Z0 0.0 0:00.00
mycheckscript.sh defunct
14961 nobody20 0 000 Z0 0.0 0:00.00 tr defunct
14966 nobody20 0 000 Z0 0.0 0:00.00 tr defunct
14969 nobody20 0 000 Z0 0.0 0:00.00 grep defunct
script functions are working, postfix is doing what my service answers.
confusing...
i have the suspicion that problem is related to the using of pipes (|) at
grep, tr, etc. and therefore subshells which could be the zombies, but in
mosz cases i cant avoid them...
I know, this is not a special postfix issue but it would be nice if someone
would have an idea of what is going wrong na dhelsp me out of my misery ;)
Thanks,
Andre
Re: shellscript as policy-service -- zombie/load
Andre H?bner: [ Charset ISO-8859-1 unsupported, converting... ] Hello, Andre H?bner: Hello, for testing purposes i wrote a policy-service for postfix as a shellscript. My Script is working very well, iam happy with its functionality ;) But unfortunately there is one problem when a lot of mails are incoming. the shellscript just does some grepping in small files etc. and is giving back a allowd result.. My Shellscript is spawned from master.cf like this: policy-mycheck unix - n n - - spawn user=nobody argv=nice -n 15 /usr/lib/postfix/mycheckscript.sh When a lot of mails are incoming i got a high number of zombies. as a consequence of this my system load gets really high. Are there some general methods to avoid this? Find out what is the parent process of the zombies. This parent process is not cleaning up as it should. Wietse Unfortunately i cant bring it forward. i have no explanation for this. Is it allowed to post some codelines? I will do it as short as possible, Do not contradict the expert. You must find out the parent process of the zombies. In the output from ps axl commands, this is shown in the PPID column (use ps -el on Solaris and other SYS5-ish platforms). The parent process is not cleaning up as it should. Wietse
Re: shellscript as policy-service -- zombie/load
On Mon, Apr 27, 2009 at 09:16:31AM +0200, Andre H?bner wrote: When a lot of mails are incoming Symptom A i got a high number of zombies. Symptom B as a consequence of this By this do you mean symptom A or symptom B? my system load gets really high. Zombies are dead processes that demand zero CPU (just take up some space in the process table). So it seems unlikely that the zombies are causing high system load. Are there some general methods to avoid this? A parent process that forks children in a loop must either call wait to reap them promptly or IGNORE SIGCHLD. This is about zombies not load. A policy service process is expected to stay connected to an smtpd(8) client for the lifetime of that client, and handle multiple requests. Your load will be lower if your policy shell script processes requests in a loop and does not unilaterally disconnect from its smtpd(8) client. This is about load, not zombies. -- Viktor. Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. To unsubscribe from the postfix-users list, visit http://www.postfix.org/lists.html or click the link below: mailto:majord...@postfix.org?body=unsubscribe%20postfix-users If my response solves your problem, the best way to thank me is to not send an it worked, thanks follow-up. If you must respond, please put It worked, thanks in the Subject so I can delete these quickly.
Re: shellscript as policy-service -- zombie/load
Hello, Andre Hübner: Hello, for testing purposes i wrote a policy-service for postfix as a shellscript. My Script is working very well, iam happy with its functionality ;) But unfortunately there is one problem when a lot of mails are incoming. the shellscript just does some grepping in small files etc. and is giving back a allowd result.. My Shellscript is spawned from master.cf like this: policy-mycheck unix - n n - - spawn user=nobody argv=nice -n 15 /usr/lib/postfix/mycheckscript.sh When a lot of mails are incoming i got a high number of zombies. as a consequence of this my system load gets really high. Are there some general methods to avoid this? Find out what is the parent process of the zombies. This parent process is not cleaning up as it should. Wietse hmm, i know, it is not a postfix issue but i am afraid i need further help :( at the moment i do not have an idea how to debug this. after sending answer to postfix script is done and exits with 0, this works. script just does only formating input-data from postfix and grepping in files with basic shell commands and writing a linein maillog. i have no idea where the zombies came from... Is there a general way for debugging this? Thanks, Andre
Re: shellscript as policy-service -- zombie/load
Andre Hübner schrieb: Hello, Andre Hübner: Hello, for testing purposes i wrote a policy-service for postfix as a shellscript. My Script is working very well, iam happy with its functionality ;) But unfortunately there is one problem when a lot of mails are incoming. the shellscript just does some grepping in small files etc. and is giving back a allowd result.. My Shellscript is spawned from master.cf like this: policy-mycheck unix - n n - - spawn user=nobody argv=nice -n 15 /usr/lib/postfix/mycheckscript.sh When a lot of mails are incoming i got a high number of zombies. as a consequence of this my system load gets really high. Are there some general methods to avoid this? Find out what is the parent process of the zombies. This parent process is not cleaning up as it should. Wietse hmm, i know, it is not a postfix issue but i am afraid i need further help :( at the moment i do not have an idea how to debug this. after sending answer to postfix script is done and exits with 0, this works. script just does only formating input-data from postfix and grepping in files with basic shell commands and writing a linein maillog. i have no idea where the zombies came from... Is there a general way for debugging this? Thanks, Andre Hello Andre, what about forget this shell script (i dont think you will ever be lucky with it) and use fail2ban http://www.fail2ban.org to firewall the zombies for a configured time after all you should use all other allready implemented antispam features included in postfix ( google about it search in the list) a simple way may be i.e after you looked in log to fast reject dyn ip adresses i.e like this for smtpd_client stage smtpd_client_restrictions = ... permit_sasl_authenticated, permit_mynetworks, check_client_access hash:/etc/postfix/client_access, check_client_access pcre:/etc/postfix/dyn_spambotmap_client_access_pcre, reject_unknown_reverse_client_hostname, reject_rbl_client zen.spamhaus.org, reject_rbl_client combined.njabl.org, reject_rbl_client ix.dnsbl.manitu.net, /etc/postfix/dyn_spambotmap_client_access_pcre /c-.*hsd[0-9].*comcast.net/ REJECT Comcast worlds largest bot farm /cpe-.*res.rr.com/ REJECT Time Warner Road Runner cable spam bots /dsl.*\.ttnet.net.tr/ REJECT Turk Telekom spam bots /pool-.*verizon.net/ REJECT Verizon spam bots /.*dynamic\..*\.retail\.telecomitalia\.it/i REJECT SPAM_dyn_ip-add_networks /.*\.dyn-ip\.SPb\.SkyLink\.RU/i REJECT SPAM_dyn-ip-SPb-SkyLink /ppp.*\.home\.otenet\.gr/i REJECT clean your net from spam bots /ppp.*\.pppoe\.avangarddsl\.ru/i REJECT clean your net from spam bots /ppp.*\..*\.asianet\.co\.th/i REJECT clean your net from spam bots /dhcp-.*\.chello\.nl/i REJECT clean your net from spam bots /etc/postfix/client_access trafficmonkey.info REJECT Spam mailer ... 208.53.3.66 REJECT ... stopping Zombies is hard work and you will never win with all of them target should be beat them that way that you have no problem with your legal mail every domain has its own spam and zombies so analyse your logs you might try many combinations from antispam features fit to your needs there is no all around match kill them all i have a three letter domain which is bombed since years from zombies no feature got zombies out of the way, seems 3 letters are simply to easy to type in bot programming, but others reported bots stopping after a time by whatever speculated reason -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: shellscript as policy-service -- zombie/load
Andre H?bner: Hello, for testing purposes i wrote a policy-service for postfix as a shellscript. My Script is working very well, iam happy with its functionality ;) But unfortunately there is one problem when a lot of mails are incoming. the shellscript just does some grepping in small files etc. and is giving back a allowd result.. My Shellscript is spawned from master.cf like this: policy-mycheck unix - n n - - spawn user=nobody argv=nice -n 15 /usr/lib/postfix/mycheckscript.sh When a lot of mails are incoming i got a high number of zombies. as a consequence of this my system load gets really high. Are there some general methods to avoid this? Find out what is the parent process of the zombies. This parent process is not cleaning up as it should. Wietse
