Re: [spamdyke-users] My logfile parser (Script)
you could also use fail2ban for that. You just have to specify a custom rule (filter) for the spamdyke-log output. Then the sender ip will be released after a specified timeframe and not blocked forever ;). (IMHO it is still not a very good idea to block by firewall) Otto Sebastian Grewe schrieb: Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with tail -f .. ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with tail. Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
I totally forgot about that - but I am not using the script to block them forever, just to monitor qmail when a large amount of connections is coming in (which happens ever so often). Even so I did turn off the blocking feature since qmail handles it just fine and connections clear up after a while. I was just concerned that legitimate e-mail wouldn't be coming through - but since they try to resend if no connection could be established that's not a concern anymore. So yeah, I use it to see what's being blocked and for what reason - even added whitelist matches now. It's basically just colored and filtered output of your qmail logfiles now :D Cheers, Sebastian Otto Berger wrote: you could also use fail2ban for that. You just have to specify a custom rule (filter) for the spamdyke-log output. Then the sender ip will be released after a specified timeframe and not blocked forever ;). (IMHO it is still not a very good idea to block by firewall) Otto Sebastian Grewe schrieb: Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with tail -f .. ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with tail. Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
Sorry to say that I haven't had a chance to check out your script yet, Sebastian. :( Speaking of colored and filtered qmail logfiles though, there's a nice 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus package). It allows easy viewing and searching of qmail (et al) logs. I'm wondering if your 'coloring and filtering' might be a nice enhancement to that script. Care to have a look into it? Sebastian Grewe wrote: I totally forgot about that - but I am not using the script to block them forever, just to monitor qmail when a large amount of connections is coming in (which happens ever so often). Even so I did turn off the blocking feature since qmail handles it just fine and connections clear up after a while. I was just concerned that legitimate e-mail wouldn't be coming through - but since they try to resend if no connection could be established that's not a concern anymore. So yeah, I use it to see what's being blocked and for what reason - even added whitelist matches now. It's basically just colored and filtered output of your qmail logfiles now :D Cheers, Sebastian Otto Berger wrote: you could also use fail2ban for that. You just have to specify a custom rule (filter) for the spamdyke-log output. Then the sender ip will be released after a specified timeframe and not blocked forever ;). (IMHO it is still not a very good idea to block by firewall) Otto Sebastian Grewe schrieb: Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with tail -f .. ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with tail. Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
Hey Eric, As I undestand it qmlog is just a tool to find a specific logfile entry if you are looking for certain times where a connection has been made. My script is just checking for spamdyke output, and only specific output at that. I also am using a while loop to read the lines in instead of just tail so I can process them. I will have a quick look and see if I am able to add it - usually I just write stuff - not used to changing other peoples code :P Cheers, Sebastian Eric Shubert wrote: Sorry to say that I haven't had a chance to check out your script yet, Sebastian. :( Speaking of colored and filtered qmail logfiles though, there's a nice 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus package). It allows easy viewing and searching of qmail (et al) logs. I'm wondering if your 'coloring and filtering' might be a nice enhancement to that script. Care to have a look into it? Sebastian Grewe wrote: I totally forgot about that - but I am not using the script to block them forever, just to monitor qmail when a large amount of connections is coming in (which happens ever so often). Even so I did turn off the blocking feature since qmail handles it just fine and connections clear up after a while. I was just concerned that legitimate e-mail wouldn't be coming through - but since they try to resend if no connection could be established that's not a concern anymore. So yeah, I use it to see what's being blocked and for what reason - even added whitelist matches now. It's basically just colored and filtered output of your qmail logfiles now :D Cheers, Sebastian Otto Berger wrote: you could also use fail2ban for that. You just have to specify a custom rule (filter) for the spamdyke-log output. Then the sender ip will be released after a specified timeframe and not blocked forever ;). (IMHO it is still not a very good idea to block by firewall) Otto Sebastian Grewe schrieb: Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with tail -f .. ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with tail. Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
After checking out the code in that script I think it might be easier for me to just start on my script and extend it's functionality to look for all lines in those logfiles instead of just spamdyke. I will see what I can do. Cheers, Sebastian Eric Shubert wrote: Sorry to say that I haven't had a chance to check out your script yet, Sebastian. :( Speaking of colored and filtered qmail logfiles though, there's a nice 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus package). It allows easy viewing and searching of qmail (et al) logs. I'm wondering if your 'coloring and filtering' might be a nice enhancement to that script. Care to have a look into it? Sebastian Grewe wrote: I totally forgot about that - but I am not using the script to block them forever, just to monitor qmail when a large amount of connections is coming in (which happens ever so often). Even so I did turn off the blocking feature since qmail handles it just fine and connections clear up after a while. I was just concerned that legitimate e-mail wouldn't be coming through - but since they try to resend if no connection could be established that's not a concern anymore. So yeah, I use it to see what's being blocked and for what reason - even added whitelist matches now. It's basically just colored and filtered output of your qmail logfiles now :D Cheers, Sebastian Otto Berger wrote: you could also use fail2ban for that. You just have to specify a custom rule (filter) for the spamdyke-log output. Then the sender ip will be released after a specified timeframe and not blocked forever ;). (IMHO it is still not a very good idea to block by firewall) Otto Sebastian Grewe schrieb: Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with tail -f .. ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with tail. Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
Thanks. I'm sure you'll keep us posted! :) Sebastian Grewe wrote: After checking out the code in that script I think it might be easier for me to just start on my script and extend it's functionality to look for all lines in those logfiles instead of just spamdyke. I will see what I can do. Cheers, Sebastian Eric Shubert wrote: Sorry to say that I haven't had a chance to check out your script yet, Sebastian. :( Speaking of colored and filtered qmail logfiles though, there's a nice 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus package). It allows easy viewing and searching of qmail (et al) logs. I'm wondering if your 'coloring and filtering' might be a nice enhancement to that script. Care to have a look into it? Sebastian Grewe wrote: I totally forgot about that - but I am not using the script to block them forever, just to monitor qmail when a large amount of connections is coming in (which happens ever so often). Even so I did turn off the blocking feature since qmail handles it just fine and connections clear up after a while. I was just concerned that legitimate e-mail wouldn't be coming through - but since they try to resend if no connection could be established that's not a concern anymore. So yeah, I use it to see what's being blocked and for what reason - even added whitelist matches now. It's basically just colored and filtered output of your qmail logfiles now :D Cheers, Sebastian Otto Berger wrote: you could also use fail2ban for that. You just have to specify a custom rule (filter) for the spamdyke-log output. Then the sender ip will be released after a specified timeframe and not blocked forever ;). (IMHO it is still not a very good idea to block by firewall) Otto Sebastian Grewe schrieb: Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with tail -f .. ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with tail. Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
Sure will, so here we go. Attached a modified qmlog script that can be run with the -c option to add colored output for most log entries in Qmail Toaster. If something is missing or doesn't match correctly it will have the FIXME tag before the line. When running in color mode less output will be disabled. Thanks for pointing the script out, will use that one now since I added the colors to it :D Maybe even recreate the output and remove the log file look with a tabled output. It's written on Bash 3.0.15 again, but with minor changes to the filters it can be easily ported to higher versions of Bash too (tried it out, but didn't finish the rewrite since I am running only 3.0.15 servers). Try it out on your system and let me know what you think! Cheers, Sebastian PS: Yay to Fridays! Eric Shubert wrote: Thanks. I'm sure you'll keep us posted! :) Sebastian Grewe wrote: After checking out the code in that script I think it might be easier for me to just start on my script and extend it's functionality to look for all lines in those logfiles instead of just spamdyke. I will see what I can do. Cheers, Sebastian Eric Shubert wrote: Sorry to say that I haven't had a chance to check out your script yet, Sebastian. :( Speaking of colored and filtered qmail logfiles though, there's a nice 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus package). It allows easy viewing and searching of qmail (et al) logs. I'm wondering if your 'coloring and filtering' might be a nice enhancement to that script. Care to have a look into it? Sebastian Grewe wrote: I totally forgot about that - but I am not using the script to block them forever, just to monitor qmail when a large amount of connections is coming in (which happens ever so often). Even so I did turn off the blocking feature since qmail handles it just fine and connections clear up after a while. I was just concerned that legitimate e-mail wouldn't be coming through - but since they try to resend if no connection could be established that's not a concern anymore. So yeah, I use it to see what's being blocked and for what reason - even added whitelist matches now. It's basically just colored and filtered output of your qmail logfiles now :D Cheers, Sebastian Otto Berger wrote: you could also use fail2ban for that. You just have to specify a custom rule (filter) for the spamdyke-log output. Then the sender ip will be released after a specified timeframe and not blocked forever ;). (IMHO it is still not a very good idea to block by firewall) Otto Sebastian Grewe schrieb: Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with tail -f .. ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with tail. Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users #!/bin/bash # # Copyright (C) 2006-2008 Eric Shubert e...@shubes.net # # Utility for listing/searching qmail log files # Original script by Fabio Olaechea # # Future Enhancements # .) find .sed file w/out hard coded path # # # Change Log # 02/27/09 sebastian - added colored output function # 04/05/08 shubes - changed `` to $() # 10/17/07 shubes - fixed -t option # 12/17/06 shubes - added sed, grep, date/time parameters # 11/24/06 shubes - restructured, added numerous capabilities # 11/21/06 shubes - added -f option, thanks to phi...@ows.ch
