Re: [spamdyke-users] My logfile parser (Script)
Sure will, so here we go.
Attached a modified qmlog script that can be run with the "-c" option
to add colored output for most log entries in Qmail Toaster. If something
is missing or doesn't match correctly it will have the FIXME tag before
the line.
When running in color mode less output will be disabled.
Thanks for pointing the script out, will use that one now since I added
the colors to it :D Maybe even recreate the output and remove the log
file look with a tabled output.
It's written on Bash 3.0.15 again, but with minor changes to the filters
it can be easily ported to higher versions of Bash too (tried it out, but
didn't finish the rewrite since I am running only 3.0.15 servers).
Try it out on your system and let me know what you think!
Cheers,
Sebastian
PS: Yay to Fridays!
Eric Shubert wrote:
Thanks. I'm sure you'll keep us posted! :)
Sebastian Grewe wrote:
After checking out the code in that script I think it might be easier
for me to just start on my script and extend it's functionality to look
for all lines in those logfiles instead of just spamdyke.
I will see what I can do.
Cheers,
Sebastian
Eric Shubert wrote:
Sorry to say that I haven't had a chance to check out your script yet,
Sebastian. :(
Speaking of colored and filtered qmail logfiles though, there's a nice
'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus
package). It allows easy viewing and searching of qmail (et al) logs.
I'm wondering if your 'coloring and filtering' might be a nice
enhancement to that script. Care to have a look into it?
Sebastian Grewe wrote:
I totally forgot about that - but I am not using the script to block
them forever, just to monitor qmail when a large amount of connections
is coming in (which happens ever so often). Even so I did turn off
the blocking feature since qmail handles it just fine and connections
clear up after a while. I was just concerned that legitimate e-mail
wouldn't
be coming through - but since they try to resend if no connection could
be established that's not a concern anymore.
So yeah, I use it to see what's being blocked and for what reason - even
added whitelist matches now.
It's basically just colored and filtered output of your qmail logfiles
now :D
Cheers,
Sebastian
Otto Berger wrote:
you could also use fail2ban for that. You just have to specify a custom
rule ("filter") for the spamdyke-log output. Then the sender ip will be
released after a specified timeframe and not blocked forever ;).
(IMHO it is still not a very good idea to block by firewall)
Otto
Sebastian Grewe schrieb:
Hey Guys,
I have been working on a simple bash script that will read from it's
standard input and presents some statistics from the logfile in realtime
(when used with "tail -f .." ).
After a few days that we have been attacked by spambots I got curious
how to avoid these things in the future. The script we use is able to
count the denied connections
per IP and, if desired, adds this IP to the Firewall to reject incoming
connections (brutal, I know). As the firewalling is optional you might
still be interested in it to run just
to see what's going on.
It's written for BASH 3.0.15 but with a little change in the pattern
matcher it runs on higher versions too. To start it in live mode run it
like this:
tail -f /var/log/qmail/smtp/current | qmail_parser.sh
and if you just want to scan some files and see what happened to this:
cat /var/log/qmail/smtp/* | qmail_parser.sh
Since it's BASH it's not very good when it comes to performance but does
the trick well when used with "tail". Also it's not catching everything
(yet) since I was looking for only
some very specific lines in the logfile. Anyhow, try it out and tell me
what you think - attached the current script to this mail.
Cheers,
Sebastian
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
#!/bin/bash
#
# Copyright (C) 2006-2008 Eric Shubert
#
# Utility for listing/searching qmail log files
# Original script by Fabio Olaechea
#
# Future Enhancements
# .) find .sed file w/out hard coded path
#
#
# Change Log
# 02/27/09 sebastian - added colored output function
# 04/05/08 shubes - changed `` to $()
# 10/17/07 shubes - fixed -t option
# 12/17/06 shubes - added sed, grep, date/time parameters
# 11/24/06 shubes - restructured, added numerous capabilities
# 11/21/06 shubes - added -f option, thanks to phi...@ows.ch
#
Re: [spamdyke-users] My logfile parser (Script)
Thanks. I'm sure you'll keep us posted! :)
Sebastian Grewe wrote:
> After checking out the code in that script I think it might be easier
> for me to just start on my script and extend it's functionality to look
> for all lines in those logfiles instead of just spamdyke.
>
> I will see what I can do.
>
> Cheers,
> Sebastian
>
> Eric Shubert wrote:
>> Sorry to say that I haven't had a chance to check out your script yet,
>> Sebastian. :(
>>
>> Speaking of colored and filtered qmail logfiles though, there's a nice
>> 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus
>> package). It allows easy viewing and searching of qmail (et al) logs.
>> I'm wondering if your 'coloring and filtering' might be a nice
>> enhancement to that script. Care to have a look into it?
>>
>> Sebastian Grewe wrote:
>>
>>> I totally forgot about that - but I am not using the script to block
>>> them forever, just to monitor qmail when a large amount of connections
>>> is coming in (which happens ever so often). Even so I did turn off
>>> the blocking feature since qmail handles it just fine and connections
>>> clear up after a while. I was just concerned that legitimate e-mail
>>> wouldn't
>>> be coming through - but since they try to resend if no connection could
>>> be established that's not a concern anymore.
>>>
>>> So yeah, I use it to see what's being blocked and for what reason - even
>>> added whitelist matches now.
>>>
>>> It's basically just colored and filtered output of your qmail logfiles
>>> now :D
>>>
>>> Cheers,
>>> Sebastian
>>>
>>> Otto Berger wrote:
>>>
you could also use fail2ban for that. You just have to specify a custom
rule ("filter") for the spamdyke-log output. Then the sender ip will be
released after a specified timeframe and not blocked forever ;).
(IMHO it is still not a very good idea to block by firewall)
Otto
Sebastian Grewe schrieb:
> Hey Guys,
>
> I have been working on a simple bash script that will read from it's
> standard input and presents some statistics from the logfile in realtime
> (when used with "tail -f .." ).
> After a few days that we have been attacked by spambots I got curious
> how to avoid these things in the future. The script we use is able to
> count the denied connections
> per IP and, if desired, adds this IP to the Firewall to reject incoming
> connections (brutal, I know). As the firewalling is optional you might
> still be interested in it to run just
> to see what's going on.
>
> It's written for BASH 3.0.15 but with a little change in the pattern
> matcher it runs on higher versions too. To start it in live mode run it
> like this:
>
> tail -f /var/log/qmail/smtp/current | qmail_parser.sh
>
> and if you just want to scan some files and see what happened to this:
>
> cat /var/log/qmail/smtp/* | qmail_parser.sh
>
> Since it's BASH it's not very good when it comes to performance but does
> the trick well when used with "tail". Also it's not catching everything
> (yet) since I was looking for only
> some very specific lines in the logfile. Anyhow, try it out and tell me
> what you think - attached the current script to this mail.
>
> Cheers,
> Sebastian
>
>
>
>
> ___
> spamdyke-users mailing list
> spamdyke-users@spamdyke.org
> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>
>
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>
>>
--
-Eric 'shubes'
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
After checking out the code in that script I think it might be easier
for me to just start on my script and extend it's functionality to look
for all lines in those logfiles instead of just spamdyke.
I will see what I can do.
Cheers,
Sebastian
Eric Shubert wrote:
> Sorry to say that I haven't had a chance to check out your script yet,
> Sebastian. :(
>
> Speaking of colored and filtered qmail logfiles though, there's a nice
> 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus
> package). It allows easy viewing and searching of qmail (et al) logs.
> I'm wondering if your 'coloring and filtering' might be a nice
> enhancement to that script. Care to have a look into it?
>
> Sebastian Grewe wrote:
>
>> I totally forgot about that - but I am not using the script to block
>> them forever, just to monitor qmail when a large amount of connections
>> is coming in (which happens ever so often). Even so I did turn off
>> the blocking feature since qmail handles it just fine and connections
>> clear up after a while. I was just concerned that legitimate e-mail
>> wouldn't
>> be coming through - but since they try to resend if no connection could
>> be established that's not a concern anymore.
>>
>> So yeah, I use it to see what's being blocked and for what reason - even
>> added whitelist matches now.
>>
>> It's basically just colored and filtered output of your qmail logfiles
>> now :D
>>
>> Cheers,
>> Sebastian
>>
>> Otto Berger wrote:
>>
>>> you could also use fail2ban for that. You just have to specify a custom
>>> rule ("filter") for the spamdyke-log output. Then the sender ip will be
>>> released after a specified timeframe and not blocked forever ;).
>>>
>>> (IMHO it is still not a very good idea to block by firewall)
>>>
>>> Otto
>>>
>>> Sebastian Grewe schrieb:
>>>
>>>
Hey Guys,
I have been working on a simple bash script that will read from it's
standard input and presents some statistics from the logfile in realtime
(when used with "tail -f .." ).
After a few days that we have been attacked by spambots I got curious
how to avoid these things in the future. The script we use is able to
count the denied connections
per IP and, if desired, adds this IP to the Firewall to reject incoming
connections (brutal, I know). As the firewalling is optional you might
still be interested in it to run just
to see what's going on.
It's written for BASH 3.0.15 but with a little change in the pattern
matcher it runs on higher versions too. To start it in live mode run it
like this:
tail -f /var/log/qmail/smtp/current | qmail_parser.sh
and if you just want to scan some files and see what happened to this:
cat /var/log/qmail/smtp/* | qmail_parser.sh
Since it's BASH it's not very good when it comes to performance but does
the trick well when used with "tail". Also it's not catching everything
(yet) since I was looking for only
some very specific lines in the logfile. Anyhow, try it out and tell me
what you think - attached the current script to this mail.
Cheers,
Sebastian
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>> ___
>>> spamdyke-users mailing list
>>> spamdyke-users@spamdyke.org
>>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>>
>>>
>
>
>
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
Hey Eric,
As I undestand it qmlog is just a tool to find a specific logfile entry if
you are looking for certain times where a connection has been made.
My script is just checking for spamdyke output, and only specific output
at that. I also am using a while loop to read the lines in instead of
just tail
so I can process them.
I will have a quick look and see if I am able to add it - usually I just
write stuff
- not used to changing other peoples code :P
Cheers,
Sebastian
Eric Shubert wrote:
> Sorry to say that I haven't had a chance to check out your script yet,
> Sebastian. :(
>
> Speaking of colored and filtered qmail logfiles though, there's a nice
> 'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus
> package). It allows easy viewing and searching of qmail (et al) logs.
> I'm wondering if your 'coloring and filtering' might be a nice
> enhancement to that script. Care to have a look into it?
>
> Sebastian Grewe wrote:
>
>> I totally forgot about that - but I am not using the script to block
>> them forever, just to monitor qmail when a large amount of connections
>> is coming in (which happens ever so often). Even so I did turn off
>> the blocking feature since qmail handles it just fine and connections
>> clear up after a while. I was just concerned that legitimate e-mail
>> wouldn't
>> be coming through - but since they try to resend if no connection could
>> be established that's not a concern anymore.
>>
>> So yeah, I use it to see what's being blocked and for what reason - even
>> added whitelist matches now.
>>
>> It's basically just colored and filtered output of your qmail logfiles
>> now :D
>>
>> Cheers,
>> Sebastian
>>
>> Otto Berger wrote:
>>
>>> you could also use fail2ban for that. You just have to specify a custom
>>> rule ("filter") for the spamdyke-log output. Then the sender ip will be
>>> released after a specified timeframe and not blocked forever ;).
>>>
>>> (IMHO it is still not a very good idea to block by firewall)
>>>
>>> Otto
>>>
>>> Sebastian Grewe schrieb:
>>>
>>>
Hey Guys,
I have been working on a simple bash script that will read from it's
standard input and presents some statistics from the logfile in realtime
(when used with "tail -f .." ).
After a few days that we have been attacked by spambots I got curious
how to avoid these things in the future. The script we use is able to
count the denied connections
per IP and, if desired, adds this IP to the Firewall to reject incoming
connections (brutal, I know). As the firewalling is optional you might
still be interested in it to run just
to see what's going on.
It's written for BASH 3.0.15 but with a little change in the pattern
matcher it runs on higher versions too. To start it in live mode run it
like this:
tail -f /var/log/qmail/smtp/current | qmail_parser.sh
and if you just want to scan some files and see what happened to this:
cat /var/log/qmail/smtp/* | qmail_parser.sh
Since it's BASH it's not very good when it comes to performance but does
the trick well when used with "tail". Also it's not catching everything
(yet) since I was looking for only
some very specific lines in the logfile. Anyhow, try it out and tell me
what you think - attached the current script to this mail.
Cheers,
Sebastian
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>> ___
>>> spamdyke-users mailing list
>>> spamdyke-users@spamdyke.org
>>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>>
>>>
>
>
>
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
Sorry to say that I haven't had a chance to check out your script yet,
Sebastian. :(
Speaking of colored and filtered qmail logfiles though, there's a nice
'qmlog' script at qtp.qmailtoaster.com (part of the qmailtoaster-plus
package). It allows easy viewing and searching of qmail (et al) logs.
I'm wondering if your 'coloring and filtering' might be a nice
enhancement to that script. Care to have a look into it?
Sebastian Grewe wrote:
> I totally forgot about that - but I am not using the script to block
> them forever, just to monitor qmail when a large amount of connections
> is coming in (which happens ever so often). Even so I did turn off
> the blocking feature since qmail handles it just fine and connections
> clear up after a while. I was just concerned that legitimate e-mail
> wouldn't
> be coming through - but since they try to resend if no connection could
> be established that's not a concern anymore.
>
> So yeah, I use it to see what's being blocked and for what reason - even
> added whitelist matches now.
>
> It's basically just colored and filtered output of your qmail logfiles
> now :D
>
> Cheers,
> Sebastian
>
> Otto Berger wrote:
>> you could also use fail2ban for that. You just have to specify a custom
>> rule ("filter") for the spamdyke-log output. Then the sender ip will be
>> released after a specified timeframe and not blocked forever ;).
>>
>> (IMHO it is still not a very good idea to block by firewall)
>>
>> Otto
>>
>> Sebastian Grewe schrieb:
>>
>>> Hey Guys,
>>>
>>> I have been working on a simple bash script that will read from it's
>>> standard input and presents some statistics from the logfile in realtime
>>> (when used with "tail -f .." ).
>>> After a few days that we have been attacked by spambots I got curious
>>> how to avoid these things in the future. The script we use is able to
>>> count the denied connections
>>> per IP and, if desired, adds this IP to the Firewall to reject incoming
>>> connections (brutal, I know). As the firewalling is optional you might
>>> still be interested in it to run just
>>> to see what's going on.
>>>
>>> It's written for BASH 3.0.15 but with a little change in the pattern
>>> matcher it runs on higher versions too. To start it in live mode run it
>>> like this:
>>>
>>> tail -f /var/log/qmail/smtp/current | qmail_parser.sh
>>>
>>> and if you just want to scan some files and see what happened to this:
>>>
>>> cat /var/log/qmail/smtp/* | qmail_parser.sh
>>>
>>> Since it's BASH it's not very good when it comes to performance but does
>>> the trick well when used with "tail". Also it's not catching everything
>>> (yet) since I was looking for only
>>> some very specific lines in the logfile. Anyhow, try it out and tell me
>>> what you think - attached the current script to this mail.
>>>
>>> Cheers,
>>> Sebastian
>>>
>>>
>>>
>>>
>>> ___
>>> spamdyke-users mailing list
>>> spamdyke-users@spamdyke.org
>>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>>
>> ___
>> spamdyke-users mailing list
>> spamdyke-users@spamdyke.org
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>
--
-Eric 'shubes'
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
I totally forgot about that - but I am not using the script to block
them forever, just to monitor qmail when a large amount of connections
is coming in (which happens ever so often). Even so I did turn off
the blocking feature since qmail handles it just fine and connections
clear up after a while. I was just concerned that legitimate e-mail
wouldn't
be coming through - but since they try to resend if no connection could
be established that's not a concern anymore.
So yeah, I use it to see what's being blocked and for what reason - even
added whitelist matches now.
It's basically just colored and filtered output of your qmail logfiles
now :D
Cheers,
Sebastian
Otto Berger wrote:
> you could also use fail2ban for that. You just have to specify a custom
> rule ("filter") for the spamdyke-log output. Then the sender ip will be
> released after a specified timeframe and not blocked forever ;).
>
> (IMHO it is still not a very good idea to block by firewall)
>
> Otto
>
> Sebastian Grewe schrieb:
>
>> Hey Guys,
>>
>> I have been working on a simple bash script that will read from it's
>> standard input and presents some statistics from the logfile in realtime
>> (when used with "tail -f .." ).
>> After a few days that we have been attacked by spambots I got curious
>> how to avoid these things in the future. The script we use is able to
>> count the denied connections
>> per IP and, if desired, adds this IP to the Firewall to reject incoming
>> connections (brutal, I know). As the firewalling is optional you might
>> still be interested in it to run just
>> to see what's going on.
>>
>> It's written for BASH 3.0.15 but with a little change in the pattern
>> matcher it runs on higher versions too. To start it in live mode run it
>> like this:
>>
>> tail -f /var/log/qmail/smtp/current | qmail_parser.sh
>>
>> and if you just want to scan some files and see what happened to this:
>>
>> cat /var/log/qmail/smtp/* | qmail_parser.sh
>>
>> Since it's BASH it's not very good when it comes to performance but does
>> the trick well when used with "tail". Also it's not catching everything
>> (yet) since I was looking for only
>> some very specific lines in the logfile. Anyhow, try it out and tell me
>> what you think - attached the current script to this mail.
>>
>> Cheers,
>> Sebastian
>>
>>
>>
>>
>> ___
>> spamdyke-users mailing list
>> spamdyke-users@spamdyke.org
>> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>>
> ___
> spamdyke-users mailing list
> spamdyke-users@spamdyke.org
> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
>
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] My logfile parser (Script)
you could also use fail2ban for that. You just have to specify a custom
rule ("filter") for the spamdyke-log output. Then the sender ip will be
released after a specified timeframe and not blocked forever ;).
(IMHO it is still not a very good idea to block by firewall)
Otto
Sebastian Grewe schrieb:
> Hey Guys,
>
> I have been working on a simple bash script that will read from it's
> standard input and presents some statistics from the logfile in realtime
> (when used with "tail -f .." ).
> After a few days that we have been attacked by spambots I got curious
> how to avoid these things in the future. The script we use is able to
> count the denied connections
> per IP and, if desired, adds this IP to the Firewall to reject incoming
> connections (brutal, I know). As the firewalling is optional you might
> still be interested in it to run just
> to see what's going on.
>
> It's written for BASH 3.0.15 but with a little change in the pattern
> matcher it runs on higher versions too. To start it in live mode run it
> like this:
>
> tail -f /var/log/qmail/smtp/current | qmail_parser.sh
>
> and if you just want to scan some files and see what happened to this:
>
> cat /var/log/qmail/smtp/* | qmail_parser.sh
>
> Since it's BASH it's not very good when it comes to performance but does
> the trick well when used with "tail". Also it's not catching everything
> (yet) since I was looking for only
> some very specific lines in the logfile. Anyhow, try it out and tell me
> what you think - attached the current script to this mail.
>
> Cheers,
> Sebastian
>
>
>
>
> ___
> spamdyke-users mailing list
> spamdyke-users@spamdyke.org
> http://www.spamdyke.org/mailman/listinfo/spamdyke-users
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] My logfile parser (Script)
Hey Guys, I have been working on a simple bash script that will read from it's standard input and presents some statistics from the logfile in realtime (when used with "tail -f .." ). After a few days that we have been attacked by spambots I got curious how to avoid these things in the future. The script we use is able to count the denied connections per IP and, if desired, adds this IP to the Firewall to reject incoming connections (brutal, I know). As the firewalling is optional you might still be interested in it to run just to see what's going on. It's written for BASH 3.0.15 but with a little change in the pattern matcher it runs on higher versions too. To start it in live mode run it like this: tail -f /var/log/qmail/smtp/current | qmail_parser.sh and if you just want to scan some files and see what happened to this: cat /var/log/qmail/smtp/* | qmail_parser.sh Since it's BASH it's not very good when it comes to performance but does the trick well when used with "tail". Also it's not catching everything (yet) since I was looking for only some very specific lines in the logfile. Anyhow, try it out and tell me what you think - attached the current script to this mail. Cheers, Sebastian qmail_parser.sh Description: Bourne shell script ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
