My script has some heritage from one found on the ipset site and
heavily hacked. I also broke down the one I saw into different bits.
In /etc/cron.daily:
#!/bin/sh
#
# Update emerging fwrules ipset
#
# * checks online for newer fwrev
# * downloads new ip list only if the online fwrev is not the
local one
# * generates ipset --restore file with temporary ipsets
#
# Changelog:
# 08 Dec 2009 / 1.0 [email protected] initial version
# 10 Feb 2014 / 1.1 [email protected] patched for debug and
iptables entries in ClearOS
# 21 Feb 2014 / 1.2a [email protected] Split into two
programs. This part downloads and formats
# the file and load the ipset data sets
ET_FWREV_URL="https://rules.emergingthreats.net/fwrules/FWrev"
ET_FWREV="/etc/snort.d/rules/emerging_threats/FWrev"
ET_FWRULES="https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt"
ET_FWRULES_TEMP="/etc/snort.d/rules/emerging_threats/emerging-Block-IPs.txt"
ET_FWREV_ENABLE_LOGGING=1
do_log () {
if [ $ET_FWREV_ENABLE_LOGGING -eq 1 ]
then
local PRIO=$1; shift;
logger -p "$PRIO" -t "EMERGING-IPSET-UPDATE" "$*"
fi
}
# See if the file FWrev exists. If not populate it.
if ! [ -f $ET_FWREV ];
then
echo "none" > $ET_FWREV
fi
# get fwrev online
if ! wget -O $ET_FWREV.temp -q $ET_FWREV_URL;
then
do_log error "can't download $ET_FWREV_URL to $ET_FWREV.temp"
exit 1
fi
do_log notice "Local fwrev version " `cat $ET_FWREV`
do_log notice "Online fwrev version " `cat $ET_FWREV.temp`
# Check if local version of rules is the same as the ET version.
If so, exit with no update.
if cmp -s $ET_FWREV $ET_FWREV.temp; then
do_log notice "no update required"
rm -f $ET_FWREV.temp
exit
fi
do_log notice "Local fwrev " `cat $ET_FWREV` " does not match
online fwrev " `cat $ET_FWREV.temp` ". Start update"
# Download new list
if ! "wget" -O "$ET_FWRULES_TEMP" -q "$ET_FWRULES"
then
do_log error "can't download $ET_FWRULES to $ET_FWRULES_TEMP"
exit 1
fi
# ensure that ipsets exist
ipset destroy -q blacklist-temp
ipset destroy -q blacklistnet-temp
ipset -N blacklist-temp iphash --hashsize 26244 -exist
ipset -N blacklistnet-temp nethash --hashsize 3456 -exist
ipset -N blacklist iphash --hashsize 26244 -exist
ipset -N blacklistnet nethash --hashsize 3456 -exist
# Host IP Adresses
for i in $(egrep
'^[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}$'
"$ET_FWRULES_TEMP")
do
ipset -A -exist blacklist-temp $i
done
# NET addresses
for i in $(egrep
'^[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}/[[:digit:]]{1,2}$'
"$ET_FWRULES_TEMP")
do
ipset -A -exist blacklistnet-temp $i
done
mv -f $ET_FWREV.temp $ET_FWREV
rm -f $ET_FWRULES_TEMP
do_log notice "ipset rules committed at revision " `cat $ET_FWREV`
# swap sets
ipset swap blacklist blacklist-temp
ipset swap blacklistnet blacklistnet-temp
# ensure that temp sets do not exist
ipset destroy -q "blacklist-temp"
ipset destroy -q "blacklistnet-temp"
ipset save blacklist > /usr/src/ipset_blacklist.save
sed -i 's/create/create -exist/g' /usr/src/ipset_blacklist.save
sed -i 's/add/add -exist/g' /usr/src/ipset_blacklist.save
ipset save blacklistnet > /usr/src/ipset_blacklistnet.save
sed -i 's/create/create -exist/g' /usr/src/ipset_blacklistnet.save
sed -i 's/add/add -exist/g' /usr/src/ipset_blacklistnet.save
In the firewall script:
if [ "`lsmod | grep ip_set`" = "" ]; then
modprobe ip_set
fi
#
# Block using ET blacklists
#
# ensure that ipsets exist
ipset create blacklist iphash --hashsize 26244 -exist
ipset create blacklistnet nethash --hashsize 3456 -exist
OpenTCPPorts="25,443,587,993,51413"
OpenUDPPorts="1194,51413"
# commit iptables rules
iptables -N ET_BLOCK > /dev/null 2>&1
# comment out the next line if you do not want any firewall
logging
#iptables -A ET_BLOCK -j LOG --log-level INFO --log-prefix
"ET_BLOCK: "
iptables -A ET_BLOCK -j DROP
iptables -I INPUT -m set --match-set blacklist src -m state
--state NEW -j ET_BLOCK
iptables -I FORWARD -m set --match-set blacklist src -j ET_BLOCK
iptables -I FORWARD -m set --match-set blacklist dst -j ET_BLOCK
iptables -I OUTPUT -p tcp ! --dport 25 -m set --match-set
blacklist dst -m state --state NEW -j ET_BLOCK
iptables -I OUTPUT -p udp -m set --match-set blacklist dst -m
state --state NEW -j ET_BLOCK
iptables -I INPUT -m set --match-set blacklistnet src -m state
--state NEW -p tcp -m multiport --dports $OpenTCPPorts -j ET_BLOCK
iptables -I INPUT -m set --match-set blacklistnet src -m state
--state NEW -p udp -m multiport --dports $OpenUDPPorts -j ET_BLOCK
iptables -I FORWARD -m set --match-set blacklistnet src -j
ET_BLOCK
iptables -I FORWARD -m set --match-set blacklistnet dst -j
ET_BLOCK
iptables -I OUTPUT -p tcp ! --dport 25 -m set --match-set
blacklistnet dst -m state --state NEW -j ET_BLOCK
iptables -I OUTPUT -p udp -m set --match-set blacklistnet dst -m
state --state NEW -j ET_BLOCK
In /etc/rc.d/rc.local (distro dependant):
# Load in all previously saved ipset sets
if [ "`lsmod | grep ip_set`" = "" ]; then
modprobe ip_set
fi
for file in /usr/src/ipset_*.save ; do
ipset restore < $file
done
You'll need to sort out some of your file locations for the download
and temporary files. You'll also need to modify the firewall rules
to suite. I only block the ones I have open normally, but you could
choose to block a much bigger range of ports or all ports. I
modprobe everywhere because I don't know the startup order of the
various bits of the distro but it is probably unnecessary.
This is all part of something bigger where I also parse the
https://rules.emergingthreats.net/open-nogpl/snort-2.9.0/rules/compromised-ips.txt
and https://rules.emergingthreats.net/blockrules/emerging-tor.rules
and parse them for IP's which I throw into another ipset set.
HTH,
Nick
On 12/02/2016 14:40, Jeremy Baker
wrote:
I would be interested in the script you use to place the IPs into
the ipsets if you feel like sharing.
--
Jeremy Baker <[email protected]>
GnuPGP fingerprint =
EE66 AC49 E008 E09A 7A2A 0195 50EF 580B EDBB 95B6
On 02/12/2016 09:13 AM, Nick Howitt
wrote:
Charles,
For #2 you can easily manually add the 17,000 IP's to ipset
using the command "ipset add {set-name} IP_address". It would be
trivial to create a script to do it, or put your list of IP's
into Excel, create a text field with "ipset add {set-name} "
then join the fields together and copy and paste the results
into PuTTy or a single executable file.
Can I give you another idea? Have a look at the file on Emerging
Threats, https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt.
I've scripted that into a couple of ipset sets (one for IP's,
one for subnets). I also do some other stuff, but this is a very
good start. Blocking subnets gives less rules than individual
IP's.
Regards,
Nick
On 12/02/2016 13:49, Charles
Bradshaw wrote:
Bill,
Sorry again, I actually miss read your first reply. I read actionban
instead of actionunban.
I am indeed saving and restoring the ipset. At least, that's what I used
to do until I found fail2ban taking hours to shutdown. Last time I hit
the boot button after about an hour with the result that the ipset was
left intact. ipset has built in and well documented method for backup
and restore.
While I understand your proposed method and see how it would work, I
make the following observations:
1 - Your method has a certain pragmatic elegance, but is devious and
will certainly confuse the uninitiated!
2 - I can see how your method will work if implemented from square one,
but what about the 17000 odds IP which have been previously band with a
ban time of forever? I've been running the particular jail with bantime
= -1 for well over a year now.
3 - Why store anything at all in an external database. Ipsets are just
that, a highly efficient linked to iptables database. The botnet problem
is increasing rapidly. Today I'm seeing 8/hour originally it was 2 or 3.
In the meantime > 17000 IP have been permanently banned. That says there
are botnets out there with orders more than 10000 infected machines! We
know not when this will, in effect, escalate to Denial of Service!
Several hours to shutdown is a kind of DNS!
Back on a pragmatic front, storing and manipulating vast amounts of
duplicate data is simply not good practice. If you look out there you
will find much discussion on the subject of how to unban the
inadvertently banned. I might be wrong, but I suspect because sqlite
permanent banning was implemented without due consideration of the
consequences on existing installations.
I think what I really need to understand now is; how does fail2ban
'think' an IP is banned or not. Where is the database? When is it
written/read? In what version of fail2ban did sqlite get implemented. At
present my /var/lib/fail2ban/fail2ban.sqlite3 has 7.9MB of entries.
I ask again how do I turn sqlite activity off? Just point me at the
documentation.
Charles Bradshaw
On Thu, 2016-02-11 at 22:31 -0500, Bill Shirley wrote:
When you said:
This leaves the ipset intact.
I made the the assumption, maybe incorrectly, that you were saving
your ipset with some utility on shutdown and restoring after a
re-boot.
If that IS the case then change your jail to:
bantime = 60
and make actionunban empty in your .local action:
#actionunban = ipset -exist del fail2ban-<name> <ip>
actionunban =
fail2ban will ban the IP address and in one minute it will unban it.
However, with actionunban being empty, the IP address will not be
removed from the ipset. So now fail2ban thinks very few, if any,
addresses are banned. With very few addresses to 'remove', shutdown
should be quick.
Bill
On 2/11/2016 7:03 PM, Charles Bradshaw wrote:
Thanks Bill,
Sorry I'm being a bit dim. Do you mean to temporarily modify the
actionban in /etc/fail2ban/action.d/myaction.conf before the shutdown?
How does that affect the shutdown? I can see how it affects the restart
but eh.. no action actionban no bans at all after restart!
Surely deleting the actionstop clause altogether, thus preventing
deletion of the ipset and a modified actionstart to do nothing if the
ipset already exists. Then neither start nor stop take time.
I see the new sqlite behavior, but then where is the reference to dbfile
forcing all the bans into /var/lib/fail2ban/fail2ban.sqlite3 it is not
in my fail2ban.conf! If its use is default behaviour how do I disable
it?
On Thu, 2016-02-11 at 12:19 -0500, Bill Shirley wrote:
Try using an empty actionunban in your action and set the bantime = 60 in your jail. This way fail2ban thinks it's unbanning
after a minute. fail2ban shutdown should be quick.
Bill
On 2/11/2016 5:15 AM, Charles Bradshaw wrote:
Hello list,
I am running fail2ban.noarch 0.9.3-1.el6.1 as installed from the CentOS
repository.
I have one ipset jail which over time has accumulated more than 17000
permanent bans. This is causing a severe problem during restarts.
(obviously!)
First it would take many hours to shut down fail2ban gracefully the
solution is to force a power down. This leaves the ipset intact.
Next when the fail2ban server restarts it takes a similar many hours for
the server to redundantly restore the bans from the database to the
already intact ipset.
This a ridiculous process! The whole purpose of ipsets is to efficiently
hold vast numbers of blocked IPs.
The most importantly problem here is fail2ban is preventing fast clean
shutdowns. Understand 17000 bans is nothing! an ipset can efficiently
hold > 65K, under which circumstances the shutdown and restart delays
would extend to weeks!! The startup delay is not a severe problem except
that 17000 emails and all the disk activity is a total pain in the ass.
So the question is: how to turn off fail2ban gracefully without these
ridiculous delays.
Also note when fail2ban shuts down the ipset entries in iptables do not
get deleted, but that's another story.
Thanks in advance, Charles Bradshaw
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________ Fail2ban-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
|
