Hi Nick,
Below is a verbatim copy
of /var/fail2ban/action.d//iptables-ipset-proto4.conf
it is dated Aug 1 2015 and was installed from the CentOS repo. Observe
the note in the actionstop clause. There is a similar file for IPv6. The
jail evoking it is below that.
~ o ~
# Fail2Ban configuration file
#
# Author: Daniel Black
#
# This is for ipset protocol 4 (ipset v4.2). If you have a later version
# of ipset try to use the iptables-ipset-proto6.conf as it does some
things
# nicer.
#
# This requires the program ipset which is normally in package called
ipset.
#
# IPset was a feature introduced in the linux kernel 2.6.39 and 3.0.0
kernels.
#
# If you are running on an older kernel you make need to patch in
external
# modules. Debian squeeze can do this with:
# apt-get install xtables-addons-source
# module-assistant auto-install xtables-addons
#
# Debian wheezy and above uses protocol 6
[INCLUDES]
before = iptables-common.conf
[Definition]
# Option: actionstart
# Notes.: command executed once at the start of Fail2Ban.
# Values: CMD
#
actionstart = ipset --create f2b-<name> iphash
<iptables> -I <chain> -p <protocol> -m multiport --dports
<port> -m set --match-set f2b-<name> src -j <blocktype>
# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
# Note - (Charles Bradshaw Sept 2015) the following does not appear to
# work with iptables.i686 1.4.7-16.el6, therefore delete the
# iptables entries manually by rule number. (see iptables -D)
#
actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports
<port> -m set --match-set f2b-<name> src -j <blocktype>
ipset --flush f2b-<name>
ipset --destroy f2b-<name>
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionban = ipset --test f2b-<name> <ip> || ipset --add f2b-<name> <ip>
# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionunban = ipset --test f2b-<name> <ip> && ipset --del f2b-<name>
<ip>
[Init]
~ o ~
And the relevant parts of my /etc/fail2ban/jail.conf
~ o ~
[forum-bots]
enabled = true
filter = forum-bots
action = iptables-ipset-proto4[name=nobots, port="http,https",
protocol=tcp]
sendmail-whois[name=nobots, dest=root]
logpath = /var/log/httpd/some_access_log ; << this is fake. ;-}
bantime = -1 ; forever!!
maxretry = 1 ; because the bastards are fast and very persistent.
~ o ~
The filter (in my case forum-bots) depends, of course, on what exactly
you want to ban.
Please understand I'm researching how to change
iptables-ipset-proto4.conf because fail2ban has a severe problem with
large ipsets during shutdown.
I will post my solution when I get to the bottom of it, but please treat
all code I post as a starting point only. Be warned, different versions
of fail2ban might do things differently!
Charles Bradshaw
On Fri, 2016-02-12 at 09:40 -0500, Jeremy Baker wrote:
> I would be interested in the script you use to place the IPs into the
> ipsets if you feel like sharing.
> --
> Jeremy Baker <[email protected]>
> GnuPGP fingerprint =
> EE66 AC49 E008 E09A 7A2A 0195 50EF 580B EDBB 95B6
>
> On 02/12/2016 09:13 AM, Nick Howitt wrote:
>
> > Charles,
> >
> > For #2 you can easily manually add the 17,000 IP's to ipset using
> > the command "ipset add {set-name} IP_address". It would be trivial
> > to create a script to do it, or put your list of IP's into Excel,
> > create a text field with "ipset add {set-name} " then join the
> > fields together and copy and paste the results into PuTTy or a
> > single executable file.
> >
> > Can I give you another idea? Have a look at the file on Emerging
> > Threats,
> > https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt.
> > I've scripted that into a couple of ipset sets (one for IP's, one
> > for subnets). I also do some other stuff, but this is a very good
> > start. Blocking subnets gives less rules than individual IP's.
> >
> > Regards,
> >
> > Nick
> >
> > On 12/02/2016 13:49, Charles Bradshaw wrote:
> >
> > > Bill,
> > >
> > > Sorry again, I actually miss read your first reply. I read actionban
> > > instead of actionunban.
> > >
> > > I am indeed saving and restoring the ipset. At least, that's what I used
> > > to do until I found fail2ban taking hours to shutdown. Last time I hit
> > > the boot button after about an hour with the result that the ipset was
> > > left intact. ipset has built in and well documented method for backup
> > > and restore.
> > >
> > > While I understand your proposed method and see how it would work, I
> > > make the following observations:
> > >
> > > 1 - Your method has a certain pragmatic elegance, but is devious and
> > > will certainly confuse the uninitiated!
> > >
> > > 2 - I can see how your method will work if implemented from square one,
> > > but what about the 17000 odds IP which have been previously band with a
> > > ban time of forever? I've been running the particular jail with bantime
> > > = -1 for well over a year now.
> > >
> > > 3 - Why store anything at all in an external database. Ipsets are just
> > > that, a highly efficient linked to iptables database. The botnet problem
> > > is increasing rapidly. Today I'm seeing 8/hour originally it was 2 or 3.
> > > In the meantime > 17000 IP have been permanently banned. That says there
> > > are botnets out there with orders more than 10000 infected machines! We
> > > know not when this will, in effect, escalate to Denial of Service!
> > > Several hours to shutdown is a kind of DNS!
> > >
> > > Back on a pragmatic front, storing and manipulating vast amounts of
> > > duplicate data is simply not good practice. If you look out there you
> > > will find much discussion on the subject of how to unban the
> > > inadvertently banned. I might be wrong, but I suspect because sqlite
> > > permanent banning was implemented without due consideration of the
> > > consequences on existing installations.
> > >
> > > I think what I really need to understand now is; how does fail2ban
> > > 'think' an IP is banned or not. Where is the database? When is it
> > > written/read? In what version of fail2ban did sqlite get implemented. At
> > > present my /var/lib/fail2ban/fail2ban.sqlite3 has 7.9MB of entries.
> > >
> > > I ask again how do I turn sqlite activity off? Just point me at the
> > > documentation.
> > >
> > > Charles Bradshaw
> > >
> > > On Thu, 2016-02-11 at 22:31 -0500, Bill Shirley wrote:
> > > > When you said:
> > > > This leaves the ipset intact.
> > > > I made the the assumption, maybe incorrectly, that you were saving
> > > > your ipset with some utility on shutdown and restoring after a
> > > > re-boot.
> > > >
> > > > If that IS the case then change your jail to:
> > > > bantime = 60
> > > >
> > > > and make actionunban empty in your .local action:
> > > > #actionunban = ipset -exist del fail2ban-<name> <ip>
> > > > actionunban =
> > > >
> > > > fail2ban will ban the IP address and in one minute it will unban it.
> > > > However, with actionunban being empty, the IP address will not be
> > > > removed from the ipset. So now fail2ban thinks very few, if any,
> > > > addresses are banned. With very few addresses to 'remove', shutdown
> > > > should be quick.
> > > >
> > > > Bill
> > > >
> > > >
> > > > On 2/11/2016 7:03 PM, Charles Bradshaw wrote:
> > > >
> > > > > Thanks Bill,
> > > > >
> > > > > Sorry I'm being a bit dim. Do you mean to temporarily modify the
> > > > > actionban in /etc/fail2ban/action.d/myaction.conf before the shutdown?
> > > > > How does that affect the shutdown? I can see how it affects the
> > > > > restart
> > > > > but eh.. no action actionban no bans at all after restart!
> > > > >
> > > > > Surely deleting the actionstop clause altogether, thus preventing
> > > > > deletion of the ipset and a modified actionstart to do nothing if the
> > > > > ipset already exists. Then neither start nor stop take time.
> > > > >
> > > > > I see the new sqlite behavior, but then where is the reference to
> > > > > dbfile
> > > > > forcing all the bans into /var/lib/fail2ban/fail2ban.sqlite3 it is not
> > > > > in my fail2ban.conf! If its use is default behaviour how do I disable
> > > > > it?
> > > > >
> > > > > On Thu, 2016-02-11 at 12:19 -0500, Bill Shirley wrote:
> > > > > > Try using an empty actionunban in your action and set the bantime =
> > > > > > 60 in your jail. This way fail2ban thinks it's unbanning
> > > > > > after a minute. fail2ban shutdown should be quick.
> > > > > >
> > > > > > Bill
> > > > > >
> > > > > > On 2/11/2016 5:15 AM, Charles Bradshaw wrote:
> > > > > > > Hello list,
> > > > > > >
> > > > > > > I am running fail2ban.noarch 0.9.3-1.el6.1 as installed from the
> > > > > > > CentOS
> > > > > > > repository.
> > > > > > >
> > > > > > > I have one ipset jail which over time has accumulated more than
> > > > > > > 17000
> > > > > > > permanent bans. This is causing a severe problem during restarts.
> > > > > > > (obviously!)
> > > > > > >
> > > > > > > First it would take many hours to shut down fail2ban gracefully
> > > > > > > the
> > > > > > > solution is to force a power down. This leaves the ipset intact.
> > > > > > >
> > > > > > > Next when the fail2ban server restarts it takes a similar many
> > > > > > > hours for
> > > > > > > the server to redundantly restore the bans from the database to
> > > > > > > the
> > > > > > > already intact ipset.
> > > > > > >
> > > > > > > This a ridiculous process! The whole purpose of ipsets is to
> > > > > > > efficiently
> > > > > > > hold vast numbers of blocked IPs.
> > > > > > >
> > > > > > > The most importantly problem here is fail2ban is preventing fast
> > > > > > > clean
> > > > > > > shutdowns. Understand 17000 bans is nothing! an ipset can
> > > > > > > efficiently
> > > > > > > hold > 65K, under which circumstances the shutdown and restart
> > > > > > > delays
> > > > > > > would extend to weeks!! The startup delay is not a severe problem
> > > > > > > except
> > > > > > > that 17000 emails and all the disk activity is a total pain in
> > > > > > > the ass.
> > > > > > >
> > > > > > > So the question is: how to turn off fail2ban gracefully without
> > > > > > > these
> > > > > > > ridiculous delays.
> > > > > > >
> > > > > > > Also note when fail2ban shuts down the ipset entries in iptables
> > > > > > > do not
> > > > > > > get deleted, but that's another story.
> > > > > > >
> > > > > > > Thanks in advance, Charles Bradshaw
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------------------------------------------------------
> > > > > > > Site24x7 APM Insight: Get Deep Visibility into Application
> > > > > > > Performance
> > > > > > > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> > > > > > > Monitor end-to-end web transactions and take corrective actions
> > > > > > > now
> > > > > > > Troubleshoot faster and improve end-user experience. Signup Now!
> > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> > > > > > > _______________________________________________
> > > > > > > Fail2ban-users mailing list
> > > > > > > [email protected]
> > > > > > > https://lists.sourceforge.net/lists/listinfo/fail2ban-users
> > > > > > ------------------------------------------------------------------------------
> > > > > > Site24x7 APM Insight: Get Deep Visibility into Application
> > > > > > Performance
> > > > > > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> > > > > > Monitor end-to-end web transactions and take corrective actions now
> > > > > > Troubleshoot faster and improve end-user experience. Signup Now!
> > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> > > > > > _______________________________________________
> > > > > > Fail2ban-users mailing list
> > > > > > [email protected]
> > > > > > https://lists.sourceforge.net/lists/listinfo/fail2ban-users
> > > > > ------------------------------------------------------------------------------
> > > > > Site24x7 APM Insight: Get Deep Visibility into Application Performance
> > > > > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> > > > > Monitor end-to-end web transactions and take corrective actions now
> > > > > Troubleshoot faster and improve end-user experience. Signup Now!
> > > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> > > > > _______________________________________________
> > > > > Fail2ban-users mailing list
> > > > > [email protected]
> > > > > https://lists.sourceforge.net/lists/listinfo/fail2ban-users
> > > > ------------------------------------------------------------------------------
> > > > Site24x7 APM Insight: Get Deep Visibility into Application Performance
> > > > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> > > > Monitor end-to-end web transactions and take corrective actions now
> > > > Troubleshoot faster and improve end-user experience. Signup Now!
> > > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> > > > _______________________________________________ Fail2ban-users mailing
> > > > list [email protected]
> > > > https://lists.sourceforge.net/lists/listinfo/fail2ban-users
> > >
> > > ------------------------------------------------------------------------------
> > > Site24x7 APM Insight: Get Deep Visibility into Application Performance
> > > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> > > Monitor end-to-end web transactions and take corrective actions now
> > > Troubleshoot faster and improve end-user experience. Signup Now!
> > > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> > > _______________________________________________
> > > Fail2ban-users mailing list
> > > [email protected]
> > > https://lists.sourceforge.net/lists/listinfo/fail2ban-users
> >
> >
> >
> > ------------------------------------------------------------------------------
> > Site24x7 APM Insight: Get Deep Visibility into Application Performance
> > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> > Monitor end-to-end web transactions and take corrective actions now
> > Troubleshoot faster and improve end-user experience. Signup Now!
> > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> >
> >
> > _______________________________________________
> > Fail2ban-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> _______________________________________________ Fail2ban-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
