-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi,
Maybe an interesting side note: fail2ban is built to quickly ban *and*
unban problematic ip addresses. The whole nature of fail2ban is (IMHO)
in the fact that it automatically unbans ip addresses after a while.
However, you state that you have a list of 17000 ip adresses that are
permanently banned. There is no reason to have fail2ban maintain this
list. I fixed this by having an action in f2b that sent the addresses to
the shorewall blacklist (which IS devised for perm bans). An empty unban
action and irrelevant unban time complete the setup.
You can replace shorewall with any other solution you like, of course.
But keeping the perm bans in f2b is IMHO simply using the wrong tool
for the job.
Regards,
Tom
On 12-02-16 14:49, Charles Bradshaw wrote:
> Bill,
>
> Sorry again, I actually miss read your first reply. I read
> actionban instead of actionunban.
>
> I am indeed saving and restoring the ipset. At least, that's what I
> used to do until I found fail2ban taking hours to shutdown. Last
> time I hit the boot button after about an hour with the result that
> the ipset was left intact. ipset has built in and well documented
> method for backup and restore.
>
> While I understand your proposed method and see how it would work,
> I make the following observations:
>
> 1 - Your method has a certain pragmatic elegance, but is devious
> and will certainly confuse the uninitiated!
>
> 2 - I can see how your method will work if implemented from square
> one, but what about the 17000 odds IP which have been previously
> band with a ban time of forever? I've been running the particular
> jail with bantime = -1 for well over a year now.
>
> 3 - Why store anything at all in an external database. Ipsets are
> just that, a highly efficient linked to iptables database. The
> botnet problem is increasing rapidly. Today I'm seeing 8/hour
> originally it was 2 or 3. In the meantime > 17000 IP have been
> permanently banned. That says there are botnets out there with
> orders more than 10000 infected machines! We know not when this
> will, in effect, escalate to Denial of Service! Several hours to
> shutdown is a kind of DNS!
>
> Back on a pragmatic front, storing and manipulating vast amounts
> of duplicate data is simply not good practice. If you look out
> there you will find much discussion on the subject of how to unban
> the inadvertently banned. I might be wrong, but I suspect because
> sqlite permanent banning was implemented without due consideration
> of the consequences on existing installations.
>
> I think what I really need to understand now is; how does fail2ban
> 'think' an IP is banned or not. Where is the database? When is it
> written/read? In what version of fail2ban did sqlite get
> implemented. At present my /var/lib/fail2ban/fail2ban.sqlite3 has
> 7.9MB of entries.
>
> I ask again how do I turn sqlite activity off? Just point me at
> the documentation.
>
> Charles Bradshaw
>
> On Thu, 2016-02-11 at 22:31 -0500, Bill Shirley wrote:
>> When you said: This leaves the ipset intact. I made the the
>> assumption, maybe incorrectly, that you were saving your ipset
>> with some utility on shutdown and restoring after a re-boot.
>>
>> If that IS the case then change your jail to: bantime = 60
>>
>> and make actionunban empty in your .local action: #actionunban =
>> ipset -exist del fail2ban-<name> <ip> actionunban =
>>
>> fail2ban will ban the IP address and in one minute it will unban
>> it. However, with actionunban being empty, the IP address will
>> not be removed from the ipset. So now fail2ban thinks very few,
>> if any, addresses are banned. With very few addresses to
>> 'remove', shutdown should be quick.
>>
>> Bill
>>
>>
>> On 2/11/2016 7:03 PM, Charles Bradshaw wrote:
>>
>>> Thanks Bill,
>>>
>>> Sorry I'm being a bit dim. Do you mean to temporarily modify
>>> the actionban in /etc/fail2ban/action.d/myaction.conf before
>>> the shutdown? How does that affect the shutdown? I can see how
>>> it affects the restart but eh.. no action actionban no bans at
>>> all after restart!
>>>
>>> Surely deleting the actionstop clause altogether, thus
>>> preventing deletion of the ipset and a modified actionstart to
>>> do nothing if the ipset already exists. Then neither start nor
>>> stop take time.
>>>
>>> I see the new sqlite behavior, but then where is the reference
>>> to dbfile forcing all the bans into
>>> /var/lib/fail2ban/fail2ban.sqlite3 it is not in my
>>> fail2ban.conf! If its use is default behaviour how do I
>>> disable it?
>>>
>>> On Thu, 2016-02-11 at 12:19 -0500, Bill Shirley wrote:
>>>> Try using an empty actionunban in your action and set the
>>>> bantime = 60 in your jail. This way fail2ban thinks it's
>>>> unbanning after a minute. fail2ban shutdown should be
>>>> quick.
>>>>
>>>> Bill
>>>>
>>>> On 2/11/2016 5:15 AM, Charles Bradshaw wrote:
>>>>> Hello list,
>>>>>
>>>>> I am running fail2ban.noarch 0.9.3-1.el6.1 as installed
>>>>> from the CentOS repository.
>>>>>
>>>>> I have one ipset jail which over time has accumulated more
>>>>> than 17000 permanent bans. This is causing a severe problem
>>>>> during restarts. (obviously!)
>>>>>
>>>>> First it would take many hours to shut down fail2ban
>>>>> gracefully the solution is to force a power down. This
>>>>> leaves the ipset intact.
>>>>>
>>>>> Next when the fail2ban server restarts it takes a similar
>>>>> many hours for the server to redundantly restore the bans
>>>>> from the database to the already intact ipset.
>>>>>
>>>>> This a ridiculous process! The whole purpose of ipsets is
>>>>> to efficiently hold vast numbers of blocked IPs.
>>>>>
>>>>> The most importantly problem here is fail2ban is preventing
>>>>> fast clean shutdowns. Understand 17000 bans is nothing! an
>>>>> ipset can efficiently hold > 65K, under which circumstances
>>>>> the shutdown and restart delays would extend to weeks!! The
>>>>> startup delay is not a severe problem except that 17000
>>>>> emails and all the disk activity is a total pain in the
>>>>> ass.
>>>>>
>>>>> So the question is: how to turn off fail2ban gracefully
>>>>> without these ridiculous delays.
>>>>>
>>>>> Also note when fail2ban shuts down the ipset entries in
>>>>> iptables do not get deleted, but that's another story.
>>>>>
>>>>> Thanks in advance, Charles Bradshaw
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
- ------------
>>>>>
>>>>>
Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just
>>>>> $35/Month Monitor end-to-end web transactions and take
>>>>> corrective actions now Troubleshoot faster and improve
>>>>> end-user experience. Signup Now!
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>>>>>
>>>>>
_______________________________________________
>>>>> Fail2ban-users mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>>
>>>>
>>>>>
- ------------------------------------------------------------------------
- ------
>>>> Site24x7 APM Insight: Get Deep Visibility into Application
>>>> Performance APM + Mobile APM + RUM: Monitor 3 App instances
>>>> at just $35/Month Monitor end-to-end web transactions and
>>>> take corrective actions now Troubleshoot faster and improve
>>>> end-user experience. Signup Now!
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>>>>
>>>>
_______________________________________________
>>>> Fail2ban-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>
>>>
>>> --------------------------------------------------------------------
- ----------
>>>
>>>
Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> APM + Mobile APM + RUM: Monitor 3 App instances at just
>>> $35/Month Monitor end-to-end web transactions and take
>>> corrective actions now Troubleshoot faster and improve end-user
>>> experience. Signup Now!
>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>>>
>>>
_______________________________________________
>>> Fail2ban-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>
>> ---------------------------------------------------------------------
- ---------
>>
>>
Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just
>> $35/Month Monitor end-to-end web transactions and take corrective
>> actions now Troubleshoot faster and improve end-user experience.
>> Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>> _______________________________________________ Fail2ban-users
>> mailing list [email protected]
>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>
>
>
> ----------------------------------------------------------------------
- --------
>
>
Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions
> now Troubleshoot faster and improve end-user experience. Signup
> Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> _______________________________________________ Fail2ban-users
> mailing list [email protected]
> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWvelNAAoJEJPfMZ19VO/1qX4P/1XjghWFSDPUY1S40/lxsXtb
MTyk6UWcD0AqqSMVzcLfV+eK+WGNJqwgTGvyQ7xECurrHCDFQrXeTCkfYVyRw7Hn
RH6IswOdjZFeP8gP3kDErdu932ccXiBdu9pBkSQQlO+M2+ISWnPr2hKJla80jDKM
n0xK+vRjrUjlUormlRIFdDmtY0ITdGrN2j3yzAUDrgCxw/nHlsD+aA2ohbEs5jtp
1EGyOZ5Bz3VXlKlQ2uEIx3kYK4brGUplX0tG+KkValgb8KSmjaxItcez4kyut9oM
GjvwHVNz4Pc/5lsvBXovj8deDwXPHi8jzqxUiBrlmmXFGHB7Fk1Xo4gG5UJOJvtt
m2MS2f16/dqNSBQBDmkj5p9XUFsT2/2TtyC3CNTNgdUHWI7kppUoW7DICvQPJlKM
q1k6oLHeNKfqkM/NvJsOb0A+nHlVjRUDmvZQeAygwcvdPpsYUdnzvFJckx1GltVI
7yasTR68afWEFvnz6Cc1Up/R0zQV2LS7nxT/F1s5yhcy/+jGzPsPSBJm/xqjZiF+
DZ3Ii7omUaH9SgjqhuVIK0UgW571nLJoNMENXu6rLgu/kQt+gS5grKzXTvVhUUPS
iqZHKxC3rBk4lg3JeprjV2bN3AA5mbS93MCBqfpzc/YiIpSH/E3i5xZI8Hb1M2VK
WMwHBjXeNjmuZjOGAe5C
=GnPZ
-----END PGP SIGNATURE-----
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
