I would be interested in the script you use to place the IPs into the
ipsets if you feel like sharing.
--
Jeremy Baker <[email protected]>
GnuPGP fingerprint =
EE66 AC49 E008 E09A 7A2A 0195 50EF 580B EDBB 95B6
On 02/12/2016 09:13 AM, Nick Howitt wrote:
> Charles,
>
> For #2 you can easily manually add the 17,000 IP's to ipset using the
> command "ipset add {set-name} IP_address". It would be trivial to
> create a script to do it, or put your list of IP's into Excel, create
> a text field with "ipset add {set-name} " then join the fields
> together and copy and paste the results into PuTTy or a single
> executable file.
>
> Can I give you another idea? Have a look at the file on Emerging
> Threats,
> https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt. I've
> scripted that into a couple of ipset sets (one for IP's, one for
> subnets). I also do some other stuff, but this is a very good start.
> Blocking subnets gives less rules than individual IP's.
>
> Regards,
>
> Nick
>
> On 12/02/2016 13:49, Charles Bradshaw wrote:
>> Bill,
>>
>> Sorry again, I actually miss read your first reply. I read actionban
>> instead of actionunban.
>>
>> I am indeed saving and restoring the ipset. At least, that's what I used
>> to do until I found fail2ban taking hours to shutdown. Last time I hit
>> the boot button after about an hour with the result that the ipset was
>> left intact. ipset has built in and well documented method for backup
>> and restore.
>>
>> While I understand your proposed method and see how it would work, I
>> make the following observations:
>>
>> 1 - Your method has a certain pragmatic elegance, but is devious and
>> will certainly confuse the uninitiated!
>>
>> 2 - I can see how your method will work if implemented from square one,
>> but what about the 17000 odds IP which have been previously band with a
>> ban time of forever? I've been running the particular jail with bantime
>> = -1 for well over a year now.
>>
>> 3 - Why store anything at all in an external database. Ipsets are just
>> that, a highly efficient linked to iptables database. The botnet problem
>> is increasing rapidly. Today I'm seeing 8/hour originally it was 2 or 3.
>> In the meantime > 17000 IP have been permanently banned. That says there
>> are botnets out there with orders more than 10000 infected machines! We
>> know not when this will, in effect, escalate to Denial of Service!
>> Several hours to shutdown is a kind of DNS!
>>
>> Back on a pragmatic front, storing and manipulating vast amounts of
>> duplicate data is simply not good practice. If you look out there you
>> will find much discussion on the subject of how to unban the
>> inadvertently banned. I might be wrong, but I suspect because sqlite
>> permanent banning was implemented without due consideration of the
>> consequences on existing installations.
>>
>> I think what I really need to understand now is; how does fail2ban
>> 'think' an IP is banned or not. Where is the database? When is it
>> written/read? In what version of fail2ban did sqlite get implemented. At
>> present my /var/lib/fail2ban/fail2ban.sqlite3 has 7.9MB of entries.
>>
>> I ask again how do I turn sqlite activity off? Just point me at the
>> documentation.
>>
>> Charles Bradshaw
>>
>> On Thu, 2016-02-11 at 22:31 -0500, Bill Shirley wrote:
>>> When you said:
>>> This leaves the ipset intact.
>>> I made the the assumption, maybe incorrectly, that you were saving
>>> your ipset with some utility on shutdown and restoring after a
>>> re-boot.
>>>
>>> If that IS the case then change your jail to:
>>> bantime = 60
>>>
>>> and make actionunban empty in your .local action:
>>> #actionunban = ipset -exist del fail2ban-<name> <ip>
>>> actionunban =
>>>
>>> fail2ban will ban the IP address and in one minute it will unban it.
>>> However, with actionunban being empty, the IP address will not be
>>> removed from the ipset. So now fail2ban thinks very few, if any,
>>> addresses are banned. With very few addresses to 'remove', shutdown
>>> should be quick.
>>>
>>> Bill
>>>
>>>
>>> On 2/11/2016 7:03 PM, Charles Bradshaw wrote:
>>>
>>>> Thanks Bill,
>>>>
>>>> Sorry I'm being a bit dim. Do you mean to temporarily modify the
>>>> actionban in /etc/fail2ban/action.d/myaction.conf before the shutdown?
>>>> How does that affect the shutdown? I can see how it affects the restart
>>>> but eh.. no action actionban no bans at all after restart!
>>>>
>>>> Surely deleting the actionstop clause altogether, thus preventing
>>>> deletion of the ipset and a modified actionstart to do nothing if the
>>>> ipset already exists. Then neither start nor stop take time.
>>>>
>>>> I see the new sqlite behavior, but then where is the reference to dbfile
>>>> forcing all the bans into /var/lib/fail2ban/fail2ban.sqlite3 it is not
>>>> in my fail2ban.conf! If its use is default behaviour how do I disable
>>>> it?
>>>>
>>>> On Thu, 2016-02-11 at 12:19 -0500, Bill Shirley wrote:
>>>>> Try using an empty actionunban in your action and set the bantime = 60 in
>>>>> your jail. This way fail2ban thinks it's unbanning
>>>>> after a minute. fail2ban shutdown should be quick.
>>>>>
>>>>> Bill
>>>>>
>>>>> On 2/11/2016 5:15 AM, Charles Bradshaw wrote:
>>>>>> Hello list,
>>>>>>
>>>>>> I am running fail2ban.noarch 0.9.3-1.el6.1 as installed from the CentOS
>>>>>> repository.
>>>>>>
>>>>>> I have one ipset jail which over time has accumulated more than 17000
>>>>>> permanent bans. This is causing a severe problem during restarts.
>>>>>> (obviously!)
>>>>>>
>>>>>> First it would take many hours to shut down fail2ban gracefully the
>>>>>> solution is to force a power down. This leaves the ipset intact.
>>>>>>
>>>>>> Next when the fail2ban server restarts it takes a similar many hours for
>>>>>> the server to redundantly restore the bans from the database to the
>>>>>> already intact ipset.
>>>>>>
>>>>>> This a ridiculous process! The whole purpose of ipsets is to efficiently
>>>>>> hold vast numbers of blocked IPs.
>>>>>>
>>>>>> The most importantly problem here is fail2ban is preventing fast clean
>>>>>> shutdowns. Understand 17000 bans is nothing! an ipset can efficiently
>>>>>> hold > 65K, under which circumstances the shutdown and restart delays
>>>>>> would extend to weeks!! The startup delay is not a severe problem except
>>>>>> that 17000 emails and all the disk activity is a total pain in the ass.
>>>>>>
>>>>>> So the question is: how to turn off fail2ban gracefully without these
>>>>>> ridiculous delays.
>>>>>>
>>>>>> Also note when fail2ban shuts down the ipset entries in iptables do not
>>>>>> get deleted, but that's another story.
>>>>>>
>>>>>> Thanks in advance, Charles Bradshaw
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>>>>> Monitor end-to-end web transactions and take corrective actions now
>>>>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>>>>>> _______________________________________________
>>>>>> Fail2ban-users mailing list
>>>>>> [email protected]
>>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>>> ------------------------------------------------------------------------------
>>>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>>>> Monitor end-to-end web transactions and take corrective actions now
>>>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>>>>> _______________________________________________
>>>>> Fail2ban-users mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>>> ------------------------------------------------------------------------------
>>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>>> Monitor end-to-end web transactions and take corrective actions now
>>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>>>> _______________________________________________
>>>> Fail2ban-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>> ------------------------------------------------------------------------------
>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>> Monitor end-to-end web transactions and take corrective actions now
>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>>> _______________________________________________ Fail2ban-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>>
>> ------------------------------------------------------------------------------
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>> _______________________________________________
>> Fail2ban-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
>
>
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>
>
> _______________________________________________
> Fail2ban-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/fail2ban-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
