-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi,
Please don't reply off-list. See comments below. On 12-02-16 17:17, Charles Bradshaw wrote: > Hi Tom > > Humm.. yes we agree that storing vast numbers of bans in fail2ban > is the wrong approach. Duplicate data always is. > > I'm unfamiliar shorewall, I'm almost sure ipset is iptables > equivalent of shorewall blacklists. You are wrong. Shorewall is a management wrapper around iptables/ipset, just like f2b is. It is just better suited for managing static rules. > The whole point is ipsets are fast, efficient and separate and > therefore ease the maintenance problem. Yes. But f2b is not fast. Which is why you should not look for a solution to your problem within f2b. > Looking out there, it's pretty obvious the uninitiated are having > problems with fail2bans complexity, especially with permanent bans > and how to unban the inadvertent ones. Your bans are static, not dynamic, so *don't* try to manage it with a tool that is dedicated towards managing dynamic bans. KISS principle applies here. > > I'm not even sure that permanent banning is a good idea. I started > out trying to reduce the frequency of fake attempts at my forum, > but after more than 12 months I'm still seeing an increase in the > number of bans per hour! If you've read up on botnets, you known this si game you'll lose. The bots won't disappear, and it wont help you to ban them forever. But I wasn't really looking into this part of your problem, just trying to solve your "how do I manage a lot of perm bans efficiently" problem. > > If you know how to cleanly disable the sqlite functionality I would > be grateful for the heads-up. > > Charles Bradshaw > > On Fri, 2016-02-12 at 15:16 +0100, Tom Hendrikx wrote: >> Hi, >> >> Maybe an interesting side note: fail2ban is built to quickly ban >> *and* unban problematic ip addresses. The whole nature of >> fail2ban is (IMHO) in the fact that it automatically unbans ip >> addresses after a while. >> >> However, you state that you have a list of 17000 ip adresses that >> are permanently banned. There is no reason to have fail2ban >> maintain this list. I fixed this by having an action in f2b that >> sent the addresses to the shorewall blacklist (which IS devised >> for perm bans). An empty unban action and irrelevant unban time >> complete the setup. >> >> >> You can replace shorewall with any other solution you like, of >> course. But keeping the perm bans in f2b is IMHO simply using the >> wrong tool for the job. >> >> Regards, Tom >> >> On 12-02-16 14:49, Charles Bradshaw wrote: >>> Bill, >>> >>> Sorry again, I actually miss read your first reply. I read >>> actionban instead of actionunban. >>> >>> I am indeed saving and restoring the ipset. At least, that's >>> what I used to do until I found fail2ban taking hours to >>> shutdown. Last time I hit the boot button after about an hour >>> with the result that the ipset was left intact. ipset has built >>> in and well documented method for backup and restore. >>> >>> While I understand your proposed method and see how it would >>> work, I make the following observations: >>> >>> 1 - Your method has a certain pragmatic elegance, but is >>> devious and will certainly confuse the uninitiated! >>> >>> 2 - I can see how your method will work if implemented from >>> square one, but what about the 17000 odds IP which have been >>> previously band with a ban time of forever? I've been running >>> the particular jail with bantime = -1 for well over a year >>> now. >>> >>> 3 - Why store anything at all in an external database. Ipsets >>> are just that, a highly efficient linked to iptables database. >>> The botnet problem is increasing rapidly. Today I'm seeing >>> 8/hour originally it was 2 or 3. In the meantime > 17000 IP >>> have been permanently banned. That says there are botnets out >>> there with orders more than 10000 infected machines! We know >>> not when this will, in effect, escalate to Denial of Service! >>> Several hours to shutdown is a kind of DNS! >>> >>> Back on a pragmatic front, storing and manipulating vast >>> amounts of duplicate data is simply not good practice. If you >>> look out there you will find much discussion on the subject of >>> how to unban the inadvertently banned. I might be wrong, but I >>> suspect because sqlite permanent banning was implemented >>> without due consideration of the consequences on existing >>> installations. >>> >>> I think what I really need to understand now is; how does >>> fail2ban 'think' an IP is banned or not. Where is the database? >>> When is it written/read? In what version of fail2ban did sqlite >>> get implemented. At present my >>> /var/lib/fail2ban/fail2ban.sqlite3 has 7.9MB of entries. >>> >>> I ask again how do I turn sqlite activity off? Just point me >>> at the documentation. >>> >>> Charles Bradshaw >>> >>> On Thu, 2016-02-11 at 22:31 -0500, Bill Shirley wrote: >>>> When you said: This leaves the ipset intact. I made the the >>>> assumption, maybe incorrectly, that you were saving your >>>> ipset with some utility on shutdown and restoring after a >>>> re-boot. >>>> >>>> If that IS the case then change your jail to: bantime = 60 >>>> >>>> and make actionunban empty in your .local action: >>>> #actionunban = ipset -exist del fail2ban-<name> <ip> >>>> actionunban = >>>> >>>> fail2ban will ban the IP address and in one minute it will >>>> unban it. However, with actionunban being empty, the IP >>>> address will not be removed from the ipset. So now fail2ban >>>> thinks very few, if any, addresses are banned. With very few >>>> addresses to 'remove', shutdown should be quick. >>>> >>>> Bill >>>> >>>> >>>> On 2/11/2016 7:03 PM, Charles Bradshaw wrote: >>>> >>>>> Thanks Bill, >>>>> >>>>> Sorry I'm being a bit dim. Do you mean to temporarily >>>>> modify the actionban in >>>>> /etc/fail2ban/action.d/myaction.conf before the shutdown? >>>>> How does that affect the shutdown? I can see how it affects >>>>> the restart but eh.. no action actionban no bans at all >>>>> after restart! >>>>> >>>>> Surely deleting the actionstop clause altogether, thus >>>>> preventing deletion of the ipset and a modified actionstart >>>>> to do nothing if the ipset already exists. Then neither >>>>> start nor stop take time. >>>>> >>>>> I see the new sqlite behavior, but then where is the >>>>> reference to dbfile forcing all the bans into >>>>> /var/lib/fail2ban/fail2ban.sqlite3 it is not in my >>>>> fail2ban.conf! If its use is default behaviour how do I >>>>> disable it? >>>>> >>>>> On Thu, 2016-02-11 at 12:19 -0500, Bill Shirley wrote: >>>>>> Try using an empty actionunban in your action and set >>>>>> the bantime = 60 in your jail. This way fail2ban thinks >>>>>> it's unbanning after a minute. fail2ban shutdown should >>>>>> be quick. >>>>>> >>>>>> Bill >>>>>> >>>>>> On 2/11/2016 5:15 AM, Charles Bradshaw wrote: >>>>>>> Hello list, >>>>>>> >>>>>>> I am running fail2ban.noarch 0.9.3-1.el6.1 as >>>>>>> installed from the CentOS repository. >>>>>>> >>>>>>> I have one ipset jail which over time has accumulated >>>>>>> more than 17000 permanent bans. This is causing a >>>>>>> severe problem during restarts. (obviously!) >>>>>>> >>>>>>> First it would take many hours to shut down fail2ban >>>>>>> gracefully the solution is to force a power down. This >>>>>>> leaves the ipset intact. >>>>>>> >>>>>>> Next when the fail2ban server restarts it takes a >>>>>>> similar many hours for the server to redundantly >>>>>>> restore the bans from the database to the already >>>>>>> intact ipset. >>>>>>> >>>>>>> This a ridiculous process! The whole purpose of ipsets >>>>>>> is to efficiently hold vast numbers of blocked IPs. >>>>>>> >>>>>>> The most importantly problem here is fail2ban is >>>>>>> preventing fast clean shutdowns. Understand 17000 bans >>>>>>> is nothing! an ipset can efficiently hold > 65K, under >>>>>>> which circumstances the shutdown and restart delays >>>>>>> would extend to weeks!! The startup delay is not a >>>>>>> severe problem except that 17000 emails and all the >>>>>>> disk activity is a total pain in the ass. >>>>>>> >>>>>>> So the question is: how to turn off fail2ban >>>>>>> gracefully without these ridiculous delays. >>>>>>> >>>>>>> Also note when fail2ban shuts down the ipset entries >>>>>>> in iptables do not get deleted, but that's another >>>>>>> story. >>>>>>> >>>>>>> Thanks in advance, Charles Bradshaw >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------------- - -- >> >>>>>>> - ------------ >>>>>>> >>>>>>> >> Site24x7 APM Insight: Get Deep Visibility into Application >> Performance >>>>>>> APM + Mobile APM + RUM: Monitor 3 App instances at >>>>>>> just $35/Month Monitor end-to-end web transactions and >>>>>>> take corrective actions now Troubleshoot faster and >>>>>>> improve end-user experience. Signup Now! >>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>>>>>> >>>>>>> >> >>>>>>> _______________________________________________ >>>>>>> Fail2ban-users mailing list >>>>>>> [email protected] >>>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>>>>> >>>>>> >>>>>>> >> >>>>>>> - ------------------------------------------------------------------------ >> ------ >>>>>> Site24x7 APM Insight: Get Deep Visibility into >>>>>> Application Performance APM + Mobile APM + RUM: Monitor 3 >>>>>> App instances at just $35/Month Monitor end-to-end web >>>>>> transactions and take corrective actions now Troubleshoot >>>>>> faster and improve end-user experience. Signup Now! >>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>>>>> >>>>>> >> >>>>>> _______________________________________________ >>>>>> Fail2ban-users mailing list >>>>>> [email protected] >>>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>>>> >>>>> >>>>> >>>>>> - -------------------------------------------------------------------- >> ---------- >>>>> >>>>> >> Site24x7 APM Insight: Get Deep Visibility into Application >> Performance >>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just >>>>> $35/Month Monitor end-to-end web transactions and take >>>>> corrective actions now Troubleshoot faster and improve >>>>> end-user experience. Signup Now! >>>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>>>> >>>>> >> >>>>> _______________________________________________ >>>>> Fail2ban-users mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>>> >>>> >>>>> - --------------------------------------------------------------------- >> --------- >>>> >>>> >> Site24x7 APM Insight: Get Deep Visibility into Application >> Performance >>>> APM + Mobile APM + RUM: Monitor 3 App instances at just >>>> $35/Month Monitor end-to-end web transactions and take >>>> corrective actions now Troubleshoot faster and improve >>>> end-user experience. Signup Now! >>>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>>> _______________________________________________ >>>> Fail2ban-users mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>> >>> >>> >>> -------------------------------------------------------------------- - -- >> >>> - -------- >>> >>> >> Site24x7 APM Insight: Get Deep Visibility into Application >> Performance >>> APM + Mobile APM + RUM: Monitor 3 App instances at just >>> $35/Month Monitor end-to-end web transactions and take >>> corrective actions now Troubleshoot faster and improve end-user >>> experience. Signup Now! >>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >>> _______________________________________________ >>> Fail2ban-users mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/fail2ban-users >>> >> >> --------------------------------------------------------------------- - --------- >> >> Site24x7 APM Insight: Get Deep Visibility into Application Performance >> APM + Mobile APM + RUM: Monitor 3 App instances at just >> $35/Month Monitor end-to-end web transactions and take corrective >> actions now Troubleshoot faster and improve end-user experience. >> Signup Now! >> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >> _______________________________________________ Fail2ban-users >> mailing list [email protected] >> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWvgtqAAoJEJPfMZ19VO/1AA8QAL7WfivoJDdu4n62ohxXM1Uf vofny27NrFWXTbtQCK8cMU/kjOxNTnhKjhSxWeo2CpUTqEdFFvFdbHPNmaU1iEmg /8KN5u3n+O/CFty0PtDHsxHFMkyD+GUGqk2liQiSMB8vvW3tC6/c6NB1jtu1owTr 2kttwuaCQrtFtQ+MhDsYZOAqbBOKOYaxRPtpUdW1SvntNLGLbl4ECDb43UTYRgSA ZvXhRZR3CNmgqzKFJJGgETScj9fWiYKlNw/SKN88N0ozJKp9334z1lWAnAS32E/n VVQciNfB9cc9fLRbKd9LCQuXXskdOzub4QN1iwci5Cr4ULRxgXo7sEZDexCdAEv4 T4AB9c0EoYh7wK05rZ4If1zJJkVMLFT6RLvWgzrXUMXhAjXNlbTOq0ZoEGLkDSIX r56ILq4Pe6PWkUCLhxmmEZHc8Tm+cG4gOD5e3VPmv+xK7ulaWtt55EknjevOvhhV 6I+Bd2kEugVUpjmm37jN+0vVC+Zmt/oLMeqhIIMB7yNZuvWa6rqJ3vJVV8AqeIO1 em9JxZgUac+I4L84e3jGyw9/ZRw5xhp7uW6h86F+31HomdpOAPptJVGJC352+cRu VoItu8dsagkoj/jAJQ+afCexyBDc17GngMyZVpwVCY6qpScx6Be//YlDicDAyrR6 g53jhdb45aVSUPf09znj =p+R3 -----END PGP SIGNATURE----- ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Fail2ban-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fail2ban-users
