Try the following apache config in your directory directive, or .htaccess file:
BrowserMatchNoCase Baiduspider botblock BrowserMatchNoCase Semrushbot botblock BrowserMatchNoCase Ahrefsbot botblock Order Deny,Allow Deny from env=botblock Should block those specific bots, while allowing others to use http. It could take a few weeks before the bots realise you've made a change to your robots.txt . Cheers, davek In the year 2017, of the month of April, on the 26th day, Carsten Haitzler wrote: > I've had to disable the whole http support for now for git.enlightenment.org > because several bots are crawling it causing our VM to basically be loaded > with > 10-20 cgit cgi's running git queries for history etc. continually. I/O and > system load is going through the roof as a result and causing other stuff like > phab to crawl and begin timing out. > > So anyone using HTTP for doing cmdline git stuff is, at this moment, going to > find things not working. SSH and GIT protocol should still work. I'll keep > this > shut down for a few hours hoping the bots give up. > > I added a robots.txt and edited the cigtrc to deny all bots from indexing > git.enlightenment.org - but the bots seem to be ignoring that now that they > have decided to start indexing. > > I am wondering if this has been the cause of our issues - being overloaded by > indexer bots. FYI I counted 3 different bots indexing cgit at the same time: > Baiduspider, Semrushbot, Ahrefsbot. > > I hope later they will start listening to robots.txt, but for now I need to > keep things off until the bots give up. > > -- > ------------- Codito, ergo sum - "I code, therefore I am" -------------- > The Rasterman (Carsten Haitzler) ras...@rasterman.com > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > enlightenment-devel mailing list > enlightenment-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
