On Wed, 26 Apr 2017 12:51:21 +1000 Dave <[email protected]> said: well i needed a drastic solution asap because the server was at a load of 20-30 and lots of stuff was falling over. i have no done this... before i read this mail :) but thanks for the pointer. :)
> Try the following apache config in your directory directive, or .htaccess > file: > > BrowserMatchNoCase Baiduspider botblock > BrowserMatchNoCase Semrushbot botblock > BrowserMatchNoCase Ahrefsbot botblock > Order Deny,Allow > Deny from env=botblock > > Should block those specific bots, while allowing others to use http. It > could take a few weeks before the bots realise you've made a change to your > robots.txt . > > Cheers, > davek > > > > In the year 2017, of the month of April, on the 26th day, Carsten Haitzler > wrote: > > I've had to disable the whole http support for now for git.enlightenment.org > > because several bots are crawling it causing our VM to basically be loaded > > with 10-20 cgit cgi's running git queries for history etc. continually. I/O > > and system load is going through the roof as a result and causing other > > stuff like phab to crawl and begin timing out. > > > > So anyone using HTTP for doing cmdline git stuff is, at this moment, going > > to find things not working. SSH and GIT protocol should still work. I'll > > keep this shut down for a few hours hoping the bots give up. > > > > I added a robots.txt and edited the cigtrc to deny all bots from indexing > > git.enlightenment.org - but the bots seem to be ignoring that now that they > > have decided to start indexing. > > > > I am wondering if this has been the cause of our issues - being overloaded > > by indexer bots. FYI I counted 3 different bots indexing cgit at the same > > time: Baiduspider, Semrushbot, Ahrefsbot. > > > > I hope later they will start listening to robots.txt, but for now I need to > > keep things off until the bots give up. > > > > -- > > ------------- Codito, ergo sum - "I code, therefore I am" -------------- > > The Rasterman (Carsten Haitzler) [email protected] > > > > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > _______________________________________________ > > enlightenment-devel mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > enlightenment-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) [email protected] ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
