Hey, Taking a look
Cheers On 25/04/2017 19:51, Dave wrote: > Try the following apache config in your directory directive, or > .htaccess > file: > > BrowserMatchNoCase Baiduspider botblock > BrowserMatchNoCase Semrushbot botblock > BrowserMatchNoCase Ahrefsbot botblock > Order Deny,Allow > Deny from env=botblock > > Should block those specific bots, while allowing others to use http. > It > could take a few weeks before the bots realise you've made a change to > your > robots.txt . > > Cheers, > davek > > > > In the year 2017, of the month of April, on the 26th day, Carsten > Haitzler wrote: >> I've had to disable the whole http support for now for >> git.enlightenment.org >> because several bots are crawling it causing our VM to basically be >> loaded with >> 10-20 cgit cgi's running git queries for history etc. continually. I/O >> and >> system load is going through the roof as a result and causing other >> stuff like >> phab to crawl and begin timing out. >> >> So anyone using HTTP for doing cmdline git stuff is, at this moment, >> going to >> find things not working. SSH and GIT protocol should still work. I'll >> keep this >> shut down for a few hours hoping the bots give up. >> >> I added a robots.txt and edited the cigtrc to deny all bots from >> indexing >> git.enlightenment.org - but the bots seem to be ignoring that now that >> they >> have decided to start indexing. >> >> I am wondering if this has been the cause of our issues - being >> overloaded by >> indexer bots. FYI I counted 3 different bots indexing cgit at the same >> time: >> Baiduspider, Semrushbot, Ahrefsbot. >> >> I hope later they will start listening to robots.txt, but for now I >> need to >> keep things off until the bots give up. >> >> -- >> ------------- Codito, ergo sum - "I code, therefore I am" >> -------------- >> The Rasterman (Carsten Haitzler) ras...@rasterman.com >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> _______________________________________________ >> enlightenment-devel mailing list >> enlightenment-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel -- Bertrand ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
