I've had to disable the whole http support for now for git.enlightenment.org because several bots are crawling it causing our VM to basically be loaded with 10-20 cgit cgi's running git queries for history etc. continually. I/O and system load is going through the roof as a result and causing other stuff like phab to crawl and begin timing out.
So anyone using HTTP for doing cmdline git stuff is, at this moment, going to find things not working. SSH and GIT protocol should still work. I'll keep this shut down for a few hours hoping the bots give up. I added a robots.txt and edited the cigtrc to deny all bots from indexing git.enlightenment.org - but the bots seem to be ignoring that now that they have decided to start indexing. I am wondering if this has been the cause of our issues - being overloaded by indexer bots. FYI I counted 3 different bots indexing cgit at the same time: Baiduspider, Semrushbot, Ahrefsbot. I hope later they will start listening to robots.txt, but for now I need to keep things off until the bots give up. -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) [email protected] ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
