On 04/27/2017 03:37 PM, Jonathan Aquilina wrote: > Hi Carsten, > > What advantage does http for git have? Isn't git better to use to pull > and push stuff to the repositories or is cgit more a web based ui to > view the repository history etc? > If you go to http://git.enlightenment.org you can browse the repo's (I guess this is whats being scraped), for those without commit access checking out a repo with https is also quite common.
> > On 2017-04-26 04:18, Carsten Haitzler wrote: > >> I've had to disable the whole http support for now for git.enlightenment.org >> because several bots are crawling it causing our VM to basically be loaded >> with >> 10-20 cgit cgi's running git queries for history etc. continually. I/O and >> system load is going through the roof as a result and causing other stuff >> like >> phab to crawl and begin timing out. >> >> So anyone using HTTP for doing cmdline git stuff is, at this moment, going to >> find things not working. SSH and GIT protocol should still work. I'll keep >> this >> shut down for a few hours hoping the bots give up. >> >> I added a robots.txt and edited the cigtrc to deny all bots from indexing >> git.enlightenment.org - but the bots seem to be ignoring that now that they >> have decided to start indexing. >> >> I am wondering if this has been the cause of our issues - being overloaded by >> indexer bots. FYI I counted 3 different bots indexing cgit at the same time: >> Baiduspider, Semrushbot, Ahrefsbot. >> >> I hope later they will start listening to robots.txt, but for now I need to >> keep things off until the bots give up. > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > enlightenment-devel mailing list > enlightenment-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
