> If they do not respect them, then you can use this program: > http://danielwebb.us/software/bot-trap/ to catch them. > If you are doing this, Martin, use the German version instead: > http://www.spider-trap.de/ > because it has a few useful additions. I forget what now. > > Most scrapers, these days, respect robots.txt which will make this > program useless for catching them. But some days you can get lucky.
That would also be an idea. I'll see how the throttling works out; if it fails (either because it still gets overloaded - which shouldn't happen - or because legitimate users complain), I'll try that one. Regards, Martin _______________________________________________ Catalog-SIG mailing list [email protected] http://mail.python.org/mailman/listinfo/catalog-sig
