Benjamin Peterson added the comment: Unfortunately, I don't think it will be that easy because I don't think robots.txt supports wildcard paths like that. Possibly, we should just whitelist a few important repositories.
---------- nosy: +benjamin.peterson _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue13924> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com