Senthil Kumaran <orsent...@gmail.com> added the comment: I modified the patch slightly (so that it takes care of path, query, params and fragments).
Fixed in r83209,r83210 and r83211. I also think that we need to move the robotparser to allow regexs in the allow and disallow patterns. ( Shall open an issue in the tracker, if it is not already present). ---------- resolution: -> fixed stage: -> committed/rejected status: open -> closed _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue6325> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
