Berker Peksag added the comment: The Crawl-delay part(issue 16099) is now committed.
---------- stage: -> needs patch title: Support the Sitemap and Crawl-delay extensions in robotparser -> Support the Sitemap extension in robotparser versions: +Python 3.6 -Python 3.5 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue21475> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
