[issue16099] robotparser doesn't support request rate and crawl delay parameters

Berker Peksag added the comment: I've finally committed your patch to default. Thank you for not giving up, Nikolay :) Note that currently the link in the example section doesn't work. I will open a new issue for that. -- resolution: -> fixed stage: patch review -> resolved status:

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Roundup Robot added the comment: New changeset dbed7cacfb7e by Berker Peksag in branch 'default': Issue #16099: RobotFileParser now supports Crawl-delay and Request-rate https://hg.python.org/cpython/rev/dbed7cacfb7e -- nosy: +python-dev ___ Python

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Hey, Friendly reminder that there has been no activity on this issue for more than an year. Cheers, Nick -- ___ Python tracker

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Hey, Just a friendly reminder that the patch is pending for review and there has been no activity for 3 months (: -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Hey, Just a friendly reminder that there has been no activity for a month and a half and v3 is pending for review (: -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Changes by Berker Peksag berker.pek...@gmail.com: -- assignee: - berker.peksag ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___ ___

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Updated patch, all comments addressed, sorry for the 6 months delay. Please review -- Added file: http://bugs.python.org/file35377/robotparser_v3.patch ___ Python tracker rep...@bugs.python.org

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- assignee: - rhettinger nosy: +rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- assignee: rhettinger - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___ ___

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Hey, Just a reminder friendly reminder that there hasn't been any activity for a month and I have released a v2, pending for review (: -- ___ Python tracker rep...@bugs.python.org

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Thank you for the review! I have addressed your comments and release a v2 of the patch: Highlights: No longer crashes when provided with malformed crawl-delay/robots.txt parameter. Returns None when parameter is missing or syntax is invalid. Simplified

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Oh... Sorry for the spam, could you please verify my documentation link syntax. I'm not entirely sure I got it right. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Hey, it has been more than an year since the last activity. Is there anything else I should do in order for someone of the python devs team to review my changes and perhaps give some feedback? Nick -- ___

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Berker Peksag added the comment: I left a few comments on Rietveld. -- nosy: +berker.peksag versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Changes by Hynek Schlawack h...@ox.cx: -- nosy: +orsenthil ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___ ___ Python-bugs-list mailing

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Changes by Hynek Schlawack h...@ox.cx: -- nosy: +hynek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___ ___ Python-bugs-list mailing list

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Okay, here's a proper patch with documentation entry and test cases. Please review and comment -- Added file: http://bugs.python.org/file27476/robotparser.patch ___ Python tracker rep...@bugs.python.org

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Reformatted patch -- Added file: http://bugs.python.org/file27477/robotparser_reformatted.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Changes by Christian Heimes li...@cheimes.de: -- keywords: +needs review stage: test needed - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16099 ___

[issue16099] robotparser doesn't support request rate and crawl delay parameters

New submission from Nikolay Bogoychev: Robotparser doesn't support two quite important optional parameters from the robots.txt file. I have implemented those in the following way: (Robotparser should be initialized in the usual way: rp = robotparser.RobotFileParser() rp.set_url(..) rp.read )

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Christian Heimes added the comment: Thanks for the patch. New features must be implemented in Python 3.4. Python 2.7 is in feature freeze mode and therefore doesn't get new features. -- keywords: +gsoc nosy: +christian.heimes stage: - test needed versions: +Python 3.4 -Python 2.7

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Nikolay Bogoychev added the comment: Okay, sorry didn't know that (: Here's the same patch (Same functionality) for python3 Feedback is welcome, as always (: -- Added file: http://bugs.python.org/file27374/robotparser.patch ___ Python tracker

[issue16099] robotparser doesn't support request rate and crawl delay parameters

Christian Heimes added the comment: We have a team that mentors new contributors. If you are interested to get your patch into Python 3.4, please read http://pythonmentors.com/ . The people are really friendly and will help you with every step of the process. -- keywords: +easy -gsoc