New submission from Emmanuel Arias <emmanuelaria...@gmail.com>:
Hello I open an issue from a PR that I think need discussion: I write here the @fuwaraido's PR description: > I found some website like "sony.co.jp" requests User-Agent field in headers > when I try to fetch their robots.txt. Unless they returns 403 Forbidden > status code and RobotTextParser fails to read robots.txt. Currently there is > no way to specify headers so I propose add headers arguments in constructor. ---------- components: Library (Lib) messages: 351440 nosy: eamanu priority: normal pull_requests: 15423 severity: normal status: open title: Add headers parameter on RobotFileParser type: enhancement versions: Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue38067> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
