[issue32936] RobotFileParser.parse() should raise an exception when the robots.txt file is invalid

[Irit Katriel]

Irit Katriel  added the comment:

Please reopen this or create a new issue if this is still a problem and you can 
provide the missing information.

--
resolution:  -> rejected
stage:  -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32936] RobotFileParser.parse() should raise an exception when the robots.txt file is invalid

[Irit Katriel]

Irit Katriel  added the comment:

The link to the robots.txt file no longer works, so it's not clear how to 
reproduce the problem you are seeing. Can you post the complete information on 
this issue?

--
nosy: +iritkatriel
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32936] RobotFileParser.parse() should raise an exception when the robots.txt file is invalid

[Oudin]

New submission from Oudin :

When processing an ill-formed robots.txt file (like 
https://tiny.tobast.fr/robots-file ), the RobotFileParser.parse method does not 
instantiate the entries or the default_entry attributes.

In my opinion, the method should raise an exception when no valid User-agent 
entry (or if there exists an invalid User-agent entry) is found in the 
robots.txt file.

Otherwise, the only method available is to check the None-liness of 
default_entry, which is not documented in the documentation 
(https://docs.python.org/dev/library/urllib.robotparser.html).

According to your opinion on this, I can implement what is necessary and create 
a PR on Github.

--
components: Library (Lib)
messages: 312711
nosy: Guinness
priority: normal
severity: normal
status: open
title: RobotFileParser.parse() should raise an exception when the robots.txt 
file is invalid
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com