[issue41803] Robots.txt

Change by Irit Katriel : -- resolution: -> rejected stage: -> resolved status: pending -> closed ___ Python tracker ___ ___

[issue41803] Robots.txt

Change by Eric V. Smith : -- status: open -> pending ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41803] Robots.txt

Eric V. Smith added the comment: It's not clear what bug you're describing, or how it relates to Python. Can you provide more information, and show what behavior your seeing, and what behavior you're expecting? -- nosy: +eric.smith ___ Python

[issue41803] Robots.txt

New submission from Skyreka Agence Web : I can't documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.skyreka.com/robots.txt, I believe that this is how it's implemented now in most parsers ? -- messages: 377055