[issue36207] robotsparser deny all with some rules

STINNER Victor added the comment: I removed two comments: none of the mentioned URL contains a "Disallow: ?" rule and the comments didn't add any value to this issue. It looks like regular spam (SEO). -- ___ Python tracker

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg416847 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg416767 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

adiboo adib added the comment: Hi now it work on all my website https://www.matelesecretairemedicale.com/ -- ___ Python tracker ___

[issue36207] robotsparser deny all with some rules

adiboo adib added the comment: I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: https://www.st-info.fr/robots.txt, I believe that this is how it's implemented now in most parsers ? -- nosy: +adiboo67

[issue36207] robotsparser deny all with some rules

Irit Katriel added the comment: I restored one non-spam message from the OP that was deleted. Changing to enhancement because this is not a bug (i.e., deviation from documentation). I don't know enough about this to have a view on whether this enhancement request should be accepted.

[issue36207] robotsparser deny all with some rules

wats0ns added the comment: I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented now in most parsers ? -- nosy: +quentin-maire

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg402889 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Nico added the comment: Had same problem today for my website (https://www.bonus4casino.fr/), following for a fix -- nosy: +nico.bonefato ___ Python tracker ___

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg338298 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

STINNER Victor added the comment: I removed almost all messages of this issue since most of them looked list SPAM. I also blocked user accounts who posted SPAM. If it was a mistake, contact me. This is the Python bug tracker, not a forum to ask questions how to use Python, or to report

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg365770 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg370275 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg377058 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg377125 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg374642 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg376032 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg366509 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg367546 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg374629 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg372112 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg378070 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg379615 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg379616 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg385859 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg381443 ___ Python tracker ___ ___ Python-bugs-list

[issue36207] robotsparser deny all with some rules

Change by STINNER Victor : -- title: référencement naturel -> robotsparser deny all with some rules ___ Python tracker ___ ___

[issue36207] robotsparser deny all with some rules

jeanotlapin added the comment: Il semblerait que le script continue d'afficher des erreurs et rencontre des bugs. Preuve en est puisque j'ai testé sur ce site d'auto hypnose https://www.lautohypnose.com/ en vain... -- nosy: +jeanotlapin ___ Python

[issue36207] robotsparser deny all with some rules

idee Animation Anniversaire added the comment: idee animation anniversaire est une agence animation à Paris pour les prestations comme pour Animation entreprise, animation arbre de Noël, animation anniversaire à domicile, animation centre de loisir avec spectacle magie, Spectacle

[issue36207] robotsparser deny all with some rules

Nicolas added the comment: Sorry, I meant https://www.meridigital.com -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue36207] robotsparser deny all with some rules

Nicolas added the comment: Seems like we have the same issue with http://meridigital.com/robots.txt -- nosy: +nico702 -matthieuhemea ___ Python tracker ___

[issue36207] robotsparser deny all with some rules

Matthieu hemea added the comment: Hi, Does anyone find the solution ? It would help me for this one : https://www.hemea.com/fr/devis-travaux -- nosy: +matthieuhemea -Jmgray47, Patrick Valibus 410 Gone, amiir.mascud, arnaud, calamina, jeanotlapin

[issue36207] robotsparser deny all with some rules

jeanotlapin added the comment: Bonjour, j'ai rencontré le même soucis. J'ai essayé de le faire fonctionner en vain. J'ai tenté de faire crawler par les robots la page de notre agence de communication à Montpellier https://www.monagencedecommunication.com/agence/montpellier/ mais cela n'a

[issue36207] robotsparser deny all with some rules

amiir mascud added the comment: Can you share the robot file that you are using for your website? I am using the default robot file for my site https://meilleurdumoniteur.fr/ -- nosy: +amiir.mascud ___ Python tracker

[issue36207] robotsparser deny all with some rules

calamina added the comment: I have a problem with my robot.txt on https://www.sondage-remunere.com/ -- nosy: +calamina ___ Python tracker ___

[issue36207] robotsparser deny all with some rules

Arnaud LIPERINI-DIAZ added the comment: Do you have documentation about robotParser ? The robot.txt of this website works fine : https://vauros.com/ -- nosy: +Arnaud LIPERINI-DIAZ ___ Python tracker

[issue36207] robotsparser deny all with some rules

James Gray added the comment: Bonjour, je vois que nous ne sommes pas les seuls dans ce cas, nous avons besoin que les robots indexent nos pages html mais qu'elles n'indexent pas celles en /*.php$ ainsi que les ressources PC en PDF. Nous avons tenté en vain plusieurs solutions en passant par

[issue36207] robotsparser deny all with some rules

Patrick Valibus 410 Gone added the comment: Bonjour, nous n'avons pas réussi à le faire fonctionner. Nous l'avons utilisé dans le cadre d'un test seo car nous essayons e reproduire des alternatives à scrappy. Par exemple le robots devrait bine crawler la page de notre agence seo

[issue36207] robotsparser deny all with some rules

mathias44 added the comment: I can't display my robot.TXT. I want to ban robots https://ereputation-dereferencement.fr/ -- nosy: +mathias44 ___ Python tracker ___

[issue36207] robotsparser deny all with some rules

Fred AYERS added the comment: I tried this one http://gtxgamer.fr/robots.txt/;>http://gtxgamer.fr/robots.txt and it seems to work. -- nosy: +Fred AYERS ___ Python tracker

[issue36207] robotsparser deny all with some rules

asca added the comment: I thought it was going to work but apparently when I try https://www.actusite.fr/robots.txt, it doesn't -- nosy: +artasca ___ Python tracker ___

[issue36207] robotsparser deny all with some rules

Rodriguez added the comment: I can't display my robot.TXT. I want to ban robots https://melwynn-rodriguez.fr/robots.txt -- nosy: +lagustais ___ Python tracker ___

[issue36207] robotsparser deny all with some rules

wats0ns added the comment: I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented now in most parsers ? --

[issue36207] robotsparser deny all with some rules

Cheryl Sabella added the comment: Can you provide a link to documentation showing that "Disallow: ?" shouldn't be the same as deny all? Thanks! -- nosy: +cheryl.sabella ___ Python tracker

[issue36207] robotsparser deny all with some rules

New submission from wats0ns : RobotsParser parse a "Disallow: ?" rule as a deny all, but this is a valid rule that should be interpreted as "Disallow: /?*" or "Disallow: /*?*" -- components: Library (Lib) messages: 337285 nosy: quentin-maire priority: normal severity: normal status: