New submission from Ivaylo Popov <popov....@gmail.com>: http://hg.python.org/robots.txt currently disallows all robots from all paths. This means that the site doesn't show up in Google search results seeking, for instance, browsing access to the python source https://www.google.com/search?ie=UTF-8&q=python+source+browse https://www.google.com/search?ie=UTF-8&q=python+repo+browse https://www.google.com/search?ie=UTF-8&q=hg+python+browse etc...
Instead, robots.txt should allow access to the landing page, http://hg.python.org/, and the landing pages for hosted projects, e.g. http://hg.python.org/cpython/, while prohibiting access to the */rev/*, */shortlog/*, ..., directories. This change would be very easy, cost virtually nothing, and let users find the mercurial repository viewer from search engines. Note that http://svn.python.org/ does show up in search results, as an illustration of how convenient this is. ---------- components: None messages: 152446 nosy: Ivaylo.Popov priority: normal severity: normal status: open title: Mercurial robots.txt should let robots crawl landing pages. type: enhancement _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue13924> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
