On Tue, Dec 22, 2015 at 4:35 PM, Benjamin Peterson <benja...@python.org> wrote: > We've played around with robots.txt, but it's still useful for old docs > to be indexed (e.g., for removed features), which just need to figure > out how to get them deprecation in results. I wonder if <link > ref="canonical"> in the old docs would help.
Yes, this is probably the correct approach (though it's rel="canonical"): https://support.google.com/webmasters/answer/139066?hl=en It's always been an inconvenience when Google displays the docs for different, old versions (3.2, 3.3, etc) -- seemingly at random, and sometimes instead of the newest version. Fortunately, this seems to be improving over time. By using rel="canonical", you would have control over this and can signal to Google to display only the newest, stable version of a given doc. This would probably have other positive benefits like consolidating the "search juice" onto one page, so it's no longer spread thinly across multiple versions. There would still be a question of how you want to handle 2 versus 3. --Chris > > On Sat, Dec 19, 2015, at 11:02, A.M. Kuchling wrote: >> On Sat, Dec 19, 2015 at 08:55:26PM +1000, Nick Coghlan wrote: >> > Even once the new docs are in place, getting them to the top of search >> > of results ahead of archived material that may be years out of date is >> > likely to still be a challenge - for example, even considering just >> > the legacy distutils docs, the "3.1" and "2" docs appear ... >> >> We probably need to update https://docs.python.org/robots.txt, which >> currently contains: >> >> # Prevent development and old documentation from showing up in search >> results. >> User-agent: * >> # Disallow: /dev >> Disallow: /release >> >> The intent was to allow the latest version of the docs to be crawled. >> Unfortunately, with the current hierarchy we'd have to disallow each >> version, e.g. >> >> Disallow: /2.6/* >> Disallow: /3.0/* >> Disallow: /3.1/* >> >> And we'd need to update it for each new major release. >> >> --amk >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com