ianmcook commented on issue #4853: URL: https://github.com/apache/arrow-adbc/issues/4853#issuecomment-6443176588
I have been looking at how our docs show up in search results. For queries like "ADBC" or "Arrow Database Connectivity", `https://arrow.apache.org/docs/format/ADBC.html` often ranks above `https://arrow.apache.org/adbc/current/index.html`. I would like to revisit canonical URLs with a narrower policy that I think could help fix this while addressing the concerns raised in #4854 and #4855. ### Current state: - Apart from a few generated redirect stubs, none of our published pages have `rel=canonical`. We publish 27 copies of the docs (0.1.0 through 25 plus `main/` and `current/`). `current/` is byte-identical to `25/`, so each page's search ranking is split across all of those copies. - Our home page's `<title>` is "ADBC 25 Documentation", and its only `<h1>` ("Apache Arrow ADBC") is hidden. The Arrow page's `<title>` and `<h1>` are both "ADBC: Arrow Database Connectivity". ### Canonical vs. the alternatives: I agree with @amoeba's point in #4855 that doc versions are separate resources. Google's docs say canonical is for "duplicate or very similar pages". But: - I think it will have an effect. Python, PostgreSQL, Django, Node.js, and Arrow's own docs all point the canonical tags in old versions at the latest version. Someone on the PostgreSQL web mailing list [said](https://postgresql.org/message-id/CALWCfd%2Bnj%3D4j%3DhWdd_EgFhTX%3D8ZgEbUWZrQCnJU5h-N-xFMMCw%40mail.gmail.com) that, based on Citus's docs, this makes old versions nearly impossible to find in Google. That's why I'm proposing it only for the 0.x versions. - I would avoid `noindex` and robots.txt. They would hide old versions from search even more than canonical tags would. Google [recommends against](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) using `noindex` for this, since it "will completely block the page from Search", and says not to use robots.txt for it either. Also, robots.txt belongs to arrow-site and applies to all of Arrow, not just ADBC. ### What I would propose: - Pages in `current/` and in the latest release directory get a canonical tag pointing to `current/`. (These are exact copies of each other, which is what canonical tags are meant for.) - Pages in the 0.x versions get a canonical tag pointing to the same page in `current/`. These versions are very old, and the 0.3.0 docs are what Google was showing in this issue. Pages that no longer exist in `current/`, or that are now redirects, get no tag. If a page is removed from `current/` later, the tag on the 0.x page is removed too. - Pages in versions 13 and later, except the latest release, won't have canonical tags pointing to `current/`, so they can still show up in search on their own. When a new release comes out, the canonical tags in the previous release are changed to point to that release's own pages instead. (We can reconsider adding canonical tags to these later.) - Pages in `main/` get no tag, as suggested in #4854, because they can include new pages that aren't in `current/` yet. - I'd also add a sitemap for `current/` that we can submit to Google Search Console, and change the home page's `<title>` and `<h1>` to "ADBC: Arrow Database Connectivity". I have this ready on a branch and can open a PR. Release builds will add the canonical tags, and a new script run nightly by `website_build.sh` will keep the tags in all published versions up to date as new releases come out. For the docs that are already published, I'd also make a one-time commit to asf-site. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
