For the Ref Guide a good amount of the traffic is probably the old redirect rules in Confluence that explicitly went to 6_6 for a good reason *at the time*. A blanket fix to those rules (it was a discrete list of pages in a text file) to remove the URL in the path would cause them to be automatically redirected to the latest. I’d honestly be fine just removing the redirect rules entirely…it’s been almost 4 years, when do we get to move on?
A lot of the 6.6 traffic is also coming from Google itself since that version of a page frequently appears at the top of search results, possibly at least partially due to that Confluence redirect rule, but I’m sure there are also other factors. IIRC people have proposed general ideas about how to fix that, but I don’t recall seeing actual code or PRs yet to actually address the issue. On Mar 4, 2021, 5:34 AM -0600, Uwe Schindler <u...@thetaphi.de>, wrote: > I disagree with excluding older Java docs or refguide using robots! When I > look for documentation of a class I generally enter class name and version > number into google. > > We can maybe handle this with priorities inside a sitemap.xml or custom http > headers (X-Robots) using a htaccess rule. I can check this out. > > Uwe > > Am March 4, 2021 10:41:52 AM UTC schrieb "Jan Høydahl" > <jan....@cominvent.com>: > > Sure we could do robots. > > > > But I suspect that we put ourselves in this situation through > > https://issues.apache.org/jira/browse/SOLR-10595 ourselves > > Check out the attachment solr_redirects.conf on that JIRA (also here > > https://gist.github.com/janhoy/a3149e1ed27df020194a2de1a7fa2c16) > > > > Here, we explicitly map all pages that used to be in Confluence (which > > there are a ton of links to on the net), to version 6.6 of the guide. > > Of course, if we change those to "latest", some of those links will break, > > but perhaps it would still be better? > > Or can we be more intelligent in the rewrite rules on Solr site - that if > > you try a "/guide/foo.html" link and it is not found, that you display a > > custom error page or go to front page of latest guide? > > > > Jan > > > > > 4. mar. 2021 kl. 10:21 skrev Ishan Chattopadhyaya > > > <ichattopadhy...@gmail.com>: > > > > > > We can add robots.txt to stop Google from indexing/showing in results. > > > > > > On Thu, 4 Mar, 2021, 2:34 pm Jan Høydahl, <jan....@cominvent.com> wrote: > > > > Hi, sending to this list since dev@solr list is not yet announced > > > > properly. > > > > > > > > We have a few days of traffic to the new site and can see the most > > > > visited pages at https://uls.apache.org/exports/solr.apache.org.yaml > > > > (see copy below). > > > > When I search google for "solr query parser", I get the 6.6 guide on > > > > top, which is probably why /guide/6_6/the-standard-query-parser.html > > > > shows up, and the same for the other /guide/6_6/ links. > > > > Some questions: > > > > > > > > • How can we make Google forget about version 6.6? I know we had a > > > > bunch of redirects from Confluence to the 6.6 guide, are they still in > > > > place? > > > > • Why is /docs/6_6_0/solr-core/index.html the 2nd most visited page? > > > > Anywhere that links to it? > > > > • Why is /docs/4_8_1/solr-solrj/index.html so high? Ahywhere that > > > > links to it? > > > > • The /mirrors-solr-latest-redir.html redirect was not working. I just > > > > pushed a fix > > > > > > > > > > > > Sheet3: > > > > Name: Most visited pages, past month > > > > Values: > > > > /index.html: 443 > > > > /docs/6_6_0/solr-core/index.html: 281 > > > > /guide/8_8/solr-tutorial.html: 104 > > > > /news.html: 92 > > > > /guide/solr-tutorial.html: 91 > > > > /resources.html: 75 > > > > /features.html: 69 > > > > /docs/8_7_0/solr-core/index.html: 68 > > > > /downloads.html: 68 > > > > /guide/6_6/the-standard-query-parser.html: 65 > > > > /docs/4_8_1/solr-solrj/index.html: 62 > > > > /guide/6_6/common-query-parameters.html: 50 > > > > /guide/index.html: 46 > > > > /docs/8_8_1/solr-solrj/index.html: 44 > > > > /docs/8_7_0/solr-solrj/index.html: 38 > > > > /docs/8_8_1/solr-core/index.html: 37 > > > > /community.html: 24 > > > > /guide/8_8/: 23 > > > > /guide/6_6/uploading-data-with-index-handlers.html: 22 > > > > /docs/8_6_3/solr-core/index.html: 21 > > > > /guide/6_6/filter-descriptions.html: 21 > > > > /guide/6_6/collections-api.html: 18 > > > > /docs/8_6_2/solr-solrj/overview-summary.html: 16 > > > > /guide/6_6/faceting.html: 16 > > > > /mirrors-solr-latest-redir.html: 15 > > > > /whoweare.html: 15 > > > > /guide/6_6/solrcloud.html: 14 > > > > /guide/6_6/tokenizers.html: 13 > > > > /guide/7_0/solr-configuration-files.html: 13 > > > > /guide/8_8/query-syntax-and-parsing.html: 13 > > > > /security.html: 13 > > > > /guide/6_6/introduction-to-solr-indexing.html: 12 > > > > /guide/8_8/solr-upgrade-notes.html: 12 > > > > /docs/8_0_0/solr-solrj/allclasses-frame.html: 11 > > > > /docs/8_6_3/solr-solrj/index.html: 11 > > > > /guide/6_6/the-dismax-query-parser.html: 11 > > > > /guide/8_8/getting-started.html: 11 > > > > /guide/solr-upgrade-notes.html: 11 > > > > /docs/8_0_0/solr-solrj/overview-summary.html: 10 > > > > /docs/8_6_2/solr-solrj/index.html: 10 > > > > /guide/6_6/running-solr.html: 10 > > > > /docs/7_2_1/solr-solrj/overview-summary.html: 9 > > > > /docs/8_0_0/solr-solrj/overview-frame.html: 9 > > > > /guide/6_6/format-of-solr-xml.html: 9 > > > > /guide/6_6/index.html: 9 > > > > /guide/6_6/learning-to-rank.html: 9 > > > > /guide/6_6/making-and-restoring-backups.html: 9 > > > > /guide/6_6/working-with-dates.html: 9 > > > > /guide/8_0/reindexing.html: 9 > > > > /docs/7_2_1/solr-solrj/allclasses-frame.html: 8 > > > > > > > > > > > > -- > Uwe Schindler > Achterdiek 19, 28357 Bremen > https://www.thetaphi.de
