magibney commented on PR #846: URL: https://github.com/apache/solr/pull/846#issuecomment-1125007411
I realize this PR is merged (and thanks!), but I have a couple of questions that follow logically on the conversation here, so: 1. Noticing that we don't have (and haven't historically had, I think) an old-style `robots.txt`, I wonder: is old-style `robots.txt` completely obviated? i.e., should we not bother having one? 2. Sitemap.xml is being generated I think, and is [present in nightlies](https://nightlies.apache.org/solr/draft-guides/solr-reference-guide-nightly/sitemap.xml). But it doesn't appear to be accessible on the main site. I think it was previously present. Is there still a purpose served by sitemap.xml? Currently the nightlies version looks like it points to all (antora) versions -- I'm not sure whether we'd want to pare down the referenced pages to make sitemap.xml a proper complement to "canonical, no-index/no-follow/no-archive" approach taken by this PR? 3. If sitemap.xml is still relevant and we want to make it accessible, I think the sitemap spec calls for sitemap.xml to be referenced from an old-style robots.txt ... I'm not aware of other/newer approaches to referencing sitemap.xml. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
