[
https://issues.apache.org/jira/browse/FLINK-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692292#comment-16692292
]
Ken Krugler commented on FLINK-9541:
------------------------------------
Hi [~fhueske] - thanks for the input, some comments below.
* In order to avoid crawling flink-docs-master, we'd have to add a Disallow
entry to the robots.txt file at ci.apache.org. Which is doable, yes, though I
imagine we might get some push-back from the ops team to avoid having lots of
groups editing the same file.
* Yes, we could update the sitemap to adjust weights with each new release.
I'm wondering about the easiest way to build the sitemap as part of the release
process, as it needs both the (rendered) markdown files and the generated
JavaDoc files.
* At some point you'll want to remove older versions of the documents. But
anyone who isn't upgrading will then have problems, so I imagine you want to be
pretty conservative. E.g. currently you've got the 1.0 - 1.3 versions, but
seems like everyone should be on 1.4 at least, yes?
* And to cover [~Zentol]'s point, it would be better to figure out how (if
possible) to have the httpd server return the canonical link info for the
current release, so that you don't have to edit all of the older release files
(and this would work for JavaDocs too).
> Add robots.txt and sitemap.xml to Flink website
> -----------------------------------------------
>
> Key: FLINK-9541
> URL: https://issues.apache.org/jira/browse/FLINK-9541
> Project: Flink
> Issue Type: Improvement
> Components: Project Website
> Reporter: Fabian Hueske
> Priority: Major
>
> From the [dev mailing
> list|https://lists.apache.org/thread.html/71ce1bfbed1cf5f0069b27a46df1cd4dccbe8abefa75ac85601b088b@%3Cdev.flink.apache.org%3E]:
> {quote}
> It would help to add a sitemap (and the robots.txt required to reference it)
> for flink.apache.org and ci.apache.org (for /projects/flink)
> You can see what Tomcat did along these lines -
> http://tomcat.apache.org/robots.txt references
> http://tomcat.apache.org/sitemap.xml, which is a sitemap index file pointing
> to http://tomcat.apache.org/sitemap-main.xml
> By doing this, you can emphasize more recent versions of docs. There are
> other benefits, but reducing poor Google search results (to me) is the
> biggest win.
> E.g. https://www.google.com/search?q=flink+reducingstate
> <https://www.google.com/search?q=flink+reducingstate> (search on flink
> reducing state) shows the 1.3 Javadocs (hit #1), master (1.6-SNAPSHOT)
> Javadocs (hit #2), and then many pages of other results.
> Whereas the Javadocs for 1.5
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/api/java/org/apache/flink/api/common/state/ReducingState.html>
> and 1.4
> <https://ci.apache.org/projects/flink/flink-docs-release-1.4/api/java/org/apache/flink/api/common/state/ReducingState.html>
> are nowhere to be found.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
