[
https://issues.apache.org/jira/browse/CONNECTORS-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219054#comment-17219054
]
Karl Wright commented on CONNECTORS-1657:
-----------------------------------------
This is a warning only:
{code}
Logging.connectors.warn("Web: Unknown robots.txt line from '"+hostName+"':
'"+problemLine+"'");
{code}
No problems are caused when the robots.txt line is found.
> Web connector - Handle sitemap instruction in robot.txt
> -------------------------------------------------------
>
> Key: CONNECTORS-1657
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1657
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
> Affects Versions: ManifoldCF 2.17
> Reporter: Julien Massiera
> Priority: Major
>
> Currently the web connector does not understand when the robot.txt file
> points a sitemap. As an example, for the site
> [https://www.persee.fr,|https://www.persee.fr%2C/] in the simple history one
> can find the following error:
> Unknown robots.txt line: 'Sitemap: [https://www.persee.fr/sitemap.xml']
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
