[
https://issues.apache.org/jira/browse/CONNECTORS-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219079#comment-17219079
]
Julien Massiera commented on CONNECTORS-1657:
---------------------------------------------
Yes a warning in the log but an ERROR in the simple history. We should at least
change the return code of the activity don't you agree ?
> Web connector - Handle sitemap instruction in robot.txt
> -------------------------------------------------------
>
> Key: CONNECTORS-1657
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1657
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
> Affects Versions: ManifoldCF 2.17
> Reporter: Julien Massiera
> Priority: Major
>
> Currently the web connector does not understand when the robot.txt file
> points a sitemap. As an example, for the site
> [https://www.persee.fr,|https://www.persee.fr%2C/] in the simple history one
> can find the following error:
> Unknown robots.txt line: 'Sitemap: [https://www.persee.fr/sitemap.xml']
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
