Julien Massiera created CONNECTORS-1657:
-------------------------------------------
Summary: Web connector - Handle sitemap instruction in robot.txt
Key: CONNECTORS-1657
URL: https://issues.apache.org/jira/browse/CONNECTORS-1657
Project: ManifoldCF
Issue Type: Improvement
Components: Web connector
Affects Versions: ManifoldCF 2.17
Reporter: Julien Massiera
Currently the web connector does not understand when the robot.txt file points
a sitemap. As an example, for the site
[https://www.persee.fr,|https://www.persee.fr%2C/] in the simple history one
can find the following error:
Unknown robots.txt line: 'Sitemap: [https://www.persee.fr/sitemap.xml']
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
