[
https://issues.apache.org/jira/browse/CONNECTORS-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548691#comment-13548691
]
Karl Wright commented on CONNECTORS-604:
----------------------------------------
One URL that crawls correctly in 1.0.1 and in curl but not in MCF 1.1 is:
http://lucene.jugem.jp/?eid=39 .
> Header differences from ManifoldCF 1.0.1 cause some crawls to fail
> ------------------------------------------------------------------
>
> Key: CONNECTORS-604
> URL: https://issues.apache.org/jira/browse/CONNECTORS-604
> Project: ManifoldCF
> Issue Type: Bug
> Components: Elastic Search connector, LiveLink connector, Meridio
> connector, RSS connector, SharePoint connector, Web connector, Wiki connector
> Affects Versions: ManifoldCF 1.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Priority: Blocker
> Fix For: ManifoldCF 1.1
>
>
> Certain sites that were crawlable before with MCF 1.0.1 fail with trunk.
> Analysis shows that the differences are due to the Host header (which
> includes an explicit port in trunk but not in MCF 1.0.1) and an "Accept: */*"
> header in MCF 1.0.1 that is not present in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
