Hi all, Allow me to propose a new feature for crawling mobile sites that restrict access to content based on User-Agent header. Actually, Our customer's mobile website was failing to crawl because access was restricted based on whether the User-Agent request header is related to the mobile platform. For this reason, we added the "User-Agent platform" option to the new "Request Headers" tab on the web repository connector page so that this mobile website doesn't fail to crawl.
I've created a jira issue and PR if you are interested: https://issues.apache.org/jira/projects/CONNECTORS/issues/CONNECTORS-1748 https://github.com/apache/manifoldcf/pull/153 Kind regards, Mingchun
