Mingchun Zhao created CONNECTORS-1748: -----------------------------------------
Summary: Add "User-Agent platform" option for crawling mobile websites Key: CONNECTORS-1748 URL: https://issues.apache.org/jira/browse/CONNECTORS-1748 Project: ManifoldCF Issue Type: New Feature Reporter: Mingchun Zhao Assignee: Mingchun Zhao Hi [~daddywri] , Allow me propose a new feature for crawling mobile sites which restrict access to content based on User-Agent header. Actually, Our customer's mobile website was failing to crawl because access was restricted based on whether the User-Agent request header includes the mobile info. For this reason, we added the "User-Agent platform" option to the new "Request Headers" tab on the web repository connector page so that this mobile website doesn't fail to crawl. -- This message was sent by Atlassian Jira (v8.20.10#820010)