mingchun-zhao opened a new pull request, #154: URL: https://github.com/apache/manifoldcf/pull/154
Allow me to propose a new feature for crawling mobile sites which restrict access to content based on User-Agent header. Actually, Our customer's mobile website was failing to crawl because access was restricted based on whether the User-Agent request header includes the mobile info. For this reason, we added the "User-Agent platform" option to the new "Request Headers" tab on the web repository connector page so that this mobile website doesn't fail to crawl. - The screenshot of "User-Agent platform" option within the new "Request Headers" tab <img width="1024" alt="Screenshot 2023-10-08 at 22 50 43" src="https://github.com/apache/manifoldcf/assets/100677090/b938a3bd-db7e-48f1-ad89-ef8353b0eba3";> - Crawling a mobile site will be failed when using the desktop User-Agent - Crawling a mobile site will be successful when using the mobile User-Agent -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@manifoldcf.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
