I am not able to crawl these websites.  They do not have robots.txt file. Can 
any one suggest a solution for this

https://www.cmde.org.cn/ 

https://www.bfarm.de/EN/Home/_node.html


Thanks and Regards

Raj Chidara



Reply via email to