I am not able to crawl these websites. They do not have robots.txt file. Can any one suggest a solution for this
https://www.cmde.org.cn/ https://www.bfarm.de/EN/Home/_node.html Thanks and Regards Raj Chidara
I am not able to crawl these websites. They do not have robots.txt file. Can any one suggest a solution for this
https://www.cmde.org.cn/ https://www.bfarm.de/EN/Home/_node.html Thanks and Regards Raj Chidara