If I specify a custom user agent for wget, eg "MyBot 1.0 (info@mybot...)" Will wget check this in robots.txt as well, if the bot was banned, or only the general robot exclusions? Does wget check if "MyBot" is allowed to crawl? If not, this would be a nice feature. If yes, it would be great to include this info in the robots overview here https://www.gnu.org/software/wget
I originally posted this question here , but then I found this list http://stackoverflow.com/questions/24316018/does-wget-check-if-specified-user-agent-is-allowed-in-robots-txt -- Gyuri 274 44 98 06 30 5888 744
