Hello, Given the following robots.txt file:
User-agent: * Disallow: /folder/bob.php? ... One would expect that if wget tries to download a link to /folder/bob.php?a=1 that it would exclude it because of the robots rule line - but it doesn't (my reading of the RFC indicates that it should exclude it). If you remove the "?" from the robots rule, then it works as expected. However, this is not what you would expect - let's say you *want* the /folder/bob.php file to be downloaded, but not all the other dynamic content (eg, /folder/bob.php?a=1 and /folder/bob.php?a=100, etc). I've been searching around but cannot find an obvious answer: is this behaviour expected, or is this a bug? Any comments would be appreciated.
