Chris Schneider wrote:
Gang,
I had a webmaster complain that our crawler was following his <form action> links.
Although he admits that his use of the GET method is a bit unorthodox, he feels strongly
that form submissions with input fields shouldn't be followed by crawlers. Would it make
sense to modify the HTML parser so that it checked to see whether such input fields exist
before following <form action> links?
I read through your email exchange, and setting aside all emotional
content I think this is a valid request - indeed, as far as I can tell
other major crawlers don't follow these links. We could either remove
this, or make it optional (default not to use them).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
