In "lynx-dev Non-interactive lynx"
[17/Mar/2001Sat 13:13:00]
Ilya Zakharevich wrote:
> Re prohibiting lynx from visiting sites due to misusage of
> unattended-operation mode. What about prefixing "Non-interactive " to
> the default user-agent string for non-interractive robot-like runs of
> lynx?
I think it would still provoke those who spend time and consideration
on which of their files have;
<META NAME="robots" CONTENT="all/none/nofollow/noindex">
and so forth. Also bear in mind that no robot can read copyright
notices in the body of a page.
Just wondered: how easy/hard would it be to make Lynx obey robot
exclusion protocols in non-interactive mode? This is also done
with HTTP headers?
Patrick
<mailto:[EMAIL PROTECTED]>
<http://www.island.net/~pboylan/>
; To UNSUBSCRIBE: Send "unsubscribe lynx-dev" to [EMAIL PROTECTED]
