On Sun, Dec 23, 2001 at 09:17:54PM +0100, Russell Coker wrote: > > I wasn't aware that there was any format to robots.txt, I thought that the > mere presense of such a file would prevent robots from visiting.
Here is an example of my robots.txt User-agent: * Disallow: /webalizer/ Disallow: contacts.txt Disallow: /dl/ > As for rule-abiding spiders, such programs will not download files ending in > .wav, .mp3, .gz, .tgz, or .zip so I won't even see them. > > That's why I usually don't even notice responsible web spiders such as google > when browsing my web logs! Hmm, I have had spiders grab .tgz's from me before, but not anymore. User-agent can be set to a specific spider agent-name, or * for all spiders. -- Nick Jennings -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]