On Thu, 26 Sep 2002 23:21:54 +0200 Christian Langreiter <[EMAIL PROTECTED]> wrote:
>> Hmm. I thought that there was not supposed to be a >>direct >> link anywhere to that archive to prevent spiders getting >> to it ... as email addresses are visible on those pages. > >Well, spiders don't get to sites if they observe >robots.txt, which >spam address harvesters most certainly don't. We had a discussion before about the rebol.org archive, and Jeff removed all links to the archive. To get the address, you have to use the link on the RT rebsite. So even a spam address harvesting robot should not get there. But I guess google had it in it's database prior to that happening. > >> I think that all dynamically created sites tend to be >> invisible to search engines. Zope is an example. > >Not exactly. Search engines have avoided URLs with query >strings for a >long time, but how would they distinguish static from >dynamic content? >They cannot, it's all bits to them (and us, for that >matter ;-). In practise they tend to be invisible. Here's an article from Paul Graham making that point http://www.paulgraham.com/mistakes.html and in my Zope site, I have this in my robots.txt to make me invisible! User-agent: * Disallow: /Shopping/ # This is an infinite virtual URL space -- Graham Chiu -- To unsubscribe from this list, please send an email to [EMAIL PROTECTED] with "unsubscribe" in the subject, without the quotes.
