> "Mary" == Mary Gardiner <[EMAIL PROTECTED]> writes:
Mary> On Thu, Oct 23, 2008, Tony Sceats wrote:
>> Can't you use robots.txt (or the modern equiv, is there anything
>> newer actually?) to stop mass indexing, perhaps point it to pages
>> you want indexed and also tell it to exclude images e
On Thu, Oct 23, 2008, Rev Simon Rumble wrote:
> You might want to look into the Crawl-delay extension to the robots.txt
> standard, which can limit by robot:
> http://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive
There's also the Sitemaps protocol, in which you can suggest how
frequently
On Thu, Oct 23, 2008, Tony Sceats wrote:
> Can't you use robots.txt (or the modern equiv, is there anything newer
> actually?) to stop mass indexing, perhaps point it to pages you want indexed
> and also tell it to exclude images etc etc?
As I understand it, robots.txt is still the way to do this.
This one time, at band camp, Peter Chubb wrote:
> I'm a little cheesed off. In the last three months, people have
> downloaded 9G per month from our website; search engines have
> downloaded 21G per month. Only Google generated significant traffic
> through search engine hits (and it downloaded
Can't you use robots.txt (or the modern equiv, is there anything newer
actually?) to stop mass indexing, perhaps point it to pages you want indexed
and also tell it to exclude images etc etc?
On Thu, Oct 23, 2008 at 10:45 AM, Peter Chubb <[EMAIL PROTECTED]>wrote:
>
> Hi,
> I'm a little cheesed o
Hi,
I'm a little cheesed off. In the last three months, people have
downloaded 9G per month from our website; search engines have
downloaded 21G per month. Only Google generated significant traffic
through search engine hits (and it downloaded less than the others,
too --- around 2G per month,
