* [EMAIL PROTECTED] (Matthew Elvey) [Wed 16 Nov 2005, 01:56 CET]:
Still no word from google, or indication that there's anything wrong
with the robots.txt. Google's estimated hit count is going slightly up,
instead of way down.
robots.txt is about explicitly spidering your site; Google will
[EMAIL PROTECTED] (Matthew Elvey) [Wed 16 Nov 2005, 01:56 CET]:
Still no word from google, or indication that there's anything wrong
with the robots.txt. Google's estimated hit count is going slightly up,
instead of way down.
Way back in the early '90's someone came up with an
elegant
Ok, the bug is still there. Received replies from helpful folks who
missed various parts of my posts. I'll stop posting about this now; it
is indeed a bit OT. As I said in my initial post: I'm looking for a
fix, not a workaround, and again: See
http://www.google.com/webmasters/remove.html
Still no word from google, or indication that there's anything wrong
with the robots.txt. Google's estimated hit count is going slightly up,
instead of way down.
Why am I bugging NANOG with this? Well, I'm sure if Googlebot keeps
ignoring my robots.txt file, thereby hammering the server and
Hi there,
Looking at your robots.txt... are you sure that is correct?
On the sites I host.. robots.txt always has:
User-Agent: *
Disallow: /
In /htdocs or wherever the httpd root lives. Thus far it keeps the
spiders away.
GoogleSpider also will obey: NOARCHIVE, NOFOLLOW, NOINDEX placed
Still no word from google, or indication that there's anything wrong
with the robots.txt. Google's estimated hit count is going
slightly up,
instead of way down.
Why am I bugging NANOG with this? Well, I'm sure if Googlebot keeps
ignoring my robots.txt file, thereby hammering the
Why would they read/respond on NANOG to an application problem?
(seriously)
I'm waiting for the GoogleBot to respond.
- Nic.
