-Original Message-
From: Evan Platt [mailto:e...@espphotography.com]
Sent: Wednesday, July 22, 2009 10:44 PM
To: users@httpd.apache.org
Subject: Re: [us...@httpd] 404's to robots.txt?
At 06:03 PM 7/21/2009, you wrote:
More than 1 docroot / log %{Host}i ?
Ahh.. I believe
-Original Message-
From: Evan Platt [mailto:e...@espphotography.com]
Sent: Wednesday, July 22, 2009 1:56 AM
To: users@httpd.apache.org
Subject: [us...@httpd] 404's to robots.txt?
So I've noticed quite a lot of connections from web spider programs.
I've had a robots.txt
(User
At 06:03 PM 7/21/2009, you wrote:
More than 1 docroot / log %{Host}i ?
Ahh.. I believe that's it! Thank you and the others who suggested
that, I believe that's it - I have a few subdomains I rarely use...
My current httpd.conf is:
LogFormat %h %l %u %t \%r\ %s %b \%{Referer}i
So I've noticed quite a lot of connections from web spider programs.
I've had a robots.txt
(User-agent: *
Disallow: /) For a long time. But looking closer in my apache logs,
am I reading right that it's giving a 404?
65.55.106.173 - - [21/Jul/2009:09:44:43 -0700] GET /robots.txt
HTTP/1.1
Evan Platt wrote:
So I've noticed quite a lot of connections from web spider programs.
I've had a robots.txt
(User-agent: *
Disallow: /) For a long time. But looking closer in my apache logs, am
I reading right that it's giving a 404?
65.55.106.173 - - [21/Jul/2009:09:44:43 -0700] GET
Evan Platt wrote:
...
65.55.106.173 - - [21/Jul/2009:09:44:43 -0700] GET /robots.txt
HTTP/1.1 404 208 - msnbot/2.0b (+http://search.msn.com/msnbot.htm)
65.55.106.112 - - [21/Jul/2009:10:11:43 -0700] GET /robots.txt
HTTP/1.1 404 208 - msnbot/2.0b (+http://search.msn.com/msnbot.htm)
On Tue, Jul 21, 2009 at 7:56 PM, Evan Platte...@espphotography.com wrote:
So I've noticed quite a lot of connections from web spider programs. I've
had a robots.txt
(User-agent: *
Disallow: /) For a long time. But looking closer in my apache logs, am I
reading right that it's giving a 404?
