I have had trouble getting search engines to see my site. I built it with
struts, and use some tags from the index.html page to get business logic, to
finally get to my page. The url is http://www.theuniquepear.com
Anyway, upon talking to some co-workers, they suggested I watch my access log,
so I can see what files they are indexing. I thought I had the access log
turned on for the site, and see when someone hits my web site, but as far as
the searchbots go, I only see this in my logs daily.
$ cat localhost_access_log.2006-02-07.txt | less
67.15.16.30 - - [07/Feb/2006:03:44:55 -0600] "GET /robots.txt HTTP/1.0" 404 985
67.15.16.30 - - [07/Feb/2006:03:46:21 -0600] "GET / HTTP/1.0" 200 844
67.15.16.30 - - [07/Feb/2006:03:51:57 -0600] "GET /robots.txt HTTP/1.0" 404 985
62.114.208.233 - - [07/Feb/2006:03:52:42 -0600] "GET
/unique/welcome.do?OVRAW=home%20decorating%20ideas&OVKEY=home
62.114.208.233 - - [07/Feb/2006:03:52:44 -0600] "GET
/unique/includes/siteWide.css HTTP/1.1" 200 15402
62.114.208.233 - - [07/Feb/2006:03:52:44 -0600] "GET
/unique/images/header_pear.jpg HTTP/1.1" 200 11227
I see the entry for robots.txt, but I have no idea where they are going, or
what they are doing.
I turned on access log like this in the server.xml like so:
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs" prefix="localhost_access_log." suffix=".txt"
pattern="common" resolveHosts="false"/>
And that is a snippet of the log from above.
Does anyone know how to get more involved text, or can anyone tell me what the
robots.txt above is doing?
Thanks,
Scott
