Hi Scott, look at awstats ( http://awstats.sourceforge.net ). This projects parses Apache httpd's logs and can determine who's bot and who's not. Maybe you find there some insteresting informations.
PETR On 3/27/06, Jon Wingfield <[EMAIL PROTECTED]> wrote: > Looks like you are running the default access log settings. > For logging of user-agent and referer (when given) try: > > CustomLog logs/access_log combined > > http://httpd.apache.org/docs/1.3/logs.html#accesslog > > HTH, > > Jon > > Scott Purcell wrote: > > I really would like to find out how my (and if my site) is being > > indexed. I am using Tomcat 5.5 and I am running a ecommerce site. I have > > had nothing but trouble getting seen in search engines, so I would like > > to be able to somehow trace what pages robots are indexing. > > > > > > > > I did add a robots.txt and allow my whole domain to be indexed. So that > > being said, I added an access log but by default, I cannot really tell > > if I have users hitting my site, or when robots are visiting my site. > > > > > > > > Below is my access_log from Saturday. What I would like to be able to > > do, is distinguish between normal users, and robots. If I can verify > > that the robots are indexing my site, I would be happy, and then know I > > need to work more meta. But if they are not indexing my site, then I > > need to find out why. > > > > > > > > Does this make sense? Has anyone been through this before? Thanks ahead > > of time, > > > > Regards > > > > Scott > > > > > > > > > > > > > > > > Access log Saturday > > > > 152.163.100.73 - - [26/Mar/2006:02:27:13 -0600] "GET > > /unique/viewThumbProducts.do?type=category&searchString=Candles%20&%20Ar > > oma&categoryID=13&pageNumber=1 HTTP/1.0" 200 17139 > > > > 152.163.100.72 - - [26/Mar/2006:02:27:14 -0600] "GET > > /unique/includes/siteWide.css HTTP/1.0" 200 15402 > > > > 152.163.100.134 - - [26/Mar/2006:02:27:14 -0600] "GET > > /unique/images/head_text.jpg HTTP/1.0" 200 18661 > > > > 152.163.100.138 - - [26/Mar/2006:02:27:16 -0600] "GET > > /unique/images/spacer_transparent.gif HTTP/1.0" 200 49 > > > > 152.163.100.133 - - [26/Mar/2006:02:27:16 -0600] "GET > > /unique/images/browseProducts.gif HTTP/1.0" 200 659 > > > > 152.163.100.66 - - [26/Mar/2006:02:27:16 -0600] "GET > > /unique/images/moregreatfinds.gif HTTP/1.0" 200 564 > > > > 152.163.100.69 - - [26/Mar/2006:02:27:16 -0600] "GET > > /unique/images/searchText.gif HTTP/1.0" 200 624 > > > > 152.163.100.138 - - [26/Mar/2006:02:27:16 -0600] "GET > > /unique/images/go.gif HTTP/1.1" 200 308 > > > > 152.163.100.204 - - [26/Mar/2006:02:27:16 -0600] "GET > > /unique//images/thumb/UP00130.jpg HTTP/1.0" 200 10044 > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
