Thanks Jon, After reviewing your response ... I believe this fix is for a Apache server? I am running Standalone Tomcat. It appears I need to somehow log the user-agent and possibly referrer to my default access log. Is this correct?
If this is the case, does anyone know how to do this with Tomcat log4j? Here is my current properties file for logging: log4j.rootCategory=INFO, R log4j.appender.R=org.apache.log4j.RollingFileAppender log4j.appender.R.File=C:/Tomcat5.5/logs/unique.log log4j.appender.R.MaxFileSize=1500KB log4j.appender.R.MaxBackupIndex=10 log4j.appender.R.layout=org.apache.log4j.PatternLayout log4j.appender.R.layout.ConversionPattern=%d - %5p (%C:%L) - %m%n log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhos t][/unique]=INFO, R -----Original Message----- From: Jon Wingfield [mailto:[EMAIL PROTECTED] Sent: Monday, March 27, 2006 8:15 AM To: Tomcat Users List Subject: Re: Distinguish between users and robots in access log? Looks like you are running the default access log settings. For logging of user-agent and referer (when given) try: CustomLog logs/access_log combined http://httpd.apache.org/docs/1.3/logs.html#accesslog HTH, Jon Scott Purcell wrote: > I really would like to find out how my (and if my site) is being > indexed. I am using Tomcat 5.5 and I am running a ecommerce site. I have > had nothing but trouble getting seen in search engines, so I would like > to be able to somehow trace what pages robots are indexing. > > > > I did add a robots.txt and allow my whole domain to be indexed. So that > being said, I added an access log but by default, I cannot really tell > if I have users hitting my site, or when robots are visiting my site. > > > > Below is my access_log from Saturday. What I would like to be able to > do, is distinguish between normal users, and robots. If I can verify > that the robots are indexing my site, I would be happy, and then know I > need to work more meta. But if they are not indexing my site, then I > need to find out why. > > > > Does this make sense? Has anyone been through this before? Thanks ahead > of time, > > Regards > > Scott > > > > > > > > Access log Saturday > > 152.163.100.73 - - [26/Mar/2006:02:27:13 -0600] "GET > /unique/viewThumbProducts.do?type=category&searchString=Candles%20&%20Ar > oma&categoryID=13&pageNumber=1 HTTP/1.0" 200 17139 > > 152.163.100.72 - - [26/Mar/2006:02:27:14 -0600] "GET > /unique/includes/siteWide.css HTTP/1.0" 200 15402 > > 152.163.100.134 - - [26/Mar/2006:02:27:14 -0600] "GET > /unique/images/head_text.jpg HTTP/1.0" 200 18661 > > 152.163.100.138 - - [26/Mar/2006:02:27:16 -0600] "GET > /unique/images/spacer_transparent.gif HTTP/1.0" 200 49 > > 152.163.100.133 - - [26/Mar/2006:02:27:16 -0600] "GET > /unique/images/browseProducts.gif HTTP/1.0" 200 659 > > 152.163.100.66 - - [26/Mar/2006:02:27:16 -0600] "GET > /unique/images/moregreatfinds.gif HTTP/1.0" 200 564 > > 152.163.100.69 - - [26/Mar/2006:02:27:16 -0600] "GET > /unique/images/searchText.gif HTTP/1.0" 200 624 > > 152.163.100.138 - - [26/Mar/2006:02:27:16 -0600] "GET > /unique/images/go.gif HTTP/1.1" 200 308 > > 152.163.100.204 - - [26/Mar/2006:02:27:16 -0600] "GET > /unique//images/thumb/UP00130.jpg HTTP/1.0" 200 10044 > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
