Shawn wrote: > Hi, I have been trying to figure out a way to limit the massive amount > of bandwidth that search bots (Googlebot/2.1) consume daily from my > website. My problem is that I am running Apache::ASP and about 90% of > the site is dynamic content, links such as product.htm?id=100. The > dynamic content gets changed quite a bit so I don’t want to use any > caching for regular users, but it would be fine for the bots to use a > cached copy for a month or so. The solution I came up with is manually > modifying the headers to keeping sending back 304 HTTP_NOT_MODIFIED for > a month before allowing new content to be served up to only search bots > and not to regular web browsers. Can anyone tell me if there are some > problems you for see with doing something like this? I have only tested > this on a dev server and was just wondering if anyone else had this > problem or any suggestions they might have.
Set up Apache to respond to a 'If-Modified-Since somedate' request with 304 response containing the whole page. To be honest I havent't set it up, as I don't know the way to do that in httpd.conf over all host as a default way to respind. The only way I did it is via the request header negotiation in mod_perl. Best Regards, Christian -- Reporting bugs: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
