Without the Domain name, I don't. But a trick you can use to get deeper indexing on your pages is to change the server headers, IP address, and Expiration every week or so.
But Google bot responds to Change with "lets crawl everything" when it finds less change, it reduces the crawl. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Roch Delsalle Sent: Sunday, September 11, 2011 2:17 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Google Bot Is Your Enemy Ok, I agree that must have had an impact but how do you explain one month down to "almost" zero when my headers were sent by gae ? On Sun, Sep 11, 2011 at 9:13 PM, Brandon Wirtz <drak...@digerat.com> wrote: Your crawl rate changed because the Headers for serving changed from GAE to CF In a Week they will be back to normal. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Roch Delsalle Sent: Sunday, September 11, 2011 5:02 AM To: google-appengine@googlegroups.com Subject: [google-appengine] Re: Google Bot Is Your Enemy Here is what I noticied : http://www.d-ro.ch/2011/09/appengine-cloudflare-crawlrate Anyway you shouldn't block Googlebot that would de-index you form google. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ah8AAXrX-kMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com <mailto:google-appengine%2bunsubscr...@googlegroups.com> . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
