Thank you for taking the time to explain the details. Jeff
On Sat, Sep 24, 2011 at 2:36 PM, Matthew Prince <matt...@cloudflare.com> wrote: > Brandon is both right and wrong. I'd suggest you take his comments with a > bit of a grain of salt since he starting a fledging CloudFlare-like service > called CDN In A Box. The short answer is: CloudFlare today will not hurt > your SEO (and in fact usually helps it fairly significantly) and provides a > high availability solution to the AppEngine SSL problem. > Here's the longer answer: > When CloudFlare first began, we did have challenges with Google's crawler. > While Brandon's reasoning may seem sound, it's actually an incorrect > diagnosis. It was a puzzle for us for a while until we learned what the > actual issue was by talking directly with the head of the Google Crawl team. > At the root of the problem is the fact that Google sets crawl velocity based > on an IP address. If multiple sites share an IP address and one of them has > an issue then Google turns down crawl velocity in order to make sure they > aren't contributing to excess load on the server that may be causing the > problem. > CloudFlare clusters multiple sites behind a pool of IP addresses. If one of > those sites has an issue, we faithfully pass through the server error > response code. Google's crawler was picking up that error response code and > turning down crawl velocity for all the sites using that IP address. As a > result, sites that weren't having issues but shared a CloudFlare IP with > sites that were had their crawl rates decreased and therefore their SEO > hurt. > Google's crawl team had seen this problem before with other major CDNs like > Akamai. The way they had dealt with it there was by detecting the CDN's > CNAME in the DNS chain and writing a special rule for the crawler. In our > case, a CloudFlare CNAME would not always appear in the DNS chain since we > may return an IP address of our proxies directly as an A Record, so the > solution for other CDNs would not work. > We worked directly with the Google crawl team, as well as the crawl teams > from other major search providers, in order to come up with a solution. > Today, there are special rules in place for CloudFlare's IP ranges that > assign the highest crawl velocity to sites using the IPs. We have an > established channel to feed new CloudFlare IPs to the crawl teams as we are > allocated them. You can see this yourself if your site is behind CloudFlare > by logging in to Google Webmaster Tools and seeing that the option to adjust > your crawl rate is no longer available. Search engines know we can handle > their maximum crawl load, so they hammer away at us -- which is great for > our users. While Brandon is correct that this was a problem before, our work > with search crawl teams turned this problem into a feature and it is part of > the reason why today being on CloudFlare can help your overall SEO. If > you're interested in learning more, I've written about this on our blog: > http://blog.cloudflare.com/cloudflare-and-seo > > And, related: > http://blog.cloudflare.com/losing-seo-link-juice-to-traditional-cdns > In terms of SSL and AppEngine, we had a number of users ask if we could > help. We spent a significant amount of time building a cloud-based solution > that allowed for custom domains to have SSL. Since we'd already built the > frontend of that, it was relatively easy for us to extend the solution to > the backend and, essentially, mask AppEngine's non-custom domain with your > own custom domain. It was minor feature for us, but we've been surprised by > how many AppEngine users have adopted it. > Today, CloudFlare powers more than 100,000 websites. We typically will > double the performance of a site and add a security layer which you can > enable or disable depending on your preferences. If there are ways in which > we can make CloudFlare better for the AppEngine community, please don't > hesitate to let us know. > Cheers, > Matthew Prince > CEO, CloudFlare > @eastdakota > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/XNFWzT0YH3gJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
