If you've been reading about my troubles with this issue in the past, you're going to laugh at my suggestion:
Use CloudFlare. CF's IP blocks are apparently whitelisted by Google now and won't trip Google's alarms. You can disable CF's threat monitoring and response system - and even better, you get metrics so that you have some idea when/why it's being tripped when it is enabled. This seems like a silly way of routing around Google's undocumented and unwanted "service", but it should get the job done. Client -> Client's Proxy -> CF -> GAE Jeff On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren <pe...@treehouselogic.com> wrote: > I see posts about this issue going back years, so sorry if I'm kicking a > dead horse, but I haven't been able to find any resolution. > > (I’ve posted this message twice on a new account, once 5 days ago and once 3 > days ago, and neither message has actually made it into the forum. So I’m > trying my old account. Sorry if this post ends up getting duplicated.) > > We have a paid app on app engine we've been using to serve a commercial web > app for 3 years. The app is mapped to a custom domain via Google Apps. I > think that’s the crux here. > > We have one application that serves different content for different clients. > Each of our clients has reverse proxy set up on their web server to fetch > the content from our custom domain on app engine. We use reverse proxy > simply to mask our domain to the clients' domains. There is no caching, and > the reverse proxy is Apache2 with out of the box configuration. > > On March 26, after 2 years of happily serving content to a particular > client's server, Google for some reason decided that this server was > violating its Terms of Service and started denying content to that client's > reverse proxy, redirecting users to the www.google.com/sorry/misc page with > the message that: "Our systems have detected unusual traffic from your > computer network." This of course caused our application to be totally > unusable. We sent requests to Google for more information and heard nothing. > The next day App Engine decided that particular server was ok again and > resumed serving our content to the problem server. > > Then again on March 30 Google decided to ban this particular server. > > Our app is very low volume, averaging about .05 requests/second. There were > no traffic spikes that day. There were no configuration changes to the > reverse proxy or any of our infrastructure. > > The only information I can find on the issue is here: > http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640&rd=1. > > That page suggests that the client's server was doing one of these things: > > • Sending automated queries > • Using software that sends queries to Google to determine how a website > or webpage ranks on Google for various queries > • 'Meta searching' Google > • Performing 'offline' searches on Google > > I could find no evidence of any requests being sent to Google search. There > were open requests to one of Google's nameservers, presumably to look up our > app's ip from its Google Apps custom domain. Surely that isn't a violation > of Terms of Service. We found no malware on the machine. So at this point we > have no idea why Google stopped serving the content to that particular > server, or why it resumed service. Additionally all our other clients' > reverse proxies continued to work fine. There was even another reverse proxy > successfully fetching the same content that Google was denying to the other > proxy. > > Switching to the yyy.appspot.com domain from our custom domain seems to fix > the problem, so I really suspect the problem is with the domain mapping. > > I sent a support request to Google Apps, and of course they said they > couldn’t look into it, stating: “You are correct that the custom domain > mapping is created in the Google Apps Control Panel and is handled there > however any issues with the mapping of Google App Engine apps needs to be > investigated and supported by the App Engine team.” > > So I’m left wondering why Google has denied requests from this particular > server after 2 years when nothing has changed. And yet Google continues to > happily serve our other clients who are using the exact same proxy settings > on other machines. > > Searching through previous posts, the best information I can gather is that > maybe our proxies headers are malformed and Google doesn't like them. Why > would Google randomly complain after 2 years of happily serving content to > this same proxy with the same headers? > > Previous posts described this problem as a landmine, where stepping in the > wrong place can trigger it. Seems more like a surprise missile attack to me > because we were simply walking the same path we'd walked every day for 2 > years when everything blew up. > > Obviously this is totally unacceptable. We can't very well offer a > commercial service to clients with the caveat that it might blow up at any > time, and we have no idea when or why. > > I also don't understand the connection between Google Search's Terms of > Service and my paid App Engine app. Why does Google deny service to my paid > application when it thinks some machine is violating its search policies??? > Even if that machine were violating its search policies, if I want to serve > content to the violating machine from my totally un-Google-search-related > web app, I should be able to. Granted a DOS situation could be a valid > reason for denying service to my app engine app, but violating Google's > search policies is totally unrelated to my app engine app, and I should be > able to serve content from my paid application to whomever I want. > > Can Google or anyone here on the forum shed some light on why this might > have happened and what I can do to prevent it? Will turning on PageSpeed > help, since requests will presumably be served mainly by edge caches and not > hit the app engine app as frequently? > > This issue has been around for years and clearly is still a huge problem. > I'm sure many app engine users would love some transparency on this issue. > > Thanks for any help, > Peter > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
