[google-appengine] "unusual traffic from your computer network"
I see this issue has come up many times over the years, so sorry if I'm beating a dead horse, but unfortunately I have been unable to find any tips or resolution for my particular problem. We host a web application on app engine. We create custom apps for different clients that all run on the same app engine code. We then use reverse proxy on these different client web servers to fetch the content from app engine. We use reverse proxy simply to mask the url to the domain of the client, not for caching. So we have a different reverse proxy for each different client. We've been successfully using app engine for 3 years. Our app is very low volume, averaging about .05 requests/second. After 2 years of successfully serving a particular client's app via reverse proxy on her server, Google decided that her machine was violating their terms of service and started redirecting to www.google.com/sorry/misc and giving the error message that: "our systems have detected unusual traffic from your computer network". Then after a day, it started working again. This meant that her application was totally unusable for a day. We were given no clues about why this happened or how it was fixed. Google hasn't responded to requests for more info. We didn't find any malware on her machine. The help info (http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640) on the error message indicates that Google will deny service when it thinks a machine is violating its terms of service, including: - Sending automated queries - Using software that sends queries to Google to determine how a website or webpage ranks on Google for various queries - 'Meta searching' Google - Performing 'offline' searches on Google We didn't find evidence that any of this was happening. I also saw 0 dos api denials in the logs. App Engine continued to serve content to our other reverse proxies without a hitch. This is a paid app, and it seems totally unacceptable to me that Google determines who can receive content from my App Engine app. Even if her machine were somehow violating Google's Terms of Service, her machine should still be apply to receive content from my App Engine app. Her machine didn't violate my terms of service. There are no spikes in traffic to our app, nothing to indicate a dos attack. The only thing I can guess from reading past posts is that for some reason Google didn't like the fact that we were reverse proxying content from App Engine. A problem with our proxy header format maybe? We use a simple Apache reverse proxy without caching. Apache forms the headers for us. Why Google decided to ban her machine after 2 years of reliable service with no change in traffic load or other infrastructure changes is beyond me. Obviously the threat of App Engine randomly deciding to stop serving content to our reverse proxies is acceptable. If we can't get some transparency on this, and some information on how to insure it doesn't happen again, we'll have to move to another host. Can anyone, Google or otherwise, give me some clues as to why this might have happened and how to prevent it in the future? Would turning on PageSpeed make a difference since the content would be served from edge caches and not App Engine itself? Thanks for any help, Peter -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] "unusual traffic from your computer network"
I see posts about this issue going back years, so sorry if I'm kicking a dead horse, but I haven't been able to find any resolution. We have a paid app on app engine we've been using to serve a commercial web app for 3 years. We have one application that serves different content for different clients. Each client has reverse proxy set up on their web server to fetch the content from our custom domain on app engine. We use reverse proxy simply to mask our domain to the clients' domains. There is no caching, and the reverse proxy is Apache2 with out of the box configuration. On March 26, after 2 years of happily serving content to a particular client's reverse proxy, Google for some reason decided that this server was violating its Terms of Service and started denying content to that client's reverse proxy, redirecting users to the www.google.com/sorry/misc page with the message that: "Our systems have detected unusual traffic from your computer network." This of course caused our application to be totally unusable. We sent requests to Google for more information and heard nothing. The next day App Engine decided that particular server was ok again and resumed serving our content to the problem server. Our app is very low volume, averaging about .05 requests/second. There were no traffic spikes that day. There were no configuration changes to the reverse proxy or any of our infrastructure. The primary information I can find on the issue is here: http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640&rd=1. That page suggests that the client's server was doing one of these things: - Sending automated queries - Using software that sends queries to Google to determine how a website or webpage ranks on Google for various queries - 'Meta searching' Google - Performing 'offline' searches on Google I could find no evidence of any requests being sent to Google search. There were open requests to one of Google's nameservers, presumably to look up our app's ip from its Google apps custom domain name. Surely that isn't a violation of Terms of Service. We found no malware on the machine. So at this point we have no idea why Google stopped serving the content to that particular server, or why it resumed service. Additionally all our other clients' reverse proxies continued to work fine. There was even another reverse proxy successfully fetching the same content that Google was denying to the other proxy. Searching through previous posts, the best information I can gather is that maybe our proxies headers are malformed and Google doesn't like them. Why would Google randomly complain after 2 years of happily serving content to this same proxy with the same headers? Previous posts described this problem as a landmine, where stepping in the wrong place can trigger it. Seems more like a surprise missile attack to me because we were simply walking the same path we'd walked every day for 2 years when everything blew up. Obviously this is totally unacceptable. We can't very well offer a commercial service to clients with the caveat that it might blow up at any time, and we have no idea when or why. I also don't understand the connection between Google Search's Terms of Service and my paid App Engine app. Why does Google deny service to my paid application when it thinks some machine is violating its search policies??? Even if that machine were violating its search policies, if I want to serve content to the violating machine from my totally un-Google-search-related web app, I should be able to. Granted a DOS situation could be a valid reason for denying service to my app engine app, but violating Google's search policies is totally unrelated to my app engine app, and I should be able to serve content from my paid application to whomever I want. Can Google or anyone here on the forum shed some light on why this might have happened and what I can do to prevent it? Will turning on PageSpeed make a difference, since presumably content would be served by edge caches and requests wouldn't hit the app engine instance all the time? This issue has been around for years and clearly is still a huge problem. It would be great to get some transparency. Thanks for any help, Peter -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] unusual traffic from your computer network
I see posts about this issue going back years, so sorry if I'm kicking a dead horse, but I haven't been able to find any resolution. (I’ve posted this message twice on a new account, once 5 days ago and once 3 days ago, and neither message has actually made it into the forum. So I’m trying my old account. Sorry if this post ends up getting duplicated.) We have a paid app on app engine we've been using to serve a commercial web app for 3 years. The app is mapped to a custom domain via Google Apps. I think that’s the crux here. We have one application that serves different content for different clients. Each of our clients has reverse proxy set up on their web server to fetch the content from our custom domain on app engine. We use reverse proxy simply to mask our domain to the clients' domains. There is no caching, and the reverse proxy is Apache2 with out of the box configuration. On March 26, after 2 years of happily serving content to a particular client's server, Google for some reason decided that this server was violating its Terms of Service and started denying content to that client's reverse proxy, redirecting users to the www.google.com/sorry/misc page with the message that: "Our systems have detected unusual traffic from your computer network." This of course caused our application to be totally unusable. We sent requests to Google for more information and heard nothing. The next day App Engine decided that particular server was ok again and resumed serving our content to the problem server. Then again on March 30 Google decided to ban this particular server. Our app is very low volume, averaging about .05 requests/second. There were no traffic spikes that day. There were no configuration changes to the reverse proxy or any of our infrastructure. The only information I can find on the issue is here: http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640&rd=1. That page suggests that the client's server was doing one of these things: •Sending automated queries •Using software that sends queries to Google to determine how a website or webpage ranks on Google for various queries •'Meta searching' Google •Performing 'offline' searches on Google I could find no evidence of any requests being sent to Google search. There were open requests to one of Google's nameservers, presumably to look up our app's ip from its Google Apps custom domain. Surely that isn't a violation of Terms of Service. We found no malware on the machine. So at this point we have no idea why Google stopped serving the content to that particular server, or why it resumed service. Additionally all our other clients' reverse proxies continued to work fine. There was even another reverse proxy successfully fetching the same content that Google was denying to the other proxy. Switching to the yyy.appspot.com domain from our custom domain seems to fix the problem, so I really suspect the problem is with the domain mapping. I sent a support request to Google Apps, and of course they said they couldn’t look into it, stating: “You are correct that the custom domain mapping is created in the Google Apps Control Panel and is handled there however any issues with the mapping of Google App Engine apps needs to be investigated and supported by the App Engine team.” So I’m left wondering why Google has denied requests from this particular server after 2 years when nothing has changed. And yet Google continues to happily serve our other clients who are using the exact same proxy settings on other machines. Searching through previous posts, the best information I can gather is that maybe our proxies headers are malformed and Google doesn't like them. Why would Google randomly complain after 2 years of happily serving content to this same proxy with the same headers? Previous posts described this problem as a landmine, where stepping in the wrong place can trigger it. Seems more like a surprise missile attack to me because we were simply walking the same path we'd walked every day for 2 years when everything blew up. Obviously this is totally unacceptable. We can't very well offer a commercial service to clients with the caveat that it might blow up at any time, and we have no idea when or why. I also don't understand the connection between Google Search's Terms of Service and my paid App Engine app. Why does Google deny service to my paid application when it thinks some machine is violating its search policies??? Even if that machine were violating its search policies, if I want to serve content to the violating machine from my totally un-Google-search-related web app, I should be able to. Granted a DOS situation could be a valid reason for denying service to my app engine app, but violating Google's search policies is totally unrelated to my app engine app, and I should be able to serve content from my paid application to whomever I want. Can Goog
Re: [google-appengine] unusual traffic from your computer network
If you've been reading about my troubles with this issue in the past, you're going to laugh at my suggestion: Use CloudFlare. CF's IP blocks are apparently whitelisted by Google now and won't trip Google's alarms. You can disable CF's threat monitoring and response system - and even better, you get metrics so that you have some idea when/why it's being tripped when it is enabled. This seems like a silly way of routing around Google's undocumented and unwanted "service", but it should get the job done. Client -> Client's Proxy -> CF -> GAE Jeff On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren wrote: > I see posts about this issue going back years, so sorry if I'm kicking a > dead horse, but I haven't been able to find any resolution. > > (I’ve posted this message twice on a new account, once 5 days ago and once 3 > days ago, and neither message has actually made it into the forum. So I’m > trying my old account. Sorry if this post ends up getting duplicated.) > > We have a paid app on app engine we've been using to serve a commercial web > app for 3 years. The app is mapped to a custom domain via Google Apps. I > think that’s the crux here. > > We have one application that serves different content for different clients. > Each of our clients has reverse proxy set up on their web server to fetch > the content from our custom domain on app engine. We use reverse proxy > simply to mask our domain to the clients' domains. There is no caching, and > the reverse proxy is Apache2 with out of the box configuration. > > On March 26, after 2 years of happily serving content to a particular > client's server, Google for some reason decided that this server was > violating its Terms of Service and started denying content to that client's > reverse proxy, redirecting users to the www.google.com/sorry/misc page with > the message that: "Our systems have detected unusual traffic from your > computer network." This of course caused our application to be totally > unusable. We sent requests to Google for more information and heard nothing. > The next day App Engine decided that particular server was ok again and > resumed serving our content to the problem server. > > Then again on March 30 Google decided to ban this particular server. > > Our app is very low volume, averaging about .05 requests/second. There were > no traffic spikes that day. There were no configuration changes to the > reverse proxy or any of our infrastructure. > > The only information I can find on the issue is here: > http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640&rd=1. > > That page suggests that the client's server was doing one of these things: > > •Sending automated queries > •Using software that sends queries to Google to determine how a website > or webpage ranks on Google for various queries > •'Meta searching' Google > •Performing 'offline' searches on Google > > I could find no evidence of any requests being sent to Google search. There > were open requests to one of Google's nameservers, presumably to look up our > app's ip from its Google Apps custom domain. Surely that isn't a violation > of Terms of Service. We found no malware on the machine. So at this point we > have no idea why Google stopped serving the content to that particular > server, or why it resumed service. Additionally all our other clients' > reverse proxies continued to work fine. There was even another reverse proxy > successfully fetching the same content that Google was denying to the other > proxy. > > Switching to the yyy.appspot.com domain from our custom domain seems to fix > the problem, so I really suspect the problem is with the domain mapping. > > I sent a support request to Google Apps, and of course they said they > couldn’t look into it, stating: “You are correct that the custom domain > mapping is created in the Google Apps Control Panel and is handled there > however any issues with the mapping of Google App Engine apps needs to be > investigated and supported by the App Engine team.” > > So I’m left wondering why Google has denied requests from this particular > server after 2 years when nothing has changed. And yet Google continues to > happily serve our other clients who are using the exact same proxy settings > on other machines. > > Searching through previous posts, the best information I can gather is that > maybe our proxies headers are malformed and Google doesn't like them. Why > would Google randomly complain after 2 years of happily serving content to > this same proxy with the same headers? > > Previous posts described this problem as a landmine, where stepping in the > wrong place can trigger it. Seems more like a surprise missile attack to me > because we were simply walking the same path we'd walked every day for 2 > years when everything blew up. > > Obviously this is totally unacceptable. We can't very well offer a > commercial service to clients with the caveat that it might blow up at any > time
Re: [google-appengine] unusual traffic from your computer network
Funny, because in the past CloudFlare was getting banned quickly for unusually high traffic. I suppose they have been whitelisted since. On Wednesday, April 3, 2013 6:05:43 PM UTC+2, Jeff Schnitzer wrote: > > If you've been reading about my troubles with this issue in the past, > you're going to laugh at my suggestion: > > Use CloudFlare. CF's IP blocks are apparently whitelisted by Google > now and won't trip Google's alarms. You can disable CF's threat > monitoring and response system - and even better, you get metrics so > that you have some idea when/why it's being tripped when it is > enabled. > > This seems like a silly way of routing around Google's undocumented > and unwanted "service", but it should get the job done. > > Client -> Client's Proxy -> CF -> GAE > > Jeff > > > On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren > > > wrote: > > I see posts about this issue going back years, so sorry if I'm kicking a > > dead horse, but I haven't been able to find any resolution. > > > > (I’ve posted this message twice on a new account, once 5 days ago and > once 3 > > days ago, and neither message has actually made it into the forum. So > I’m > > trying my old account. Sorry if this post ends up getting duplicated.) > > > > We have a paid app on app engine we've been using to serve a commercial > web > > app for 3 years. The app is mapped to a custom domain via Google Apps. I > > think that’s the crux here. > > > > We have one application that serves different content for different > clients. > > Each of our clients has reverse proxy set up on their web server to > fetch > > the content from our custom domain on app engine. We use reverse proxy > > simply to mask our domain to the clients' domains. There is no caching, > and > > the reverse proxy is Apache2 with out of the box configuration. > > > > On March 26, after 2 years of happily serving content to a particular > > client's server, Google for some reason decided that this server was > > violating its Terms of Service and started denying content to that > client's > > reverse proxy, redirecting users to the www.google.com/sorry/misc page > with > > the message that: "Our systems have detected unusual traffic from your > > computer network." This of course caused our application to be totally > > unusable. We sent requests to Google for more information and heard > nothing. > > The next day App Engine decided that particular server was ok again and > > resumed serving our content to the problem server. > > > > Then again on March 30 Google decided to ban this particular server. > > > > Our app is very low volume, averaging about .05 requests/second. There > were > > no traffic spikes that day. There were no configuration changes to the > > reverse proxy or any of our infrastructure. > > > > The only information I can find on the issue is here: > > > http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640&rd=1. > > > > > That page suggests that the client's server was doing one of these > things: > > > > •Sending automated queries > > •Using software that sends queries to Google to determine how a > website > > or webpage ranks on Google for various queries > > •'Meta searching' Google > > •Performing 'offline' searches on Google > > > > I could find no evidence of any requests being sent to Google search. > There > > were open requests to one of Google's nameservers, presumably to look up > our > > app's ip from its Google Apps custom domain. Surely that isn't a > violation > > of Terms of Service. We found no malware on the machine. So at this > point we > > have no idea why Google stopped serving the content to that particular > > server, or why it resumed service. Additionally all our other clients' > > reverse proxies continued to work fine. There was even another reverse > proxy > > successfully fetching the same content that Google was denying to the > other > > proxy. > > > > Switching to the yyy.appspot.com domain from our custom domain seems to > fix > > the problem, so I really suspect the problem is with the domain mapping. > > > > I sent a support request to Google Apps, and of course they said they > > couldn’t look into it, stating: “You are correct that the custom domain > > mapping is created in the Google Apps Control Panel and is handled there > > however any issues with the mapping of Google App Engine apps needs to > be > > investigated and supported by the App Engine team.” > > > > So I’m left wondering why Google has denied requests from this > particular > > server after 2 years when nothing has changed. And yet Google continues > to > > happily serve our other clients who are using the exact same proxy > settings > > on other machines. > > > > Searching through previous posts, the best information I can gather is > that > > maybe our proxies headers are malformed and Google doesn't like the
Re: [google-appengine] unusual traffic from your computer network
This just impacted our live site as well. Users going to domain.com.au are blocked, users going to www.domain.com.au are not. We're using cloudflare, everything seems ok there. We do have monitoring (Pingdom) which hits the naked domain, not sure if this is what triggered this. Either way, it appears to block out all users. On Thursday, April 4, 2013 3:05:43 AM UTC+11, Jeff Schnitzer wrote: > > If you've been reading about my troubles with this issue in the past, > you're going to laugh at my suggestion: > > Use CloudFlare. CF's IP blocks are apparently whitelisted by Google > now and won't trip Google's alarms. You can disable CF's threat > monitoring and response system - and even better, you get metrics so > that you have some idea when/why it's being tripped when it is > enabled. > > This seems like a silly way of routing around Google's undocumented > and unwanted "service", but it should get the job done. > > Client -> Client's Proxy -> CF -> GAE > > Jeff > > > On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren > > > wrote: > > I see posts about this issue going back years, so sorry if I'm kicking a > > dead horse, but I haven't been able to find any resolution. > > > > (I’ve posted this message twice on a new account, once 5 days ago and > once 3 > > days ago, and neither message has actually made it into the forum. So > I’m > > trying my old account. Sorry if this post ends up getting duplicated.) > > > > We have a paid app on app engine we've been using to serve a commercial > web > > app for 3 years. The app is mapped to a custom domain via Google Apps. I > > think that’s the crux here. > > > > We have one application that serves different content for different > clients. > > Each of our clients has reverse proxy set up on their web server to > fetch > > the content from our custom domain on app engine. We use reverse proxy > > simply to mask our domain to the clients' domains. There is no caching, > and > > the reverse proxy is Apache2 with out of the box configuration. > > > > On March 26, after 2 years of happily serving content to a particular > > client's server, Google for some reason decided that this server was > > violating its Terms of Service and started denying content to that > client's > > reverse proxy, redirecting users to the www.google.com/sorry/misc page > with > > the message that: "Our systems have detected unusual traffic from your > > computer network." This of course caused our application to be totally > > unusable. We sent requests to Google for more information and heard > nothing. > > The next day App Engine decided that particular server was ok again and > > resumed serving our content to the problem server. > > > > Then again on March 30 Google decided to ban this particular server. > > > > Our app is very low volume, averaging about .05 requests/second. There > were > > no traffic spikes that day. There were no configuration changes to the > > reverse proxy or any of our infrastructure. > > > > The only information I can find on the issue is here: > > > http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640&rd=1. > > > > > That page suggests that the client's server was doing one of these > things: > > > > •Sending automated queries > > •Using software that sends queries to Google to determine how a > website > > or webpage ranks on Google for various queries > > •'Meta searching' Google > > •Performing 'offline' searches on Google > > > > I could find no evidence of any requests being sent to Google search. > There > > were open requests to one of Google's nameservers, presumably to look up > our > > app's ip from its Google Apps custom domain. Surely that isn't a > violation > > of Terms of Service. We found no malware on the machine. So at this > point we > > have no idea why Google stopped serving the content to that particular > > server, or why it resumed service. Additionally all our other clients' > > reverse proxies continued to work fine. There was even another reverse > proxy > > successfully fetching the same content that Google was denying to the > other > > proxy. > > > > Switching to the yyy.appspot.com domain from our custom domain seems to > fix > > the problem, so I really suspect the problem is with the domain mapping. > > > > I sent a support request to Google Apps, and of course they said they > > couldn’t look into it, stating: “You are correct that the custom domain > > mapping is created in the Google Apps Control Panel and is handled there > > however any issues with the mapping of Google App Engine apps needs to > be > > investigated and supported by the App Engine team.” > > > > So I’m left wondering why Google has denied requests from this > particular > > server after 2 years when nothing has changed. And yet Google continues > to > > happily serve our other clients who are using the exact same proxy > se
