[google-appengine] Re: High latency

Forgot the mandatory URL : http://code.google.com/status/appengine/detail/serving-java/2012/10/16#ae-trust-detail-helloworld-get-java-latency On Tuesday, October 16, 2012 4:47:11 PM UTC+2, Francois Masurel wrote: > > Hi Google, > > It feels like it's time to buy some new servers, we are getting h

[google-appengine] Re: High latency

It feels like Google rolled out an AppEngine update in the middle of the night, and something is going wrong. There are big latency spikes for many services right around the 4AM mark. -Vinny P On Tuesday, October 16, 2012 9:47:11 AM UTC-5, Francois Masurel wrote: > > Hi Google, > > It feels li

[google-appengine] Re: High latency

Yep it was far better 6 months ago : http://code.google.com/status/appengine/detail/serving-java/2012/04/16#ae-trust-detail-helloworld-get-java-latency (80ms latency on average) That's one big annoying thing you have to deal with when your app is hosted in the cloud and not on your own machine

[google-appengine] Re: High latency

Yep, latency is way up in the last 4 hours or so on all three apps I'm monitoring, one Python, two Go. Very spiky. Hope someone's looking at it. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit http

[google-appengine] Re: High latency

Geez, the red light is on : http://code.google.com/status/appengine/detail/serving-java/2012/10/16#ae-trust-detail-helloworld-get-java-latency Y*o**u can**

[google-appengine] Re: High latency

Things seems to have improved dramatically since a few hours, good job Google. I can verify this in my logs too. Will we ever know what happened exactly?

[google-appengine] Re: High Latency Issues

Please see the attached image which is another indication that something is very wrong on app engine's end. The chart is showing the last 6 hours of Request/Second. Do you see all those little apexes in the time line? What is causing them? 6 hours ago my app was dormant with no users on the system

[google-appengine] Re: High Latency Issues

It has been almost 1 day since I posted this to the group & I am very disappointed that in that time I haven't received even 1 response from the Google Team. I am very disappointed to say the least and that's putting it mildly. I know this is a holiday weekend here in the State but c'mon, guys, pl

[google-appengine] Re: High Latency Issues

What I worry about is if I provide (and charge) for application in Apps Marketplace and my customer will receive poor service they are going to blame me not Google... Very risky.. Something must be done. On Oct 11, 1:44 pm, Jeff Schwartz wrote: > It has been almost 1 day since I posted this to th

[google-appengine] Re: High Latency Issues

What do you mean by "cold instance startup that timed out" ? Sometimes I will get the error: Request was aborted after waiting too long to attempt to service your request. This may happen sporadically when the App Engine serving cluster is under unexpectedly high or uneven load. If you see this me

[google-appengine] Re: High Latency Issues

We (appid steprep) are starting to see bursts of 30-s and 10-s timeouts occur again. E.g., 10-12 07:59AM 45.772 /_ah/queue/deferred 500 10012ms 10-12 07:59AM 43.185 /_ah/queue/deferred 500 10202ms 10-12 07:59AM 16.844 /_ah/queue/deferred 500 11799ms 10-12 07:57AM 44.693 /_ah/queue/deferred 500 293

[google-appengine] Re: High Latency Issues

Some of these bursts of non-responsiveness are dramatic. E.g., on 10-12 1.48p log time, we had over 400 of these 10-s and 30-s timeouts (i.e., just during 1.48p to 1.49p - 1 minute). I've not pasted them here because my last post was annoying enough for most, I'm sure. j On Oct 12, 9:14 am, Jaso

[google-appengine] Re: High Latency Issues

I'm getting a fair amount of these. A serious problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you should contact the App Engine tea

[google-appengine] Re: High Latency Issues

This is still occurring. For appid steprep at 2010-10-13 12.10p (log time), we just saw a burst of around 380 10-s and 30-s timeouts (mostly the 10-s variety). j On Oct 12, 11:47 pm, pdknsk wrote: > I'm getting a fair amount of these. > > A serious problem was encountered with the process that h

[google-appengine] Re: High Latency Issues

Our queues are _really_ falling behind. (appid steprep) Can anyone from Google comment on this, please? Thanks, j On Oct 13, 1:13 pm, Jason C wrote: > This is still occurring. For appid steprep at 2010-10-13 12.10p (log > time), we just saw a burst of around 380 10-s and 30-s timeouts > (mostly

[google-appengine] Re: High Latency Issues

I should clarify. Our queues are _really_ falling behind because our handlers are timing out extensively causing lots of retries, etc. And quota usage, and presumably $$ usage. j On Oct 13, 2:59 pm, Jason C wrote: > Our queues are _really_ falling behind. (appid steprep) > > Can anyone from Goog

Re: [google-appengine] Re: high latency

I noticed a spike in memcache latency earlier today as well. It's also now showing on the GAE status page: http://code.google.com/status/appengine/detail/serving/2011/07/06#ae-trust-detail-helloworld-get-latency On Wed, Jul 6, 2011 at 1:17 PM, Felippe Bueno wrote: > Still happening, > > Now

Re: [google-appengine] Re: High latency

Please fill a new production issue on the public issue tracker with your application id: http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue Thanks in advance. On Tue, Oct 16, 2012 at 5:53 PM, Francois Masurel wrote: > Yep it was far better 6 months ago : > > > htt

Re: [google-appengine] Re: High latency

Hi Johan, thanx for chiming in. I think it's not related to my specific app but all java apps as it can be seen on the related status page : http://code.google.com/status/appengine/detail/serving-java/2012/10/16#ae-trust-detail-helloworld-get-java-latency Do you have some explanation about the

Re: [google-appengine] Re: High latency

Hi Francois, I commented about the specific of your issue on the public issue tracker. On Tue, Oct 16, 2012 at 6:30 PM, Francois Masurel wrote: > Hi Johan, thanx for chiming in. > > I think it's not related to my specific app but all java apps as it can be > seen on the related status page : >

Re: [google-appengine] Re: High latency

Hi Johan, I have added a few appstats screenshots to the related issue : http://code.google.com/p/googleappengine/issues/detail?id=8292 Francois On Tuesday, October 16, 2012 6:40:38 PM UTC+2, Johan Euphrosine (Google) wrote: > > Hi Francois, > > I commented about the specific of your issue on

Re: [google-appengine] Re: High Latency Issues

Hi Peter, Yeah, the fact that our apps are running on Google's platform isn't a consideration for our users nor should it be. I am generally optimistic, however, that the Team will work these issues out but of course, in the meantime, it is upon us, the application developers, which our users will

Re: [google-appengine] Re: High Latency Issues

Here is an example of my logs: 1) 10-11 08:08AM 05.539 /account.groovy 200 218ms 265cpu_ms 13api_cpu_ms 2kb Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 ( .NET CLR 3.5.30729; .NET4.0C),gzip(gfe) 2) 10-11 08:07AM 59.775 /account.groovy 200 2401ms 1976cpu

Re: [google-appengine] Re: High Latency Issues

Seems like if you want direct one on one comunication and service guarantees, you gotta try to get into the Appengine for Business preview: http://code.google.com/appengine/business/ But then.. you have to pay. Also, from seeing your logs below.. it re

Re: [google-appengine] Re: High Latency Issues

Hi Eli, You are unfortunately incorrect as the only log entry with additional information was for #3 and it was the following: Request was aborted after waiting too long to attempt to service your request. This may happen sporadically when the App Engine serving cluster is under unexpectedly high

Re: [google-appengine] Re: High Latency Issues

I am only guessing but I imagine that 'cluster' is some form of load balancing segment that Google uses to service and route requests to the virtual servers and this error/warning is indicative of when that queue is very busy. And regardless of if it is an error or a mere warning it still resulted

Re: [google-appengine] Re: High Latency Issues

I'm getting a ton of datastore timeouts as well. app id = networkedhub The datastore operation timed out, or the data was temporarily unavailable. Waleed On Wed, Oct 13, 2010 at 2:16 PM, Jason C wrote: > I should clarify. Our queues are _really_ falling behind because our > handlers are ti

Re: [google-appengine] Re: High Latency Issues

Jason, it looks like you're seeing transaction contention, and the datastore latency spikes are exacerbating the effect of transactional backoffs, causing task queues to back up. We're aware of the datastore spikes and are monitoring the situation. -- Ikai Lan Developer Programs Engineer, Google

[google-appengine] Re: High Latency (~1500 ms)

After deploying the same code as a new version (forcing the instances to refresh), everything seems to work fine again: Total number of instances: 10 total (3 Resident) Average QPS*: 0.517 Average Latency*: 229.8 ms Average Memory: 28.4 MBytes Best regards, Fabian -- You received this message b

[google-appengine] Re: High Latency (~1500 ms)

Unfortunately, the problem is not yet solved. Once I activate a new version, the latency is perfectly fine. But over time, things get worse and painfully slow. A couple of minutes after setting a new version to active: Total number of instances Average QPS*Average Latency*Average

Re: [google-appengine] Re: High Latency (~1500 ms)

Are you seeing large pending ms values in the logs for the slow requests? What are the cold-start times for your app like? If your app is slow to spin up instances, like when you use django or Java, it could be that app engine is preferring to send requests to existing (currently occupied) instan

[google-appengine] Re: High latency with custom domain

I wondered how they were using the same hostname for all Google services (appengine, blogger, sites, etc.), which are presumably running on different servers/systenms. Now I wonder if the ghs hostname is just proxying requests to the actual servers :o( -- You received this message because yo

[google-appengine] Re: High latency with custom domain

The latency is now almost equal (actually my custom domain has a better latency :-). Could you just rent some cheap virtual machines across the globe to monitor the latency and add this data to the status page? I think that the latency is an important factor for a "fast" application. -- You recei

[google-appengine] Re: High latency with custom domain

My dedicated server has got a really bad latency again. It is really frustrating. Dedicated Server (Germany) -> custom domain 1 xxx (xxx) 0.285 ms 0.233 ms 0.228 ms 2 xe-0-1-0.cr-nashira.cgn4.hosteurope.de (80.237.129.13) 0.405 ms 0.352 ms 0.350 ms 3 xe-0-0-0.cr-polaris.fra1.hosteurope.

[google-appengine] Re: High latency with custom domain

It seems that I get routed to an european datacenter in the morning and to a datacenter in the united states in the afternoon. I cannot understand why you are ignoring this. This routing should be harmful for anyone. The users don't get the fastest surfing experience they could get and are less sat

[google-appengine] Re: High latency with custom domain

Thanks for your hint Barry. I have created a new feature request, if you are also interested in geographical distribution of your app please star it. http://code.google.com/p/googleappengine/issues/detail?id=2741 On Feb 3, 8:57 pm, Barry Hunter wrote: > Who do you use as your DNS provider. ghs.g

[google-appengine] Re: High latency in HRD app. HELP!

Hopefully someone on the Google team can take a look for you. I have a few questions and things for you to try. 1. How young are your instances? are you seeing instance churn? 2. Can you post your app settings, max idle, how is your traffic? spikey? 3. Do you see the same issues on other appids

[google-appengine] Re: High latency for no apparent reason

Since all that is known is that you have a handler that sometimes takes a very long time under certain conditions, and it's not RPC related, you'd have to post the code from the handler in order for the issue to be investigated. If you're adverse to posting your application code on a public for

[google-appengine] Re: High latency for no apparent reason

It would be hard to post my code here, because I'd have to include HttpFilters and the whole DAO/Cache-layer. But there's really nothing going on that's needing a lot of CPU resources. One thing I asked myself: could it be GC occupying the process, because the frequency of such slow requests is

[google-appengine] Re: High latency for no apparent reason

GC pauses could very well be the culprit. What do the Utilization and Memory Usage graphs look like at the time of the slow request? On Monday, November 28, 2016 at 1:21:20 AM UTC-5, Thomas Baldauf wrote: > > It would be hard to post my code here, because I'd have to include > HttpFilters and th

[google-appengine] Re: High latency for no apparent reason

CPU megacycles went up from 1300 to 2 and down again, so this is really suspicious. At the same time, memory usage was constantly at about 215 MB and only went up slightly (a few MB) in the next minutes. Any ideas? Am Dienstag, 29. November 2016 19:55:31 UTC+1 schrieb Adam (Cloud Platform

[google-appengine] Re: High latency for no apparent reason

This seems consistent with a GC pause. The size of the allocated heap would not necessarily be reduced to show a drop in memory usage after a GC. If the issue was due to the shared core not being available the Utilization would not have shown a spike. On the standard runtime you don't have the

[google-appengine] Re: High latency for no apparent reason

Ok, thank you! I'll try to optimize my code to produce less objects where possible. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsub

[google-appengine] Re: High latency for no apparent reason

No problem, glad I could help! On Sunday, December 4, 2016 at 6:27:04 AM UTC-5, Thomas Baldauf wrote: > > Ok, thank you! I'll try to optimize my code to produce less objects where > possible. -- You received this message because you are subscribed to the Google Groups "Google App Engine" grou

[google-appengine] Re: High latency for no apparent reason

Well, it happened again, and it was a period of very high latency spikes between 12pm and 8pm (US/central) as you can see in the Stackdriver graphs below. The strange thing is that high latency stayed high after 5pm when requests/s dropped to almost 0. Can you please investigate, my paying custo

[google-appengine] Re: High latency when using custom domain

Hello Taylor, Currently, custom domains have a higher latency due to architectural reasons in both australia-southeast1 and asia-northeast1 regions. However, there is already an effort underway to make it similar in terms of latency to the appspot.com. You may follow up, or comment about it

[google-appengine] Re: High latency when using custom domain

Thanks Fady, my team and I will be watching the progress of this issue closely. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr..

[google-appengine] Re: High latency/Admin interface timing out?

Seems like the entire app engine is down. Service has been somewhat random lately... :( On May 14, 3:01 pm, gwstuff wrote: > Hi, > > I can't log into the LOGS section of the admin interface of my app + > datastore-heavy requests appear to be timing out. Anyone else facing > this issue? > > Error:

[google-appengine] Re: High latency/Admin interface timing out?

Our apps appear to be running OK, but the whole admin UI is out, as is the GAE status page (!!) Would love to hear from Google on what's going on... J On May 15, 1:01 am, gwstuff wrote: > Hi, > > I can't log into the LOGS section of the admin interface of my app + > datastore-heavy requests app

[google-appengine] Re: High latency/Admin interface timing out?

Thanks Bay and Waleed, for confirming that this is not only my problem. After hitting refresh a few dozen times I managed to enter the logs and saw that a large fraction (maybe 50%+) of my requests are timing out. Historically this number has tended to be well under 1%. On May 14, 9:04 am, Waleed

[google-appengine] Re: High latency/Admin interface timing out?

Likewise, our apps are now more responsive too. Let's wait and see if it stays this way. On May 14, 9:22 am, Waleed Abdulla wrote: > My app seems to be doing better now. But I do have A LOT of errors in my > error logs :( > > > > > > On Fri, May 14, 2010 at 6:11 AM, gwstuff wrote: > > Thanks Bay

Re: [google-appengine] Re: High latency in HRD app. HELP!

Hi Mike, Thanks for your answer. 1) The ages of the instances span from 8 minutes to 30 minutes. What do you mean by instance churn? 2) Idle instances: 1 - 6 Pending latency: Automatic - Automatic Traffic: spikey (I don't think it's relevant, since yesterday was as spikey as today, but latency was

Re: [google-appengine] Re: High latency in HRD app. HELP!

Now the app is fine. Until next time... On Wed, Apr 11, 2012 at 5:30 PM, Nicanor Babula wrote: > Hi Mike, > > Thanks for your answer. > 1) The ages of the instances span from 8 minutes to 30 minutes. What do > you mean by instance churn? > 2) > Idle instances: 1 - 6 > Pending latency: Automatic -

Re: [google-appengine] Re: High latency in HRD app. HELP!

On Wed, Apr 11, 2012 at 11:30 AM, Nicanor Babula wrote: > Hi Mike, > > Thanks for your answer. > 1) The ages of the instances span from 8 minutes to 30 minutes. What do you > mean by instance churn? > 2) > Idle instances: 1 - 6 > Pending latency: Automatic - Automatic I presume you mean you have

Re: [google-appengine] Re: High latency in HRD app. HELP!

Your app will run well if you use auto for max idle instances, but you should not have to do that and it is expensive. When things are running like they should you can tune it based on your traffic. Ours is very smooth so we can safely set it to a fairly low setting and it runs well. We leav

Re: [google-appengine] Re: High latency in HRD app. HELP!

Thanks for your answer. Yes, we have been experiencing instance churn. I suppose your app is related to webfilings.com. Have you ever experienced problems like this one? Your app is under a premier account? You are free not to answer the questions. I am asking because it is the first time I go in p

Re: [google-appengine] Re: High latency/Admin interface timing out?

My app seems to be doing better now. But I do have A LOT of errors in my error logs :( On Fri, May 14, 2010 at 6:11 AM, gwstuff wrote: > Thanks Bay and Waleed, for confirming that this is not only my > problem. After hitting refresh a few dozen times I managed to enter > the logs and saw that

[google-appengine] Re: High latency trace requests to non-existent versions

Google Groups is meant for general product discussions and not for technical support. I highly recommend you report this in a Google Issue Tracker . Just make sure you provide more information such as your project ID, and the timestamps (wit

[google-appengine] Re: High latency trace requests to non-existent versions

Morgan from the Stack Driver team reached out to me about the "empty" URIs. This was a known bug. I'm still curious as the best approach to prevent traffic to non-existent versions. I'll file a ticket. Thank you. On Tuesday, April 4, 2017 at 10:02:43 AM UTC-5, Jordan (Cloud Platform Support)

[google-appengine] Re: High latency for outgoing request on 1 on 5 requests

Hi, Would you be able to kindly provide the timeout error message you are receiving (screenshot should work as well)? This will help to better visual and diagnose as to what could have enforced high latency on your application. Also, as you have submitted your question March 1st; would you be

Re: [google-appengine] Re: High latency for outgoing request on 1 on 5 requests

Hi, The issue resolved itself at the end of day. The error we were having was java.net.SocketTimeoutException: Read timed out. Regards, On Wed, Mar 3, 2021 at 9:20 PM 'jlibera' via Google App Engine < google-appengine@googlegroups.com> wrote: > Hi, > > Would you be able to kindly provide the ti

[google-appengine] Re: high latency in urlfetch when calling an external API with google app engine

Hey ssmcookiemonster, When debugging latency of a request, it's best to only surround the request action itself with timing code, so you should profile the time to execute the following two lines only: Future future = fetcher.fetchAsync(url); HTTPResponse responseHttp = future.get(); This will

Re: [google-appengine] Re: high latency in urlfetch when calling an external API with google app engine

Dear Nick, Thank you for your response. We measured the latency as you suggested and the time is almost the same. The instance is in Europe and we are doing tests from Madrid and from London. Here is the problem we see. When we call to the API from a normal computer it takes around 300ms, however

Re: [google-appengine] Re: high latency in urlfetch when calling an external API with google app engine

Some further advice I can give you: If you check your App Engine logs, you can find that the two fields CPU milliseconds and Wall-clock milliseconds represent different metrics . CPU milliseconds is a measure of h