[google-appengine] Re: Latency problems

[troberti]

We had something similar yesterday morning for one of our apps. Lasted for 
about 3 hours. 

Today, on a different app, I saw a lot of very slow timeouts (60+) on some 
requests handlers which caused hundreds of extra instances to be launched.. 
So something is still going on.

On Thursday, June 27, 2013 5:02:49 PM UTC+2, Kyle Baley wrote:
>
> We've been having severe latency issues for about an hour now. Haven't 
> made any changes to our system for some time. We have a couple of apps and 
> it *appears* to be affecting only one at the moment. (
> bookedin-prod.appspot.com)
>
> Is anyone else having trouble? We've got some pretty testy clients on our 
> hands at the moment and I need to tell them something.
>
>
> 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.


<>

[google-appengine] Re: Latency problems

[GregF]

Same thing here - see my post further down the list. We have two virtually 
identical apps for NZ and Australia, and this affected only the NZ one. 
Latency (even for static resurces) went up by ten times, causing lots of 
timeouts and warnings that processes "had a problem". It now seems to have 
stabilised, but latency is still higher than normal.

Instances also went through the roof, but strangely billable instances 
didn't. It would be interesting to know why not.

Final point - reportability. Props to the Appengine team that this is the 
first time I've looked for it for a couple of years, but the "Report 
production issue" button seems to have disappeared. Is a mute Google group 
the only way to communicate? Our apps are billed.

On Friday, 28 June 2013 03:02:49 UTC+12, Kyle Baley wrote:
>
> We've been having severe latency issues for about an hour now. Haven't 
> made any changes to our system for some time. We have a couple of apps and 
> it *appears* to be affecting only one at the moment. (
> bookedin-prod.appspot.com)
>
> Is anyone else having trouble? We've got some pretty testy clients on our 
> hands at the moment and I need to tell them something.
>
>
> 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[burlug]

About latency problems and reporting issues:

Recurring latency problems just past 'billing reset'-time seems to 
occur/are reported since the end of february- and guess what: not any 
reaction from Google. Probably too busy with their funny glasses.

https://code.google.com/p/googleappengine/issues/detail?id=9531&can=5&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log


Op donderdag 27 juni 2013 22:05:41 UTC+2 schreef GregF het volgende:
>
> Same thing here - see my post further down the list. We have two virtually 
> identical apps for NZ and Australia, and this affected only the NZ one. 
> Latency (even for static resurces) went up by ten times, causing lots of 
> timeouts and warnings that processes "had a problem". It now seems to have 
> stabilised, but latency is still higher than normal.
>
> Instances also went through the roof, but strangely billable instances 
> didn't. It would be interesting to know why not.
>
> Final point - reportability. Props to the Appengine team that this is the 
> first time I've looked for it for a couple of years, but the "Report 
> production issue" button seems to have disappeared. Is a mute Google group 
> the only way to communicate? Our apps are billed.
>
> On Friday, 28 June 2013 03:02:49 UTC+12, Kyle Baley wrote:
>>
>> We've been having severe latency issues for about an hour now. Haven't 
>> made any changes to our system for some time. We have a couple of apps and 
>> it *appears* to be affecting only one at the moment. (
>> bookedin-prod.appspot.com)
>>
>> Is anyone else having trouble? We've got some pretty testy clients on our 
>> hands at the moment and I need to tell them something.
>>
>>
>> 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[Bartosz Kołodziej]

We have the same problems with one of our apps.

On Thursday, June 27, 2013 5:02:49 PM UTC+2, Kyle Baley wrote:
>
> We've been having severe latency issues for about an hour now. Haven't 
> made any changes to our system for some time. We have a couple of apps and 
> it *appears* to be affecting only one at the moment. (
> bookedin-prod.appspot.com)
>
> Is anyone else having trouble? We've got some pretty testy clients on our 
> hands at the moment and I need to tell them something.
>
>
> 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[GregF]

That URL doesn't work for me - have they deleted that issue? Any more 
details would be very useful.

On Friday, 28 June 2013 08:39:21 UTC+12, burlug wrote:
>
> About latency problems and reporting issues:
>
> Recurring latency problems just past 'billing reset'-time seems to 
> occur/are reported since the end of february- and guess what: not any 
> reaction from Google. Probably too busy with their funny glasses.
>
>
> https://code.google.com/p/googleappengine/issues/detail?id=9531&can=5&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log
>
>
> Op donderdag 27 juni 2013 22:05:41 UTC+2 schreef GregF het volgende:
>>
>> Same thing here - see my post further down the list. We have two 
>> virtually identical apps for NZ and Australia, and this affected only the 
>> NZ one. Latency (even for static resurces) went up by ten times, causing 
>> lots of timeouts and warnings that processes "had a problem". It now seems 
>> to have stabilised, but latency is still higher than normal.
>>
>> Instances also went through the roof, but strangely billable instances 
>> didn't. It would be interesting to know why not.
>>
>> Final point - reportability. Props to the Appengine team that this is the 
>> first time I've looked for it for a couple of years, but the "Report 
>> production issue" button seems to have disappeared. Is a mute Google group 
>> the only way to communicate? Our apps are billed.
>>
>> On Friday, 28 June 2013 03:02:49 UTC+12, Kyle Baley wrote:
>>>
>>> We've been having severe latency issues for about an hour now. Haven't 
>>> made any changes to our system for some time. We have a couple of apps and 
>>> it *appears* to be affecting only one at the moment. (
>>> bookedin-prod.appspot.com)
>>>
>>> Is anyone else having trouble? We've got some pretty testy clients on 
>>> our hands at the moment and I need to tell them something.
>>>
>>>
>>> 
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[Aleksei Rovenski]

We are having now 20-40 instances where normally there are 2-4 for this 
load, so went up about 10 times. 
Just look at this pathetic screenshot http://clip2net.com/s/5iO2VD
And look how nicely billing instances grow http://clip2net.com/s/5iO47p while 
requests per second just a bit http://clip2net.com/s/5iO5j2
What a brilliant business model Service breaks, customers pay.
Any clue somebody is fixing this right now?

четверг, 27 июня 2013 г., 18:02:49 UTC+3 пользователь Kyle Baley написал:
>
> We've been having severe latency issues for about an hour now. Haven't 
> made any changes to our system for some time. We have a couple of apps and 
> it *appears* to be affecting only one at the moment. (
> bookedin-prod.appspot.com)
>
> Is anyone else having trouble? We've got some pretty testy clients on our 
> hands at the moment and I need to tell them something.
>
>
> 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[Aleksei Rovenski]

I take my words back, my particular case was my own fault. It was caused by 
denormalized entities that we did not clean up and they gradually became so 
big that consumed all the memory. Instances did not die as i would have 
expected, instead the latency dropped and gae spinned more instances 
somehow limiting even those that did not run out of memory yet. Its a 
shame, but gae fault was the first thing that came to my mind when the 
problem emerged, it was not like that a year ago :(

пятница, 28 июня 2013 г., 9:14:12 UTC+3 пользователь Aleksei Rovenski 
написал:
>
> We are having now 20-40 instances where normally there are 2-4 for this 
> load, so went up about 10 times. 
> Just look at this pathetic screenshot http://clip2net.com/s/5iO2VD
> And look how nicely billing instances grow http://clip2net.com/s/5iO47p while 
> requests per second just a bit http://clip2net.com/s/5iO5j2
> What a brilliant business model Service breaks, customers pay.
> Any clue somebody is fixing this right now?
>
> четверг, 27 июня 2013 г., 18:02:49 UTC+3 пользователь Kyle Baley написал:
>>
>> We've been having severe latency issues for about an hour now. Haven't 
>> made any changes to our system for some time. We have a couple of apps and 
>> it *appears* to be affecting only one at the moment. (
>> bookedin-prod.appspot.com)
>>
>> Is anyone else having trouble? We've got some pretty testy clients on our 
>> hands at the moment and I need to tell them something.
>>
>>
>> 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[jon]

This happened to me once. Latency went up significantly and so did my bill. I 
challenged it but got a response along the lines of "that's just the way it 
is". I'm a long time GAE user and I've gone from really liking GAE to someone 
who constantly worries about downtime/latency and worse yet, paying more for it.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[opg]

I encountered the same issue yesterday,
Crazy response time of 30k-60kMS for basic requests.
The even more troubling issue is that many of the requests are incoming 
phone calls the comes from twillio, and when my app doesn't respond in 10s 
the customers hear "an application error has occurred" because that's the 
twillio timeout for a request.
That makes me look very very bad in front of my customers !! Moreover they 
aren't able to perform what they wish for which is a critical issue !!! i'm 
thinking seriously of migrating from app engine if this kind of downtime 
will continue.
I am very disapointed from the GAE team, ok there is no free support, but 
at least respond in the group or stack overflow about such critical issues 
 i definitely consider this as a downtime, denying that it's like 
saying that a car with a flat tire is a functional car, ok you can drive 
with it very very slow, but it's still doesn't make it a functional car 
!

GAE team, please respect your customers and respond with some feedback, 
estimated time when this issue will be fixed, or otherwise i believe many 
other customer will also start looking fo other options (and there are 
plenty of them...)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Latency problems

[Kyle Baley]

Not sure if it's related but we've seen a significant spike in the number 
of front-end instance hours used by the system. More than one might 
attribute to traffic growth. It's jumped about 60-90% since the 25th. I.e. 
it went from 25-30 hours/day to 40-60 hours/day on average. Not gradually, 
either.

Is there any way to get someone to look at this to at least identify if the 
problem is with Google or us? That's not meant to sound snarky, I would 
honestly like to know.


On Wednesday, July 3, 2013 10:05:39 AM UTC+2, o...@haitov.com wrote:
>
> I encountered the same issue yesterday,
> Crazy response time of 30k-60kMS for basic requests.
> The even more troubling issue is that many of the requests are incoming 
> phone calls the comes from twillio, and when my app doesn't respond in 10s 
> the customers hear "an application error has occurred" because that's the 
> twillio timeout for a request.
> That makes me look very very bad in front of my customers !! Moreover they 
> aren't able to perform what they wish for which is a critical issue !!! i'm 
> thinking seriously of migrating from app engine if this kind of downtime 
> will continue.
> I am very disapointed from the GAE team, ok there is no free support, but 
> at least respond in the group or stack overflow about such critical issues 
>  i definitely consider this as a downtime, denying that it's like 
> saying that a car with a flat tire is a functional car, ok you can drive 
> with it very very slow, but it's still doesn't make it a functional car 
> !
>
> GAE team, please respect your customers and respond with some feedback, 
> estimated time when this issue will be fixed, or otherwise i believe many 
> other customer will also start looking fo other options (and there are 
> plenty of them...)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Latency problems

[Kyle Baley]

Ya, the list of recent downtime
notificationsdoesn't
inspire a lot of confidence but this is the first time it's
affected us dramatically in some time. In any case, we seem to be more
stable now for about an hour so fingers are crossed. At the very least, I
can rest assured that system reliability is a top priority at
Google
.


On Thu, Jun 27, 2013 at 11:35 AM, troberti  wrote:

> We had something similar yesterday morning for one of our apps. Lasted for
> about 3 hours.
>
> Today, on a different app, I saw a lot of very slow timeouts (60+) on some
> requests handlers which caused hundreds of extra instances to be launched..
> So something is still going on.
>
> On Thursday, June 27, 2013 5:02:49 PM UTC+2, Kyle Baley wrote:
>>
>> We've been having severe latency issues for about an hour now. Haven't
>> made any changes to our system for some time. We have a couple of apps and
>> it *appears* to be affecting only one at the moment. (
>> bookedin-prod.appspot.com)
>>
>> Is anyone else having trouble? We've got some pretty testy clients on our
>> hands at the moment and I need to tell them something.
>>
>>
>> 
>>
>>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/60Q1aAlAx_I/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Latency problems

[Francois Masurel]

Yep, you're right, they should at least change their automated message 
template from time to time ;-)


On Thursday, June 27, 2013 5:48:10 PM UTC+2, Kyle Baley wrote:
>
> Ya, the list of recent downtime 
> notificationsdoesn't
>  inspire a lot of confidence but this is the first time it's 
> affected us dramatically in some time. In any case, we seem to be more 
> stable now for about an hour so fingers are crossed. At the very least, I 
> can rest assured that system reliability is a top priority at 
> Google
> .
>
>
> On Thu, Jun 27, 2013 at 11:35 AM, troberti 
> > wrote:
>
>> We had something similar yesterday morning for one of our apps. Lasted 
>> for about 3 hours. 
>>
>> Today, on a different app, I saw a lot of very slow timeouts (60+) on 
>> some requests handlers which caused hundreds of extra instances to be 
>> launched.. So something is still going on.
>>
>> On Thursday, June 27, 2013 5:02:49 PM UTC+2, Kyle Baley wrote:
>>>
>>> We've been having severe latency issues for about an hour now. Haven't 
>>> made any changes to our system for some time. We have a couple of apps and 
>>> it *appears* to be affecting only one at the moment. (
>>> bookedin-prod.appspot.com)
>>>
>>> Is anyone else having trouble? We've got some pretty testy clients on 
>>> our hands at the moment and I need to tell them something.
>>>
>>>
>>> 
>>>
>>>  -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "Google App Engine" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/google-appengine/60Q1aAlAx_I/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to 
>> google-appengi...@googlegroups.com .
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Latency problems

[Pravanjan Niranjan]

I had the same problem 2 days back .


On Fri, Jun 28, 2013 at 11:44 AM, Aleksei Rovenski <
aleksei.roven...@gmail.com> wrote:

> We are having now 20-40 instances where normally there are 2-4 for this
> load, so went up about 10 times.
> Just look at this pathetic screenshot http://clip2net.com/s/5iO2VD
> And look how nicely billing instances grow http://clip2net.com/s/5iO47p while
> requests per second just a bit http://clip2net.com/s/5iO5j2
> What a brilliant business model Service breaks, customers pay.
> Any clue somebody is fixing this right now?
>
> четверг, 27 июня 2013 г., 18:02:49 UTC+3 пользователь Kyle Baley написал:
>
>> We've been having severe latency issues for about an hour now. Haven't
>> made any changes to our system for some time. We have a couple of apps and
>> it *appears* to be affecting only one at the moment. (
>> bookedin-prod.appspot.com)
>>
>> Is anyone else having trouble? We've got some pretty testy clients on our
>> hands at the moment and I need to tell them something.
>>
>>
>> 
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
Thanks,
Pravanjan
Dev *formcreator *

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.