We see the same thing.
Our app is flailing around spawning and killing instances like a meteor
flying over Russia.
Ok, maybe not exactly like a meteor. But it is spawning a lot of instances
and destroying our QoS again.
On Friday, February 15, 2013 9:51:01 AM UTC-8, Gabriel Pugliese wrote:
>
> Same issue here. We downloaded the logs and debugged it the time (PST)
> that had more requests.
> Strangely, it is very different from the Dashboard stats. In dashboard, it
> shows 320 req/s but in logs it outputs 13k req/s:
>
> ('15/Feb/2013:06:46:23', 5910),
> ('15/Feb/2013:06:45:54', 5943),
> ('15/Feb/2013:06:46:12', 5977),
> ('15/Feb/2013:06:46:11', 6011),
> ('15/Feb/2013:06:46:41', 6051),
> ('15/Feb/2013:06:45:28', 6250),
> ('15/Feb/2013:06:46:14', 6468),
> ('15/Feb/2013:06:45:30', 6544),
> ('15/Feb/2013:06:45:57', 6601),
> ('15/Feb/2013:06:46:21', 6650),
> ('15/Feb/2013:06:46:26', 6852),
> ('15/Feb/2013:06:46:28', 6959),
> ('15/Feb/2013:06:45:24', 7034),
> ('15/Feb/2013:06:46:34', 7146),
> ('15/Feb/2013:06:45:29', 7149),
> ('15/Feb/2013:06:46:27', 7269),
> ('15/Feb/2013:06:45:27', 7274),
> ('15/Feb/2013:06:46:16', 7346),
> ('15/Feb/2013:06:46:19', 7509),
> ('15/Feb/2013:06:46:29', 7634),
> ('15/Feb/2013:06:46:38', 7767),
> ('15/Feb/2013:06:45:25', 7861),
> ('15/Feb/2013:06:46:25', 7870),
> ('15/Feb/2013:06:45:26', 8668),
> ('15/Feb/2013:06:46:39', 9174),
> ('15/Feb/2013:06:46:33', 9886),
> ('15/Feb/2013:06:46:31', 10083),
> ('15/Feb/2013:06:46:37', 10110),
> ('15/Feb/2013:06:46:32', 10845),
> ('15/Feb/2013:06:46:30', 13072)
>
> *--
>
> Gabriel Pugliese
> @gabrielsapo* +55 11 9-9374-2720
> http://www.blogcloud.com.br
>
>
> On Fri, Feb 15, 2013 at 1:34 PM, husayt <hus...@gmail.com <javascript:>>wrote:
>
>> We have awful outages on our end too. Plus we can't deploy the
>> application too.
>>
>> Can somebody at google provide situation update or acknowledge that it is
>> recognised issue.
>>
>> Thanks
>>
>>
>> On Friday, 15 February 2013 15:25:54 UTC, Mos wrote:
>>
>>> Just for the record. It seems exactly the same issue that Google had
>>> last August: http://code.google.com/p/**googleappengine/issues/detail?**
>>> id=8004 <http://code.google.com/p/googleappengine/issues/detail?id=8004>
>>>
>>> The conclusion after weeks(!) of discussion (from #8004):
>>>
>>> Me
>>> ----
>>> Thanks, but can you please be a bit more specific?
>>> What does "The reliability team performed a maintenance operation, and
>>> it seems that most application are back to a normal levels of loading
>>> requests." mean?
>>> - Could this happen from time to time again?
>>> - The issue last for weeks! It costs us and other people a lot of
>>> money. Why do we need to escalate such a critical issue with a 50+
>>> mailing-thread and why isn't GAE able to monitor such bad behavior by
>>> itself?
>>>
>>> Google:
>>> -----------
>>> App Engine is a managed service, the reliability team is constantly
>>> monitoring the overall platform health and performing maintenance
>>> operation.
>>>
>>> In this case this was apparently only impacting a small number of Java
>>> applications and not the whole platform:
>>> http://code.google.com/p/**googleappengine/issues/detail?**id=8004<http://code.google.com/p/googleappengine/issues/detail?id=8004>
>>>
>>> - 20 developers starred this bug
>>> - 5 developers provided an appid
>>>
>>> Filing a production issue or contacting premier customer support are
>>> the good ways to escalate such application specific issues.
>>>
>>> Me:
>>> -----
>>> > In this case this was apparently only impacting a small number of Java
>>> > applications and not the whole platform:
>>> > - 20 developers starred this bug
>>>
>>> Please check the average starring of Java issues lately. 20 stars
>>> seems to be significant and let extrapolate that a lot more
>>> application may be affected.
>>>
>>> Nevertheless I'm pleased this issue is resolved (at least for the
>>> moment). I hope the reliability team would be able to monitor and fix
>>> such issue much faster in the future (instead of let GAE applications
>>> feel broken for weeks)
>>>
>>>
>>>
>>>
>>> On Fri, Feb 15, 2013 at 4:19 PM, Mos <mos...@googlemail.com> wrote:
>>> >
>>> > Yes I can confirm this. Issue started again 35Minutes ago. (Pingdom is
>>> my friend)
>>> >
>>> > And again: The application gets outages and we have to pay more
>>> Instance Hours because of this repeating GAE bug!
>>> >
>>> > krisen-talk @ http://www.krisentalk.de
>>> >
>>> >
>>> > On Fri, Feb 15, 2013 at 3:58 PM, vivpuri <v...@vivekpuri.com> wrote:
>>> >>
>>> >> Another round of issues. Super high latency with HTTP 500s all over.
>>> Last 3 weeks have been a nightmare with no response from AppEngine team
>>> members. With a budget of $6k per month, we expect better response and
>>> availability.
>>> >>
>>> >>
>>> >> On Tuesday, 12 February 2013 17:12:47 UTC+5:30, vivpuri wrote:
>>> >>>
>>> >>> Admin console for our app - linksalphacom - is showing 1000+
>>> instances right now. Can someone from AppEngine team look into it.
>>> >>>
>>> >>> Thank you
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> >> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to google-appengi...@**googlegroups.com.
>>> >> To post to this group, send email to google-a...@googlegroups.**com.
>>> >> Visit this group at http://groups.google.com/**
>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>.
>>>
>>>
>>> >> For more options, visit
>>> >> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>.
>>> >>
>>>
>>> >>
>>> >>
>>> >
>>> >
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengi...@googlegroups.com <javascript:>.
>> To post to this group, send email to
>> google-a...@googlegroups.com<javascript:>
>> .
>> Visit this group at http://groups.google.com/group/google-appengine?hl=en
>> .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
