Just to add some information:
When timing the load time of my app (not from the dev server startup but
starting from the point my code is beging to load) it takes my application
less than 8 seconds to load (few cases were 5secs). So I still can't figure
out why on AppEngine it can sometimes
As far as anyone outside Google can tell, the biggest issue involves
reading files off the filesystem. This may have changed recently since
it's been a while since I (or any of the other users who commonly post
about such stuff) did any measurements, but reading files seems to be
painfully slow.
Oh, what I needed to mention is that if you're trying to time your app load
with (say) a filter, a lot of this load time elapses before your code
begins executing.
Jeff
On Fri, Apr 20, 2012 at 11:44 AM, Jeff Schnitzer j...@infohazard.orgwrote:
As far as anyone outside Google can tell, the
Do you mean Python or Java apps sitting at over 100mb after spin up?
I work on some extremely large Python apps, they can sit around ~75mb.
If your Python app is heavier than that at startup... you are
probably doing stuff in a very questionable way.
Robert
On Fri, Apr 13, 2012 at 11:02,
Thanks for the clarification, it should be part of the documentation for
AppEngine architecture.
As for my question, I believe that a Maven multi-module Spring application
has different view on loading times.
In earlier posts here it was mentioned that loading several JARs might be
an issue.
Do you have any recommendations how to reduce the amount of IO operations
during startup?
My app uses Spring and like many other Java best practices followed, my
project consists of several Maven modules which results several jars
creation.
I do not load any data from datastore or require HTTP
Woah! 30s to start your app in your local dev environment? That's nuts.
You have a mistaken perception of GAE. In nearly all respects, your local
dev environment will perform a single thread of execution faster than
production. Your local machine has dedicated CPU cores and I/O bandwidth,
all
Thanks.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/uuKWxLpFYC0J.
To post to this group, send email to google-appengine@googlegroups.com.
To
Hi Jeff
Thanks for your replay. Your answer was very important for me to start
looking to the right part of the problem.
Just to clarify :startup time is the time to start a new instance right?
How can I analyze the startup time? Where can I look for?
I installed the appstats and SpeedTracer,
Hi Jeff
Thanks for your replay. Your answer was very important for me to start
looking to the right part of the problem.
Just to clarify :startup time is the time to start a new instance right?
How can I analyze the startup time inside the server? I'm looking to the
appengine logs,
Right. The problem is the 61244ms that it takes to start your app. How
long does it normally take?
Look at past warmup requests (the ones that work) and see how long they
take. My guess is that
the number is close to 60s. If GAE gets marginally slower, it pushes you
over the edge.
As for why
My guess is that the number is close to 60s. If GAE gets marginally
slower, it pushes you over the edge.
I'll be doing a video very shortly about how to keep your Instances from
dying on startup. But one easy way to tell if this is the issue is to up
your instance size AND test that you are
FWIW, warmup requests and instance startup has been very very inconsistent
the past couple of weeks. I've had an app that usually took 2-3S show up in
the logs as DeadLineExceeded and causing our integration tests to fail when
originating from the continuous integration server and it's very
Sounds like your startup time exceeds the max (60s?) time for a request.
You need to cut down your app startup time.
Jeff
On Thu, Apr 12, 2012 at 5:00 AM, Rui Oliveira rucaso2...@gmail.com wrote:
Hi,
My appId is air-menu1. HRD.
I'm getting this kind of errors always after deploy. Usually
Hi,
Thanks for your reply.
The strange is that I have 5 modules inside my application. All of the
requests to the different modules fail at the same time. All the modules
call different functions. All my functions are very small. My database is
really small at the moment.
I don't do
Look in the logs for startup requests, and check the amount of time
they take to complete.
It's possible you have an app that normally loads in a few seconds,
but a hiccup at google extended those few seconds into a time that
extended past the deadline. This would be something to complain
about.
Just to add to Jeff's: do use Appstats too.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/vHytJH0kcDsJ.
To post to this group, send email to
Potential fix: set performance sliders to auto.
No, that doesn't help in general. I'm running from the start with auto and
having the problem.
It's still not fixed; would be nice to get some feedback from Google ?!
On Wed, Mar 14, 2012 at 3:14 AM, Mauricio Aristizabal aris...@gmail.comwrote:
We've been working on addressing the issues with HRD apps. Your experience
is probably a coincidence.
There is a very small section of apps that will have a few requests (very
small %) that will be a LOT slower.
We've scheduled a maintenance period for March 19th that will attempt to
address
Hello all. Only to share my information:
My app is doing something like this:
JS=xpto
JS_OPTOUT=xpto1
if request.cookie.has_key(optout) or request.headers.has_key(DNT)
response.write(JS_OPTOUT)
else
response.write(JS)
The normal latency is ~3ms
Now I'm seeing 13.9 ms
Earlier this morning I
In case you're keeping track of issues thinking it's generally cleared up:
I'm on HR and have noticed higher latencies (couple seconds instead of e.g.
300ms) lately and sometimes higher error rates (a few instead of 0-3).
Yesterday over about 6 hours I got a ton of 60-second requests that threw
I'm encountering the same issue with HR app. The spike got started in about
~1 hour from now.
AppID: cmsevobg
Datastore: HR
On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:
In case you're keeping track of issues thinking it's generally cleared up:
I'm on HR and have
Hello,
Same thing here, since around an hour ago:
AppID: fiveorbsgame fiveorbsgame-test
On Tue, Mar 13, 2012 at 9:59 AM, Miroslav Genov mge...@gmail.com wrote:
I'm encountering the same issue with HR app. The spike got started in
about ~1 hour from now.
AppID: cmsevobg
Datastore: HR
On
Ops my mistake, our HR id is cmsevobg-hr. Normally we get initialization
time for ~10 seconds, now the time is 38 seconds using F4 CPU model.
Here are some exception traces that might help
1.
Failed startup of context
Same here, very slow and frequent instance startups. I use the HR
datastore.
I've set idle instances to 2 F4s but I have the impression that the
scheduler prefers to start a new instance instead of using one of the
available idle ones.
--
Pieter Coucke
Onthoo BVBA
zamtam.com
Same thing the last minutes on our app (HRD, Java, Low-Traffic, one
instance, no new deployment, simple page just hitting MemCache):
Request was aborted after waiting too long to attempt to service your
request. -- User sees 500er
GAE-Team, what is going on the last days? In my opinion the
My graph showing ms/sec (attached) over last 24h. Average spikes were up
to 32 seconds, but I think all the errors were 60+-1 sec.
On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:
In case you're keeping track of issues thinking it's generally cleared up:
I'm on HR and
Hello Ikai
our app is *betscoreslive*. In our settings master/slave replication is
activated but we are NOT using the datastore at all and we have been
experiencing the DeadlineExceedExceptions and increased instance number
mentioned by the rest of the people in the discussion. Our app is only
This is very disturbing ... Our M/S app is getting higher error rates and
some instances take from 15s to 70s to start. We can't do anything about
this and even debug what is happening. If there was a issue with our code,
they should always take 70s to start! I really can't understand or think
I believe this was related to:
http://code.google.com/p/googleappengine/issues/detail?id=7130
This should now be fixed.
On Tue, Mar 13, 2012 at 9:59 AM, Miroslav Genov mge...@gmail.com wrote:
I'm encountering the same issue with HR app. The spike got started in
about ~1 hour from now.
Hi,
I believe this was also related to:
http://code.google.com/p/googleappengine/issues/detail?id=7130
And should now be fixed.
On Tue, Mar 13, 2012 at 10:02 AM, Sébastien Tromp sebastien.tr...@gmail.com
wrote:
Hello,
Same thing here, since around an hour ago:
AppID: fiveorbsgame
As Chris pointed earlier in that thread, M/S app are more vulnerable to
this kind of transient infrastructure issues because moving them around
require a maintenance period.
HRD applications are covered by the SLA, replicated around multiple
datacenter, and better distributed if we notice an
If you are not using the datastore it should be trivial to move your
application to the new HRD infrastructure.
If you don't need the same appid, just create a new HRD application and
deploy your code on it.
If you need the same appid, use the self migration tool:
What is your application id?
Feel free to open a production issue, if you want to investigate this
offthread:
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue
On Tue, Mar 13, 2012 at 11:44 AM, Mos mosa...@googlemail.com wrote:
Same thing the last minutes on
What is your application id? Did you already fill a production issue?
On Tue, Mar 13, 2012 at 3:10 PM, Jason jaso...@gmail.com wrote:
I'm using HRD and am still getting tons of 500s since early this
morning.
On Mar 13, 8:52 am, Johan Euphrosine pro...@google.com wrote:
I believe this was
What is your application id?
krisen-talk(www.krisentalk.de)
Feel free to open a production issue
There is already an issue from someone else (following this thread a lot of
people are affected):
http://code.google.com/p/googleappengine/issues/detail?id=7133
Johan, what's going on with
We migrated our app to the HRD last night. With 4 GB of data quota in
around 2M entities it took 20-30 minutes. On the new app, we are seeing
response times at about 1% of the MS app - 100 times faster. Our app was
read only for less than three seconds - enough to affect 10 requests from 2
I have had outages throughout this morning.
My app is bedbuzzserver.appspot.com, Java app on HR.
From 2am until 7.48am. I have filed Production issue 7138.
My instances keep getting reset (and then taking too long to start up),
my '*Current Load' *logs get reset to 0
(e.g.
We have noticed that many of the downtimes Pingdom reports are the result of
AppsForDomains. If you hit your app from another app, or via AOL, or
another provider that has a peering arrangement with AppEngine it will be
up.
I'm calling this AppsForDomains issues because typically during these
Ikai,
I have not moved to HRD yet. But I am pretty sure I am the only user of my
application. However, ever since couples of days back, not only that it is
slow but I kept on running out of quota, despite the fact that I turned on
the billing. I have switched off billing yesterday as it didn't
Potential fix: set performance sliders to auto.
This is purely anecdotal but it might mean something: After reading some
post this afternoon about the instance settings not really working I
switched to AUTO idle instances and AUTO pending latency (before they were
set to 1-1 and 25ms-1.5s
Hi Riley,
That's a legitimate question, and one that we haven't officially answered
yet. It's certainly the direction that things have been moving simply due
to the nature of production management. Given that the SLA applies to HRD
and not master/slave applications, you are definitely going to
Regarding that maintenance period:
https://groups.google.com/forum/?fromgroups#!topic/google-appengine-downtime-notify/CO_x02OF9Ak
It's happening next Monday, March 19th at 4pm US/Pacific (19th March, 23:00
GMT).
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com
On
Thanks a lot. FYI: That post says Wednesday the 19th instead of Monday the
19th.
Riley
On Mon, Mar 12, 2012 at 3:52 PM, Ikai Lan (Google) ika...@google.comwrote:
Regarding that maintenance period:
GAH, it's like no matter how many times I read these things over I always
make at least one mistake.
And that's why code review is a Good Thing.
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com
On Mon, Mar 12, 2012 at 1:54 PM, Riley Eynon-Lynch
Hi Ikan,
We wouldn't mind moving to HRD from M/S, but isn't it 3X more expensive?
Also, what's the minimal way to impact our users when datastore is in
read-only mode during downtimes? Consider that every action our users take
involves writing to a datastore. Will using memcache help? Will
We wouldn't mind moving to HRD from M/S, but isn't it 3X more expensive?
No.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send
Moving to HRD is the safest way to ensure that your users are not impacted
during a downtime. Memcache and other mechanisms can be used, but will
definitely not scale and aren't guaranteed to be resilient in the face of
all downtime scenarios.
For the particular issues reported on this thread, we
HRD is not 3x more expensive. We lowered the cost to make it match the
master/slave cost.
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com
On Mon, Mar 12, 2012 at 2:33 PM, Amit Sangani amit.sang...@gmail.comwrote:
Hi Ikan,
We wouldn't mind moving to HRD from M/S,
Quick update: the time has been pushed back 2 hours to 6PM US/Pacific. See
the latest message here:
https://groups.google.com/forum/?fromgroups#!topic/google-appengine-downtime-notify/CO_x02OF9Ak
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com
On Mon, Mar 12, 2012
Hi Chris,
On Monday, March 12, 2012 3:55:00 PM UTC-7, Chris Ramsdale wrote:
For the particular issues reported on this thread, we have a few root
causes that we're looking into. In terms of a fix though, all of the
affected apps are running on M/S
Our app prodicta is using HRD, not M/S.
-appengine] Re: Outages?
HRD is not 3x more expensive. We lowered the cost to make it match the
master/slave cost.
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com http://plus.ikailan.com/
On Mon, Mar 12, 2012 at 2:33 PM, Amit Sangani amit.sang...@gmail.com
Ikai,
Our apps ids:
rvaserver
rvauser
contentfinancial
contentsports
QPS and error rates differ but they've all been getting a lot of
DeadlineExceeded exceptions and the number of instances has been higher
than usual over the last couple of days.
Regards,
Alexey
On Friday, March 9, 2012
appid: textyserver
Still getting lots of exceptions, mainly:
1) com.google.apphosting.runtime.HardDeadlineExceededError exceptions,
2) Failed startup of context
com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext
3) javax.jdo.JDOException: Transaction failed to commit at
Best explanation ever.
On Wednesday, March 7, 2012 9:45:11 PM UTC+5:30, Brandon Wirtz wrote:
So, apparently, we all imagined the problem. The status page no longer
admits to anything.
In most systems the Uptime is 100% minus the summation of the downtime of
all other systems. The
Hey everyone,
Here are a few things that will help:
1. Application IDs (--- if you have nothing else, at least provide this)
2. What is your QPS?
3. What % of your requests are errors?
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com
On Fri, Mar 9, 2012 at 7:24 AM,
Hey Ikai,
Our app id: petaclasses
QPS: 5-20 requests per second
Current instances in dashboard: 110 - 160
Usual instances: 8-15
It's hard to say % of failed requests as we have also request that fail for
other reasons (e.g. non existing pages, etc) and not sure how easily
separate them.
By
Alex, to answer that question: yes. We are looking to revamp the production
issues tracker which is far from optimal. When users can join or aggregate
issues, it allows us to quickly separate actual infrastructure hiccups from
user code issues.
Thanks for the info! Is there any other behavior you
We are Python 2.5 (no concurrent).
Yes, it seems the start-up time is just crazy high for at least some or all
instances.
I also noticed that there are lot's of instances that served just 1 request
and have average latency 0ms and have QPS=0 average instance age about 8-9
minutes (up to 11
Just a follow up:
1. Application Id: oferta-unica
2. QPS: Currently around ~10 dynamic req/sec, overall ~32 req/sec
3. After disabling concurrent requests, ~0.6 errors/sec; before, ~1.5
errors/sec.
Like Alexanders said, some of the errors aren't due to this issue, but I
can confirm that we
appid: i-strive-to
java - thread safe set to true.
On Friday, March 9, 2012 2:15:35 PM UTC-5, Ikai Lan wrote:
Hey everyone,
Here are a few things that will help:
1. Application IDs (--- if you have nothing else, at least provide this)
2. What is your QPS?
3. What % of your requests are
Also now getting below exceptions -
java.lang.ExceptionInInitializerError
at
org.datanucleus.jdo.metadata.JDOAnnotationReader.processClassAnnotations(JDOAnnotationReader.java:140)
at
I forgot to ask if these were master/slave or high replication apps. I can
always check by going to the admin console, but I'm hoping to separate them
out.
We're looking into the HR apps first (one I figure out which is which).
--
Ikai Lan
Developer Programs Engineer, Google App Engine
textyserver is on master/slave.
On Fri, Mar 9, 2012 at 2:07 PM, Ikai Lan (Google) ika...@google.com wrote:
I forgot to ask if these were master/slave or high replication apps. I can
always check by going to the admin console, but I'm hoping to separate them
out.
We're looking into the HR
Yep, I figured it out (when you look at an app in the admin console, if the
app ID has a s~ prefix, that means it runs in High Replication). I was just
pointing it out for people who hadn't yet reported application IDs.
--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com
Okay, at least with this thread, it seems like the common thread is that
the applications are master/slave applications. We're going to try to tweak
a few things on our end to lessen the pain, but pay attention to the
downtime-notify@ list (
My app is responding, again.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/pJUdXZmD8uwJ.
To post to this group, send email to
Any update on this? Still seeing many errors exceptions in the logs.
On Fri, Mar 9, 2012 at 4:18 PM, John jwb...@gmail.com wrote:
My app is responding, again.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on
Our apps seem to be better since about 15:30 PT.
On Friday, March 9, 2012 5:56:02 PM UTC-8, Amit Sangani wrote:
Any update on this? Still seeing many errors exceptions in the logs.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To
Same on my side. Normally our app is booting for 6-7 seconds, now the new
instance requests are taking 40-50 seconds, which are causing request to
timeout. The status page is not displaying any errors.
Any ideas ?
On Thursday, March 8, 2012 3:07:58 AM UTC+1, Nick wrote:
I'm getting the
So, apparently, we all imagined the problem. The status page no longer
admits to anything.
A.
On Tue, Mar 6, 2012 at 4:44 PM, Francois Masurel f.masu...@gmail.com wrote:
Yep, getting quite a few errors on loading requests lately like this one for
example :
2012-03-06 20:26:42.834
Uncaught
So, apparently, we all imagined the problem. The status page no longer
admits to anything.
In most systems the Uptime is 100% minus the summation of the downtime of
all other systems. The exception to this rule is logging. When Logging
fails to record the downtime, Uptime goes up. As a result
On Wed, Mar 7, 2012 at 11:15 AM, Brandon Wirtz drak...@digerat.com wrote:
In most systems the Uptime is 100% minus the summation of the downtime of
all other systems. The exception to this rule is logging. When Logging
fails to record the downtime, Uptime goes up. As a result Google has been
I'm getting the same errors :(
On Wednesday, March 7, 2012 11:18:39 AM UTC-5, Adam Sherman wrote:
On Wed, Mar 7, 2012 at 11:15 AM, Brandon Wirtz drak...@digerat.com
wrote:
In most systems the Uptime is 100% minus the summation of the downtime of
all other systems. The exception to this
I see a lot of errors on app startup like this:
Failed startup of context
com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@15a2dc4{/,/base/data/home/apps/s~voostip/1.357238701206702102}
com.google.apphosting.api.DeadlineExceededException: This request
(71d5265cd8f687bc) started at
That is also what I am seeing.
On Tue, Mar 6, 2012 at 4:27 PM, Jeff Schnitzer j...@infohazard.org wrote:
I see a lot of errors on app startup like this:
Failed startup of context
In addition to those, I've been getting logs with a single line of text
reading Request was aborted after waiting too long to attempt to service
your request.
Too long seems to be about ten seconds.
On Tuesday, March 6, 2012 10:27:16 PM UTC+1, Jeff Schnitzer wrote:
I see a lot of errors on
77 matches
Mail list logo
