As far as anyone outside Google can tell, the biggest issue involves
reading files off the filesystem. This may have changed recently since
it's been a while since I (or any of the other users who commonly post
about such stuff) did any measurements, but reading files seems to be
painfully slow.
Some things to consider:
* Tools that do classpath scanning (Spring, Resteasy, JDO, etc) open every
jar and class file looking for annotations. These are usually the biggest
enemy. If you can disable scanning, it will usually help a lot - although
this means more manual configuration.
* Lots of little files take longer than a few big ones. I found a small
but significant improvement (maybe 20%) by jaring up my class files - not
enough that I actually still do it. The numbers might be different if you
use classpath scanning.
* Consider carefully each jar in your project and whether you really need
it.
Jeff
On Fri, Apr 20, 2012 at 3:34 AM, Eliran Bivas <eliran.bi...@gmail.com>wrote:
> Just to add some information:
> When timing the load time of my app (not from the dev server startup but
> starting from the point my code is beging to load) it takes my application
> less than 8 seconds to load (few cases were 5secs). So I still can't figure
> out why on AppEngine it can sometimes take more than 60 seconds...
>
> I'd appreciate some analysis guidelines so I can further investigate what
> makes the app (sometimes) load very slowly.
>
>
> On Monday, April 16, 2012 10:13:50 AM UTC+3, Eliran Bivas wrote:
>>
>> Thanks for the clarification, it should be part of the documentation for
>> AppEngine architecture.
>>
>> As for my question, I believe that a Maven multi-module Spring
>> application has different view on loading times.
>> In earlier posts here it was mentioned that loading several JARs might be
>> an issue. BUT, that's how Maven works.
>> Even core Spring Framework consist of more than 10 JARs.
>>
>> Are there any recommendation on what to look for (analysis tools would be
>> great) in such deployment? Or maybe how such configuration should behave
>> (prefer lazy bean initialization over pre-init singletons)? What about the
>> JARs, would flatten them to a single Uber-JAR will help? And again, my
>> context loading is without DB operations or any HTTP connection. I assume
>> that as my project gets larger, and additional JARs will be added - are
>> there any best practices for such scenario also?
>>
>> On Monday, April 16, 2012 2:53:14 AM UTC+3, Jeff Schnitzer wrote:
>>>
>>> Woah! 30s to start your app in your local dev environment? That's nuts.
>>>
>>> You have a mistaken perception of GAE. In nearly all respects, your
>>> local dev environment will perform a single thread of execution faster than
>>> production. Your local machine has dedicated CPU cores and I/O bandwidth,
>>> all local. It has a mock datastore which likely has no synchronization
>>> issues.
>>>
>>> In production, the filesystem is loading across a network. Nearly all
>>> service calls require an RPC to a remote machine somewhere else in the
>>> cluster - maybe to several machines. You're sharing CPU cores and RAM with
>>> a dozen other apps, some of which might be really busy.
>>>
>>> What you get in production is a system that is not especially fast but
>>> nearly always consistent no matter what the queries/sec or database size.
>>>
>>> Your best strategy is to figure out why your app is taking so long to
>>> start up and address that.
>>>
>>> Jeff
>>>
>>> On Fri, Apr 13, 2012 at 2:31 PM, Eliran Bivas <eliran.bi...@gmail.com>wrote:
>>>
>>>> Do you have any recommendations how to reduce the amount of IO
>>>> operations during startup?
>>>> My app uses Spring and like many other Java best practices followed, my
>>>> project consists of several Maven modules which results several jars
>>>> creation.
>>>> I do not load any data from datastore or require HTTP connection during
>>>> startup.
>>>>
>>>> I understand the AppEngine filesystem is extremely slow, but on my
>>>> local dev machine loading my app takes ~30sec, so I assume that in a much
>>>> superior infrastructure it should take even faster. I believe AppEngine
>>>> should provide some property to extend loading requests to reach over 60
>>>> seconds (like cron operations are allowed 10mins runtime).
>>>>
>>>> Thanks in advance
>>>>
>>>>
>>>>
>>>> On Friday, April 13, 2012 5:51:15 PM UTC+3, Jeff Schnitzer wrote:
>>>>>
>>>>> Right. The problem is the 61244ms that it takes to start your app.
>>>>> How long does it normally take?
>>>>> Look at past warmup requests (the ones that work) and see how long
>>>>> they take. My guess is that
>>>>> the number is close to 60s. If GAE gets marginally slower, it pushes
>>>>> you over the edge.
>>>>>
>>>>> As for why your app takes so long to startup, I can't begin to
>>>>> speculate. What does it do at startup?
>>>>> Do any of your frameworks do classpath scanning? Do you have a lot of
>>>>> big jars? Zillions of
>>>>> class files? Do you load data from the datastore or blobstore?
>>>>>
>>>>> You need to diagnose the warmup requests that *do* work. Sure, look
>>>>> at appstats - although
>>>>> that will only show issues if your warmup fetches data from services
>>>>> (ie, not the filesystem).
>>>>> However, you really should be able to think about it a few minutes and
>>>>> figure out the problem.
>>>>> App startup is almost 100% deterministic. There are no parameters.
>>>>> You know what it does.
>>>>>
>>>>> Jeff
>>>>>
>>>>> On Fri, Apr 13, 2012 at 9:55 AM, Rui Oliveira <rucaso2...@gmail.com>wrote:
>>>>>
>>>>>> Hi Jeff
>>>>>>>
>>>>>>> Thanks for your replay. Your answer was very important for me to
>>>>>>> start looking to the right part of the problem.
>>>>>>>
>>>>>>> Just to clarify :"startup time" is the time to start a new
>>>>>>> instance right?
>>>>>>>
>>>>>>> How can I analyze the startup time inside the server? I'm looking to
>>>>>>> the appengine logs, appstats, and speedtracer, but in none of them I can
>>>>>>> find whats happening during the startup.
>>>>>>>
>>>>>>> If I deploy the program I'm getting this kind of logs:
>>>>>>>
>>>>>>>
>>>>>> 1. 012-04-13 11:33:35.889 /_ah/warmup 500 6******1244ms 0kb
>>>>>> 2. W2012-04-13 11:33:35.787 EXCEPTION com.google.apphosting.api.**
>>>>>> Dead****lineExceededException: This request (08f66682e6ba5919)
>>>>>> started at 2012/04/13 11:32:35.741 UTC and was still e
>>>>>> 3. E2012-04-13 11:33:35.790 javax.servlet.**Ser****vletContext
>>>>>> log: unavailable javax.servlet.**UnavailableExcep****tion: This
>>>>>> request (08f66682e6ba5919) started at 2012/04/13 11:32:35.741 U
>>>>>> 4. W2012-04-13 11:33:35.810 Failed startup of context
>>>>>> com.google.apphosting.utils.**je****tty.**
>>>>>> RuntimeAppEngineWebAppCont****ext@**1811e2c{/,/base/data/home/****
>>>>>> **apps/s~airmenudemo/29.**35817792****1953
>>>>>> 5. C2012-04-13 11:33:35.816 Uncaught exception from servlet
>>>>>> javax.servlet.**UnavailableExcep****tion: Initialization failed.
>>>>>> at com.google.apphosting.runtime.******jetty.AppVersionHandlerMa
>>>>>> 6. I2012-04-13 11:33:35.819 This request caused a new process to
>>>>>> be started for your application, and thus caused your application
>>>>>> code to
>>>>>> be loaded for the first time. This requ
>>>>>> 7. W2012-04-13 11:33:35.819 A problem was encountered with the
>>>>>> process that handled this request, causing it to exit. This is likely
>>>>>> to
>>>>>> cause a new process to be used for the nex
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On appstats: ( http://www.airmenudemo.**appsp****
>>>>>>> ot.com/appstats/stats<http://www.airmenudemo.appspot.com/appstats/stats>
>>>>>>> )
>>>>>>>
>>>>>>> (16) 2012-04-13 11:36:31.587 "GET /appstats/" 307
>>>>>>> <http://www.airmenudemo.appspot.com/appstats/details?time=1334316991587>real=215ms
>>>>>>> api=0ms overhead=0ms (0 RPCs)
>>>>>>>
>>>>>>> (17) 2012-04-13 11:35:49.207 "GET /symbolmanifest.json" 404
>>>>>>> <http://www.airmenudemo.appspot.com/appstats/details?time=1334316949207>real=609ms
>>>>>>> api=0ms overhead=0ms (0 RPCs)
>>>>>>>
>>>>>>> (18) 2012-04-13 11:27:25.255 "GET /_ah/warmup" 200
>>>>>>> <http://www.airmenudemo.appspot.com/appstats/details?time=1334316445255>real=441ms
>>>>>>> api=0ms overhead=0ms (0 RPCs)
>>>>>>>
>>>>>>> As you can see after deploy the appstats don't log nothing.
>>>>>>
>>>>>> After deploy I can't even open appstats.
>>>>>>
>>>>>> After some minutes / hours everything starts ok.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Rui
>>>>>>
>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Google App Engine" group.
>>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>>> **ms**g/google-appengine/-/frES_**he8D**HYJ<https://groups.google.com/d/msg/google-appengine/-/frES_he8DHYJ>
>>>>>> .
>>>>>>
>>>>>> To post to this group, send email to google-appengine@googlegroups.**
>>>>>> **com <google-appengine@googlegroups.com>.
>>>>>> To unsubscribe from this group, send email to
>>>>>> google-appengine+unsubscribe@**g**ooglegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>>>> .
>>>>>> For more options, visit this group at http://groups.google.com/**
>>>>>> group**/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
>>>>>> .
>>>>>>
>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine" group.
>>>> To view this discussion on the web visit https://groups.google.com/d/**
>>>> msg/google-appengine/-/**WJ0eGsUpEgwJ<https://groups.google.com/d/msg/google-appengine/-/WJ0eGsUpEgwJ>
>>>> .
>>>>
>>>> To post to this group, send email to google-appengine@googlegroups.**
>>>> com <google-appengine@googlegroups.com>.
>>>> To unsubscribe from this group, send email to
>>>> google-appengine+unsubscribe@**googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>> .
>>>> For more options, visit this group at http://groups.google.com/**
>>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
>>>> .
>>>>
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/edj6rgd5s54J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
