[appengine-java] Re: App instance recycling and response times - is there solution?
Dear all, I've found this in the FAQ: http://code.google.com/intl/en/appengine/kb/java.html#Can_I_Pay_To_Reserve_My_JVM Can I pay to keep a JVM reserved for my application? We've seen this request from some developers with low-traffic applications who'd like to reduce the percentage of loading requests they receive. Although we have many improvements in the pipeline to improve loading request performance, we'd like to gauge the general interest in this feature. If you'd like to be able to reserve a JVM at a price, please star this issue: http://code.google.com/p/googleappengine/issues/detail?id=2456. If there's a particular pricing scheme you're interested in, let us know. Cheers, Guillermo. On 22 mar, 10:21, Guillermo Schwarz guillermo.schw...@gmail.com wrote: My wild guess is that Google still has some work to do in terms of balancing the load and preventing poorly written code in one app from causing pain for other apps. Not really. The load balacer is working fine. When the application is loaded it is incredibly fast, so the load balancer must be working right. The problem is when the app is being loaded. In order to load the app, a kind of core file must be read from disk in order to represent the app state in RAM. That process takes too long (used to be up to 3 secs, now it is up to 12 seconds AFAIK) because: 1. Some applications can't be unloaded at all (they are using the queue to ping themselves) which leaves less free RAM. When apps ping themselves GAE shouldn´t take those pings into account in order to free some memory. 2. The disk reading takes a lot of time. Probably this shouldn't take longer than 500 ms. Let me explain. Even if the war file is huge, it can be divided in multiple disks and loaded very fast. Probably using SSDs would speed things up a bit. I haven't seen the logs of GAE (not of the apps but of GAE itself) but I guess one of those problems is consuming most of the time. Cheers, Guillermo. On 22 mar, 09:30, James Koch jamesk...@gmail.com wrote: Jake, I don't know enough about the GAE internals to do anything more than speculate on root causes and interactions between the symptoms I've seen. Unfortunately Google staff have their lips sealed. Don Schwartz engaged with me for a few hours, asked me some questions, then went silent for two weeks now and hasn't responded to my follow-up. My wild guess is that Google still has some work to do in terms of balancing the load and preventing poorly written code in one app from causing pain for other apps. James -Original Message- From: google-appengine-java@googlegroups.com [mailto:google-appengine-j...@googlegroups.com] On Behalf Of Jake Sent: Friday, March 19, 2010 8:56 AM To: Google App Engine for Java Subject: [appengine-java] Re: App instance recycling and response times - is there solution? Hey James, Also thanks for the clarification. It hadn't occurred to me that what appeared to be frequent restarts could be an additional JVM. I'm still not entirely familiar with that concept, but I get the basic idea that requests served to the same client may not always come from the same instance of my application. As the only traffic I've seen is...me...I just assumed it was all one instance. Anyways, are issues #1-4 really linked together? I'm less concerned about slow startup than constant startup. Also, while I haven't read the ENTIRE thing, Issue #2456 talks about paying to keep a warm JVM, which I'm not interested in... I don't mind a 6-10 (or 15 second) delay on the first request in, say, the span of an hour. I just mind it when it's every single request. Jake On Mar 18, 1:44 pm, James Koch jamesk...@gmail.com wrote: In my short time using GAE I've seen several issues discussed under the label response times. The lines between these issues are fuzzy, and we aren't all talking about response times using the same semantics. As an example, Jake, you might not be seeing frequent restarts, you could also be seeing spinup of additional JVMs (even though you're low-traffic like me). Maybe enumerating them will help us all pull in the same direction? Here are the issues I've seen mentioned: 1 - Slow JVM startup speed (5-7s for something that uses JDO and no other frameworks) 2 - Not affecting me personally, but startup cost of other frameworks (Struts, etc.) has definitely been mentioned. 3 - Frequent JVM restarts (after 1 min of inactivity) - This interval feels too short to me, as a small app developer. 4 - Frequent server overload caused by OTHER apps, which cause a request to be re-routed to a new JVM instead of a warm one, thus incurring the startup penalty regardless of user activity levels. 5 - Request was aborted after waiting too long on normally fast-loading pages. Presumably similar to #3 (caused by server overload from OTHER apps
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
On Mon, Mar 22, 2010 at 7:21 AM, Guillermo Schwarz guillermo.schw...@gmail.com wrote: In order to load the app, a kind of core file must be read from disk in order to represent the app state in RAM. That process takes too long (used to be up to 3 secs, now it is up to 12 seconds AFAIK) because: When (re-)loading our apps, we see the full startup process; appengine doesn't appear to serialize them to disk the way your laptop might go in and out of hibernation. I suspect the hot server theory is more likely. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: App instance recycling and response times - is there solution?
It makes sense because web server apps are staless anyway, so what would be the point of storing the state? The only problem would be that Spring apps take so long to start up. Anyway, I think the problem is something else, because when loaded the application is very fast and I'm not using Spring or any other thingy that could cause the application to use more than a sec to load. Still the app sometimes takes up to 15 secs to load. Cheers, Guillermo. On 24 mar, 17:08, Jeff Schnitzer j...@infohazard.org wrote: When (re-)loading our apps, we see the full startup process; appengine doesn't appear to serialize them to disk the way your laptop might go in and out of hibernation. I suspect the hot server theory is more likely. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
As a followup, today (3/17) from 1-3PM PST I received several
instances of Request
was aborted after waiting too long to attempt to service your request. This
is on my app w/ zero users, just 3 requests/minute of a blank page as a test
load.
On Thu, Mar 11, 2010 at 1:50 PM, Don Schwarz schwa...@google.com wrote:
Can you respond privately with your app id?
On Thu, Mar 11, 2010 at 10:10 AM, James jamesk...@gmail.com wrote:
I setup some pings of my add a few minutes ago, and I'm still seeing
recycling :(
My ping setup can't go lower than 60s intervals, so I have two running
concurrently. Here's a sample of 20 log entries over 10 minutes,
with . Three recyclings occur, and they happen less than 10s after a
previous request. Really Google, you're killing my JVM after TEN
SECONDS? And I get to pay you for the ton of CPU each startup uses?
Sounds like the more recycling, the more profitable the App Engine
becomes.
-
* 03-11 08:02AM 38.506 /?Pragma=no-cache 200 2158ms 2235cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:02AM 23.144 /?Pragma=no-cache 200 53ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:01AM 06.134 /?Pragma=no-cache 200 75ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:00AM 51.707 /?Pragma=no-cache 200 49ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:00AM 05.823 /?Pragma=no-cache 200 49ms 58cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:59AM 51.499 /?Pragma=no-cache 200 56ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:59AM 05.584 /?Pragma=no-cache 200 47ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:58AM 51.274 /?Pragma=no-cache 200 61ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:58AM 05.371 /?Pragma=no-cache 200 64ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:57AM 51.025 /?Pragma=no-cache 200 74ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
* 03-11 07:56AM 57.327 /?Pragma=no-cache 200 7835ms 2119cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:56AM 50.784 /?Pragma=no-cache 200 75ms 58cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:55AM 57.008 /?Pragma=no-cache 200 50ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
* 03-11 07:55AM 46.384 /?Pragma=no-cache 200 4250ms 2060cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:54AM 56.782 /?Pragma=no-cache 200 70ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:54AM 46.157 /?Pragma=no-cache 200 54ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:53AM 56.586 /?Pragma=no-cache 200 52ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:53AM 45.934 /?Pragma=no-cache 200 51ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:52AM 56.240 /?Pragma=no-cache 200 62ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:52AM 45.718 /?Pragma=no-cache 200 57ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
-
On Jan 30, 11:02 pm, Alyxandor a.revolution.ultra.b...@gmail.com
wrote:
If you are experiencing failed requests on your long-running /
requests, consider performing some kind of pre-warming procedure of
your own... If you are getting timeout errors, Ping a do-nothing url,
and wait for it to return before running the big job. If it's a big
job, users should expect to wait anyway {and you should tell them they
are waiting!}, so the ping ensures {almost} that a warm JVM is running
in the server nearest said users, and then the big /request can
{usually} avoid getting killed with extra spin up time. Very unlucky
users would get a /ping on an old JVM, and /request a new one, but...
Technology isn't perfect... YET!
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group,
[appengine-java] Re: App instance recycling and response times - is there solution?
This is NOT just a problem with Spring -- stop talking like optimization is going to fix things. It takes too much time for a naked servlet to load (i.e; 5-10 seconds). The only jars that I have are for JPA. On Jan 12, 8:32 pm, Jeff Schnitzer j...@infohazard.org wrote: I've been thinking about this issue a little. It's not quite as straightforward as just keeping an instance warm. Even if you have an app that gets multiple hits per second, there will still be cold starts: * When a new instance comes online to serve more demand. * When you redeploy a version of your app. Is appengine smart about letting new instances added to the pool warm up before serving requests? It's hard to tell from my logs but it doesn't look like it. I know appengine is *not* smart about warming up an instance before redeploying. When I redeploy, some large number of users must wait while the appserver(s) startup. One thing to keep in mind during these discussions is how other Java EE environments solve this problem: They *don't*. For a long time it's been assumed in the EE development that server initialization time is irrelevant, and we grew fat libraries that take tens of seconds to minutes to start up. The problem is, this time has *never* been irrelevant - even in a production environment you must deploy new versions of your app, and none of the appservers I'm familiar with are smart enough to keep serving off the old version while the new one loads. Users with unlucky timing always got screwed. We just didn't care because we only deployed code once a week and we added/removed server instances far less often than that. Well guess what, now it's easy - you can deploy up to 1,000 times per day just by clicking a button in eclipse, and server provisioning is now not only trivial but 100% transparent to you. Just try that with WebSphere! You aren't going to like this, but here's the only answer that isn't going to piss off your customers: Stop using Spring. Stop performing eager initialization. Stop assuming that users don't see startup time. Yes, change the way you write code. Jeff On Tue, Jan 12, 2010 at 1:11 PM, Don Schwarz schwa...@google.com wrote: Make sure you are using offline precompilation. We are always working on optimizations to decrease the latency of loading requests, but here are some other tips: http://googleappengine.blogspot.com/2009/12/request-performance-in-ja... On Tue, Jan 12, 2010 at 3:01 PM, Locke locke2...@gmail.com wrote: I agree that making users wait 20 seconds for your app to load is not adequate for the vast majority of apps. I also agree that reengineering everything to try and hide load times from users is a poor solution in most cases. Using cron to keep your app loaded will not consume your quota; it will actually conserve your quota. Every time your app loads you will be billed for 20s of CPU time. If you keep it loaded, you will only be billed for a few milliseconds per 'keep-alive' cron execution. However, the Google engineers who post here have recommended against doing this. If everyone did it, appengine might run out of resources (RAM, I assume). I imagine that Google will need to either find a way to load apps in 1/10th the time (the ideal solution), raise prices significantly, or ration resources in some other way. If I may make a suggestion to the Google engineers: offer a keep my app loaded option and make it available ONLY for billing-enabled apps. Disable cron for apps which are not billing-enabled, so that people who just want free hosting or are merely toying with appengine won't be using up resources all the time. This way, the people who have shown that they are serious about appengine (by laying their cash down) won't be driven away by the people who are just fooling with it. Yes, we are seriously considering something like this. Please star this issue for updates: http://code.google.com/p/googleappengine/issues/detail?id=2456 On Jan 12, 1:43 pm, Konrad konradpaw...@gmail.com wrote: I asked same question on Stack Overflow (http://stackoverflow.com/ questions/2051036/google-app-engine-application-instance-recycling-and- response-times). So far proposed solutions (in SO thread and found on other websites) do not satisfy me. Creating cron or any other kind of periodic HTTP requests to keep instance up and running make no sense. First - there is no evidence that this instance will serve next coming request (eg. from different network location etc.), second - it will consume Quota (which is less a problem). Other solution - refactoring app - replacing critical functionality with lightweight servlet - sounds better, but is GAE forcing to go back to CGI programming style? And I could replace let say - API calls, but to keep UI in better performance shape I would need to throw away Spring MVC. Can anyone
[appengine-java] Re: App instance recycling and response times - is there solution?
I setup some pings of my add a few minutes ago, and I'm still seeing
recycling :(
My ping setup can't go lower than 60s intervals, so I have two running
concurrently. Here's a sample of 20 log entries over 10 minutes,
with . Three recyclings occur, and they happen less than 10s after a
previous request. Really Google, you're killing my JVM after TEN
SECONDS? And I get to pay you for the ton of CPU each startup uses?
Sounds like the more recycling, the more profitable the App Engine
becomes.
-
* 03-11 08:02AM 38.506 /?Pragma=no-cache 200 2158ms 2235cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:02AM 23.144 /?Pragma=no-cache 200 53ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:01AM 06.134 /?Pragma=no-cache 200 75ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:00AM 51.707 /?Pragma=no-cache 200 49ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:00AM 05.823 /?Pragma=no-cache 200 49ms 58cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:59AM 51.499 /?Pragma=no-cache 200 56ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:59AM 05.584 /?Pragma=no-cache 200 47ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:58AM 51.274 /?Pragma=no-cache 200 61ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:58AM 05.371 /?Pragma=no-cache 200 64ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:57AM 51.025 /?Pragma=no-cache 200 74ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
* 03-11 07:56AM 57.327 /?Pragma=no-cache 200 7835ms 2119cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:56AM 50.784 /?Pragma=no-cache 200 75ms 58cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:55AM 57.008 /?Pragma=no-cache 200 50ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
* 03-11 07:55AM 46.384 /?Pragma=no-cache 200 4250ms 2060cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:54AM 56.782 /?Pragma=no-cache 200 70ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:54AM 46.157 /?Pragma=no-cache 200 54ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:53AM 56.586 /?Pragma=no-cache 200 52ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:53AM 45.934 /?Pragma=no-cache 200 51ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:52AM 56.240 /?Pragma=no-cache 200 62ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:52AM 45.718 /?Pragma=no-cache 200 57ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
-
On Jan 30, 11:02 pm, Alyxandor a.revolution.ultra.b...@gmail.com
wrote:
If you are experiencing failed requests on your long-running /
requests, consider performing some kind of pre-warming procedure of
your own... If you are getting timeout errors, Ping a do-nothing url,
and wait for it to return before running the big job. If it's a big
job, users should expect to wait anyway {and you should tell them they
are waiting!}, so the ping ensures {almost} that a warm JVM is running
in the server nearest said users, and then the big /request can
{usually} avoid getting killed with extra spin up time. Very unlucky
users would get a /ping on an old JVM, and /request a new one, but...
Technology isn't perfect... YET!
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
Can you respond privately with your app id?
On Thu, Mar 11, 2010 at 10:10 AM, James jamesk...@gmail.com wrote:
I setup some pings of my add a few minutes ago, and I'm still seeing
recycling :(
My ping setup can't go lower than 60s intervals, so I have two running
concurrently. Here's a sample of 20 log entries over 10 minutes,
with . Three recyclings occur, and they happen less than 10s after a
previous request. Really Google, you're killing my JVM after TEN
SECONDS? And I get to pay you for the ton of CPU each startup uses?
Sounds like the more recycling, the more profitable the App Engine
becomes.
-
* 03-11 08:02AM 38.506 /?Pragma=no-cache 200 2158ms 2235cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:02AM 23.144 /?Pragma=no-cache 200 53ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:01AM 06.134 /?Pragma=no-cache 200 75ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:00AM 51.707 /?Pragma=no-cache 200 49ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 08:00AM 05.823 /?Pragma=no-cache 200 49ms 58cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:59AM 51.499 /?Pragma=no-cache 200 56ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:59AM 05.584 /?Pragma=no-cache 200 47ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:58AM 51.274 /?Pragma=no-cache 200 61ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:58AM 05.371 /?Pragma=no-cache 200 64ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:57AM 51.025 /?Pragma=no-cache 200 74ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
* 03-11 07:56AM 57.327 /?Pragma=no-cache 200 7835ms 2119cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:56AM 50.784 /?Pragma=no-cache 200 75ms 58cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:55AM 57.008 /?Pragma=no-cache 200 50ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
* 03-11 07:55AM 46.384 /?Pragma=no-cache 200 4250ms 2060cpu_ms 0kb
Site 24 X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:54AM 56.782 /?Pragma=no-cache 200 70ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:54AM 46.157 /?Pragma=no-cache 200 54ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:53AM 56.586 /?Pragma=no-cache 200 52ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:53AM 45.934 /?Pragma=no-cache 200 51ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:52AM 56.240 /?Pragma=no-cache 200 62ms 38cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
03-11 07:52AM 45.718 /?Pragma=no-cache 200 57ms 19cpu_ms 0kb Site 24
X 7 RPT-HTTPClient/0.3-3E,gzip(gfe)
-
On Jan 30, 11:02 pm, Alyxandor a.revolution.ultra.b...@gmail.com
wrote:
If you are experiencing failed requests on your long-running /
requests, consider performing some kind of pre-warming procedure of
your own... If you are getting timeout errors, Ping a do-nothing url,
and wait for it to return before running the big job. If it's a big
job, users should expect to wait anyway {and you should tell them they
are waiting!}, so the ping ensures {almost} that a warm JVM is running
in the server nearest said users, and then the big /request can
{usually} avoid getting killed with extra spin up time. Very unlucky
users would get a /ping on an old JVM, and /request a new one, but...
Technology isn't perfect... YET!
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: App instance recycling and response times - is there solution?
If you are experiencing failed requests on your long-running /
requests, consider performing some kind of pre-warming procedure of
your own... If you are getting timeout errors, Ping a do-nothing url,
and wait for it to return before running the big job. If it's a big
job, users should expect to wait anyway {and you should tell them they
are waiting!}, so the ping ensures {almost} that a warm JVM is running
in the server nearest said users, and then the big /request can
{usually} avoid getting killed with extra spin up time. Very unlucky
users would get a /ping on an old JVM, and /request a new one, but...
Technology isn't perfect... YET!
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
治本只能等GAE的升级了，这样确实只能保证你的应用不会因为两分钟没有请求被关闭！ 不知道GAE收费版本是不是没有这个问题 2010/1/13 yjun hu itswa...@gmail.com 治标不治本 On Wed, Jan 13, 2010 at 1:24 PM, 杨浩 skzr@gmail.com wrote: create a cron:every one minutes to run for keep your app online in the GAE! I try it,and it is work very good! -- dream or truth -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
Jeff, in one point i disagree. In a high available einvironment you would have a cluster of load balanced application servers and you would deploy new versions of your app in turn, one at a time. So if one instance is down the other(s) will continue serving your users (though it might require sticky sessions). Of course, this is a must for enterprise applications. Regards, Stephan 2010/1/13 Jeff Schnitzer j...@infohazard.org I've been thinking about this issue a little. It's not quite as straightforward as just keeping an instance warm. Even if you have an app that gets multiple hits per second, there will still be cold starts: * When a new instance comes online to serve more demand. * When you redeploy a version of your app. Is appengine smart about letting new instances added to the pool warm up before serving requests? It's hard to tell from my logs but it doesn't look like it. I know appengine is *not* smart about warming up an instance before redeploying. When I redeploy, some large number of users must wait while the appserver(s) startup. One thing to keep in mind during these discussions is how other Java EE environments solve this problem: They *don't*. For a long time it's been assumed in the EE development that server initialization time is irrelevant, and we grew fat libraries that take tens of seconds to minutes to start up. The problem is, this time has *never* been irrelevant - even in a production environment you must deploy new versions of your app, and none of the appservers I'm familiar with are smart enough to keep serving off the old version while the new one loads. Users with unlucky timing always got screwed. We just didn't care because we only deployed code once a week and we added/removed server instances far less often than that. Well guess what, now it's easy - you can deploy up to 1,000 times per day just by clicking a button in eclipse, and server provisioning is now not only trivial but 100% transparent to you. Just try that with WebSphere! You aren't going to like this, but here's the only answer that isn't going to piss off your customers: Stop using Spring. Stop performing eager initialization. Stop assuming that users don't see startup time. Yes, change the way you write code. Jeff On Tue, Jan 12, 2010 at 1:11 PM, Don Schwarz schwa...@google.com wrote: Make sure you are using offline precompilation. We are always working on optimizations to decrease the latency of loading requests, but here are some other tips: http://googleappengine.blogspot.com/2009/12/request-performance-in-java.html On Tue, Jan 12, 2010 at 3:01 PM, Locke locke2...@gmail.com wrote: I agree that making users wait 20 seconds for your app to load is not adequate for the vast majority of apps. I also agree that reengineering everything to try and hide load times from users is a poor solution in most cases. Using cron to keep your app loaded will not consume your quota; it will actually conserve your quota. Every time your app loads you will be billed for 20s of CPU time. If you keep it loaded, you will only be billed for a few milliseconds per 'keep-alive' cron execution. However, the Google engineers who post here have recommended against doing this. If everyone did it, appengine might run out of resources (RAM, I assume). I imagine that Google will need to either find a way to load apps in 1/10th the time (the ideal solution), raise prices significantly, or ration resources in some other way. If I may make a suggestion to the Google engineers: offer a keep my app loaded option and make it available ONLY for billing-enabled apps. Disable cron for apps which are not billing-enabled, so that people who just want free hosting or are merely toying with appengine won't be using up resources all the time. This way, the people who have shown that they are serious about appengine (by laying their cash down) won't be driven away by the people who are just fooling with it. Yes, we are seriously considering something like this. Please star this issue for updates: http://code.google.com/p/googleappengine/issues/detail?id=2456 On Jan 12, 1:43 pm, Konrad konradpaw...@gmail.com wrote: I asked same question on Stack Overflow (http://stackoverflow.com/ questions/2051036/google-app-engine-application-instance-recycling-and- response-times). So far proposed solutions (in SO thread and found on other websites) do not satisfy me. Creating cron or any other kind of periodic HTTP requests to keep instance up and running make no sense. First - there is no evidence that this instance will serve next coming request (eg. from different network location etc.), second - it will consume Quota (which is less a problem). Other solution - refactoring app - replacing critical functionality with lightweight servlet - sounds better, but is GAE forcing to go
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
On Thu, Jan 14, 2010 at 12:59 PM, Stephan Hartmann hartm...@metamesh.de wrote: Jeff, in one point i disagree. In a high available einvironment you would have a cluster of load balanced application servers and you would deploy new versions of your app in turn, one at a time. So if one instance is down the other(s) will continue serving your users (though it might require sticky sessions). Of course, this is a must for enterprise applications. In theory, yes. In practice, you often have a clustered, replicated cache which (depending on your code change) may or may not gracefully handle swapping out code on one server at a time. Also, you'll lose session state - unless have replicated sessions, which also might not survive your code changes. Of course, replicating sessions further decreases your scalability. I've spent most of the last decade deploying big enterprise systems like this and they really don't work quite like the advertising literature says. It has required a few changes in my coding practice, but I'm much happier with the GAE approach. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
I've tried this, but after 1 day, GAE recycled my instance every 40 seconds. Then I used task queue to request my site per 30 seconds, but GAE started to recycle every 20 seconds. So I don't think it's a good way. 2010/1/13 杨浩 skzr@gmail.com: create a cron:every one minutes to run for keep your app online in the GAE! I try it,and it is work very good! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: App instance recycling and response times - is there solution?
I don't have time to go into details, so this is a 10,000 ft view, but perhaps you could potentially extend the base spring context listener to work with the distributed cache (or some other mechanism between web apps) which keeps an instance of springs context so that it doesn't have to be repeatedly loaded. Also, you could write another Servlet which clears the context instance when you actually do need it to be reloaded. Hope this helps. - Derrick On Jan 12, 11:40 am, Konrad konradpaw...@gmail.com wrote: Hi, I decided to do some long-run performance tests on my application. I created some small client hitting app every 5-30 minutes and I run 3-5 of threads with such client. I noticed huge differenced in response times and started to investigate issue. I found reason very quick. I am experiencing same issues as described in following topics: http://groups.google.com/group/google-appengine/browse_thread/thread/... http://groups.google.com/group/google-appengine/browse_thread/thread/... http://groups.google.com/group/google-appengine-java/browse_thread/th... I am using Springframework, it tkes around 18-20s to start app instance, which is causing response times to take from 1s (when requests hits running app - very rare) to 22s when fresh application is created. Is there any solution for this? I was thinking about creating most basic servlet performing critical tasks (serving API call) and leave UI as is. But then I would loose all benefits of Springframework. Is there any solution for this? After solving (hacking) numerous constrains of App Engine which I hit while developing my app that is the one I think will make me move out of App Engine... that's simply to much to all the time think how to win with GAE problems than how to solve my application problems... Any help? Regards Konrad -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: App instance recycling and response times - is there solution?
I asked same question on Stack Overflow (http://stackoverflow.com/ questions/2051036/google-app-engine-application-instance-recycling-and- response-times). So far proposed solutions (in SO thread and found on other websites) do not satisfy me. Creating cron or any other kind of periodic HTTP requests to keep instance up and running make no sense. First - there is no evidence that this instance will serve next coming request (eg. from different network location etc.), second - it will consume Quota (which is less a problem). Other solution - refactoring app - replacing critical functionality with lightweight servlet - sounds better, but is GAE forcing to go back to CGI programming style? And I could replace let say - API calls, but to keep UI in better performance shape I would need to throw away Spring MVC. Can anyone confirm that this behavior will not be fixed (yes fixed - as I think it is a blocker for any serious GAE usage)? Or can anyone suggest different solution? Thanks Konrad -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
Make sure you are using offline precompilation. We are always working on optimizations to decrease the latency of loading requests, but here are some other tips: http://googleappengine.blogspot.com/2009/12/request-performance-in-java.html On Tue, Jan 12, 2010 at 3:01 PM, Locke locke2...@gmail.com wrote: I agree that making users wait 20 seconds for your app to load is not adequate for the vast majority of apps. I also agree that reengineering everything to try and hide load times from users is a poor solution in most cases. Using cron to keep your app loaded will not consume your quota; it will actually conserve your quota. Every time your app loads you will be billed for 20s of CPU time. If you keep it loaded, you will only be billed for a few milliseconds per 'keep-alive' cron execution. However, the Google engineers who post here have recommended against doing this. If everyone did it, appengine might run out of resources (RAM, I assume). I imagine that Google will need to either find a way to load apps in 1/10th the time (the ideal solution), raise prices significantly, or ration resources in some other way. If I may make a suggestion to the Google engineers: offer a keep my app loaded option and make it available ONLY for billing-enabled apps. Disable cron for apps which are not billing-enabled, so that people who just want free hosting or are merely toying with appengine won't be using up resources all the time. This way, the people who have shown that they are serious about appengine (by laying their cash down) won't be driven away by the people who are just fooling with it. Yes, we are seriously considering something like this. Please star this issue for updates: http://code.google.com/p/googleappengine/issues/detail?id=2456 On Jan 12, 1:43 pm, Konrad konradpaw...@gmail.com wrote: I asked same question on Stack Overflow (http://stackoverflow.com/ questions/2051036/google-app-engine-application-instance-recycling-and- response-times). So far proposed solutions (in SO thread and found on other websites) do not satisfy me. Creating cron or any other kind of periodic HTTP requests to keep instance up and running make no sense. First - there is no evidence that this instance will serve next coming request (eg. from different network location etc.), second - it will consume Quota (which is less a problem). Other solution - refactoring app - replacing critical functionality with lightweight servlet - sounds better, but is GAE forcing to go back to CGI programming style? And I could replace let say - API calls, but to keep UI in better performance shape I would need to throw away Spring MVC. Can anyone confirm that this behavior will not be fixed (yes fixed - as I think it is a blocker for any serious GAE usage)? Or can anyone suggest different solution? Thanks Konrad -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
I've been thinking about this issue a little. It's not quite as straightforward as just keeping an instance warm. Even if you have an app that gets multiple hits per second, there will still be cold starts: * When a new instance comes online to serve more demand. * When you redeploy a version of your app. Is appengine smart about letting new instances added to the pool warm up before serving requests? It's hard to tell from my logs but it doesn't look like it. I know appengine is *not* smart about warming up an instance before redeploying. When I redeploy, some large number of users must wait while the appserver(s) startup. One thing to keep in mind during these discussions is how other Java EE environments solve this problem: They *don't*. For a long time it's been assumed in the EE development that server initialization time is irrelevant, and we grew fat libraries that take tens of seconds to minutes to start up. The problem is, this time has *never* been irrelevant - even in a production environment you must deploy new versions of your app, and none of the appservers I'm familiar with are smart enough to keep serving off the old version while the new one loads. Users with unlucky timing always got screwed. We just didn't care because we only deployed code once a week and we added/removed server instances far less often than that. Well guess what, now it's easy - you can deploy up to 1,000 times per day just by clicking a button in eclipse, and server provisioning is now not only trivial but 100% transparent to you. Just try that with WebSphere! You aren't going to like this, but here's the only answer that isn't going to piss off your customers: Stop using Spring. Stop performing eager initialization. Stop assuming that users don't see startup time. Yes, change the way you write code. Jeff On Tue, Jan 12, 2010 at 1:11 PM, Don Schwarz schwa...@google.com wrote: Make sure you are using offline precompilation. We are always working on optimizations to decrease the latency of loading requests, but here are some other tips: http://googleappengine.blogspot.com/2009/12/request-performance-in-java.html On Tue, Jan 12, 2010 at 3:01 PM, Locke locke2...@gmail.com wrote: I agree that making users wait 20 seconds for your app to load is not adequate for the vast majority of apps. I also agree that reengineering everything to try and hide load times from users is a poor solution in most cases. Using cron to keep your app loaded will not consume your quota; it will actually conserve your quota. Every time your app loads you will be billed for 20s of CPU time. If you keep it loaded, you will only be billed for a few milliseconds per 'keep-alive' cron execution. However, the Google engineers who post here have recommended against doing this. If everyone did it, appengine might run out of resources (RAM, I assume). I imagine that Google will need to either find a way to load apps in 1/10th the time (the ideal solution), raise prices significantly, or ration resources in some other way. If I may make a suggestion to the Google engineers: offer a keep my app loaded option and make it available ONLY for billing-enabled apps. Disable cron for apps which are not billing-enabled, so that people who just want free hosting or are merely toying with appengine won't be using up resources all the time. This way, the people who have shown that they are serious about appengine (by laying their cash down) won't be driven away by the people who are just fooling with it. Yes, we are seriously considering something like this. Please star this issue for updates: http://code.google.com/p/googleappengine/issues/detail?id=2456 On Jan 12, 1:43 pm, Konrad konradpaw...@gmail.com wrote: I asked same question on Stack Overflow (http://stackoverflow.com/ questions/2051036/google-app-engine-application-instance-recycling-and- response-times). So far proposed solutions (in SO thread and found on other websites) do not satisfy me. Creating cron or any other kind of periodic HTTP requests to keep instance up and running make no sense. First - there is no evidence that this instance will serve next coming request (eg. from different network location etc.), second - it will consume Quota (which is less a problem). Other solution - refactoring app - replacing critical functionality with lightweight servlet - sounds better, but is GAE forcing to go back to CGI programming style? And I could replace let say - API calls, but to keep UI in better performance shape I would need to throw away Spring MVC. Can anyone confirm that this behavior will not be fixed (yes fixed - as I think it is a blocker for any serious GAE usage)? Or can anyone suggest different solution? Thanks Konrad -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
This is a tragedy of the commons. It may in fact work for you right now, and will continue to work... until it doesn't. It will stop working when enough people figure it out and millions of zombie applications push the working applications out of memory. Or Google pulls the plug on this workaround (more likely). It also entitles those of us with real applications that play by the rules to call you a jerk. Jeff On Tue, Jan 12, 2010 at 9:24 PM, 杨浩 skzr@gmail.com wrote: create a cron:every one minutes to run for keep your app online in the GAE! I try it,and it is work very good! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: App instance recycling and response times - is there solution?
治标不治本 On Wed, Jan 13, 2010 at 1:24 PM, 杨浩 skzr@gmail.com wrote: create a cron:every one minutes to run for keep your app online in the GAE! I try it,and it is work very good! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- dream or truth -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
