[google-appengine] Re: Startup time exceeded...on F4?!
While startup times appears to be a contentious topic. I think that we can all agree that: *Fewer Instance *==* Few Cold Starts *==* Happy Customers *==* Smaller Bills * Currently Apps of all sizes are capped at 10 concurrent requests. F4 instance can handle no more concurrent request then a F1. I have create a feature request to make this option configurable. If you believe that your App can handle Greater than 10 concurrent requests please star this issue: http://code.google.com/p/googleappengine/issues/detail?id=7927 Thank you -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/MM5pHzrCPVcJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Great news, everyone! I have replaced all String concatenation with StringBuilder, and extensively reordered all my import statements. My startup time has been reduced 50% from twice it's original value. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/kL6FzpNuUb4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
I have a large GWT app with over 50 complex data entities with very complicated relationships between them. There are over 100 RPC methods, and I use various GAE services (Users, Memcache, Blobstore, Images, and Mail). My new instance startup time ranges from 4 to 5 sec on F1. Sometimes it goes a little higher. The lowest was 3.6 sec. I use the low-level API and no DI. Productivity tools are nice, but they have their cost. In any case, if Jeff's suggestion is implemented (dispatch only to warm instances), it would not matter if the start up time is 4 sec or 40. I followed this discussion closely, and to me it is obvious that the existing GAE mechanism is inferior to what Jeff is proposing (for any number of instances). There may be a good technical reason why the GAE team chose their current implementation. It is also possible that GAE was built by engineers who did not think in terms of web user experience, and who believed that a rare 30 seconds delay is the acceptable price to pay for some other benefits of the current implementation. This proposal was already starred by many people (issue 7865). It would be nice if the GAE teams takes another look and tells us whether we should expect any changes. I am sure a lot of app developers would like to know whether they should optimize for fast cold starts. These optimizations are expensive (both in direct development time and in lost productivity), so this is not an idle question. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/VsYwv3Sf52wJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Amazing, Andrei. If you can get those startup times going with GWT and the complexities you mentioned in your app, then clearly stripping out DI and using low-level API (and likely jar'ing everything) is a mega contributor to reducing startup time. I would hate to have to spend the team's time stripping that stuff out, i would just pay for higher idle instances until request #7865 gets implemented. I agreed that only dispatching to warm instances makes the most sense and would negate the need to have high idle instances or stripping out frameworks like Guice and Objectify...but if at some point in the future as the code base grows even larger and we are consistently exceeding the hard limits then we have to either breakup the app as Brendan mentions or start stripping out frameworks as several of you have mentioned. Clear GAE was having a bad day when i posted this in the first place because we haven't had these issues of exceeding the hard limits on warmup times and we've only been adding to the code base since. Maybe it was just that day, but the other question that I asked in my original post was why were our Staging cold-startup times faster than our Production warm-up requests? I could understand a few more seconds for warmups, but the warmups in production on F4s were exceeding the hard limits and the cold starts on our staging environment on F1s were happening in like 30 seconds. Some other part of the infrastructure must be used during warmups and that part must have been having a really bad day. Why else would warmup requeests on F4s be able to lose to cold starts on F2s? Rock on, -Hardwick On Wednesday, July 25, 2012 6:28:01 AM UTC-4, Andrei Volgin wrote: I have a large GWT app with over 50 complex data entities with very complicated relationships between them. There are over 100 RPC methods, and I use various GAE services (Users, Memcache, Blobstore, Images, and Mail). My new instance startup time ranges from 4 to 5 sec on F1. Sometimes it goes a little higher. The lowest was 3.6 sec. I use the low-level API and no DI. Productivity tools are nice, but they have their cost. In any case, if Jeff's suggestion is implemented (dispatch only to warm instances), it would not matter if the start up time is 4 sec or 40. I followed this discussion closely, and to me it is obvious that the existing GAE mechanism is inferior to what Jeff is proposing (for any number of instances). There may be a good technical reason why the GAE team chose their current implementation. It is also possible that GAE was built by engineers who did not think in terms of web user experience, and who believed that a rare 30 seconds delay is the acceptable price to pay for some other benefits of the current implementation. This proposal was already starred by many people (issue 7865). It would be nice if the GAE teams takes another look and tells us whether we should expect any changes. I am sure a lot of app developers would like to know whether they should optimize for fast cold starts. These optimizations are expensive (both in direct development time and in lost productivity), so this is not an idle question. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ncTsPLWS7JUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Wed, Jul 25, 2012 at 8:42 PM, David Hardwick david.hardw...@bettercloud.com wrote: Amazing, Andrei. If you can get those startup times going with GWT and the complexities you mentioned in your app, then clearly stripping out DI and using low-level API (and likely jar'ing everything) is a mega contributor to reducing startup time. I would hate to have to spend the team's time stripping that stuff out, i would just pay for higher idle instances until request #7865 gets implemented. I just want to put this in proper perspective - we should expect Andrei's app to startup quickly. GWT-RPC is actually great for startup time because there is only a single backend servlet to start up; there's no need to build a sitemap by loading classes and looking for annotations (ie, JAX-RS). Similarly, the low-level API has zero startup overhead. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Also None of your references are weak? Do you just hate garbage collectors? That won't help much with startup, but again it would reduce your need to startup as often. You do a lot of String + String.. use stringbuffer instead You do a lot of .Trim when you should use Vector And yes in a few spots that would change your warm up. (not as much as it would drop your memory foot print) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I blew up or created some circular reference because it's late and I had a lousy week... I'm not familiar enough with the code, but you have several objects whose creation should be moved to the top of your code. And a few places you should swap to mutable returns. The mutable returns won't speed up startup, but will drop your memory usage significantly. Though it looks like a crap ton of memory is being allocated, more than should be needed, for what I can tell is going on. Do your logs have a lot of Soft Memory limit errors? There is what I would call a bug in GAE where if your only instance is killed by the Soft memory limit that the replacement instance loads really slow. I think because the pending request queue is transferred to the new instance. I think your biggest win would be in lazy creating the objects and classes that you don't use much. Next is Threading your large loops. Caching your Hashes. Changing your trims to vectors, and your Concats with + to use stringbuffer. Weak the stuff that is temporary variables so they don't get clung to. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Brandon. This pit is just getting deeper and deeper. When are you
going to give up?
On Mon, Jul 23, 2012 at 10:55 PM, Drake drak...@digerat.com wrote:
Really?
You don't think that doing lazy class loads would speed up your startup?
The JVM lazy loads classes automatically. This is just how it works.
Did you benchmark Dotted Imports vs full import, or less dotted? I
knocked 1s of the startup just by * importing objectify, that you took all
of but did so in 6 places.
Wow, I even gave you the answer ahead of time, and you *still* jumped
into the trap.
Here's a little experiment for you:
-- File m1/Main.java -
import java.util.*;
public class Main {
public static void main(String[] args) {
System.out.println(new Date() + : + new Random().nextInt());
}
}
-- File m2/Main.java -
import java.util.Date; import java.util.Random; // on same line so
line# info doesn't change
public class Main {
public static void main(String[] args) {
System.out.println(new Date() + : + new Random().nextInt());
}
}
Notice that the only difference is the wildcard import vs the
fully-qualified import.
Compile each. Diff the resulting classfiles.
Good work on that startup time improvement.
I'm pretty much done here.
Jeff
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Odd. I get different results than you do. Quite different actually. .* in the sandbox is slower (by a lot) .* on f4 is faster Also, I think you are very wrong about dynamically/lazily loading classes. Since my version warms up in 2.5s on an F2. (though I have an initialization missing somewhere cause when I do places I get an error about an object) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I like that you use +'s to concat strings, it shows a real lack of experience doing optimizations since that is the very first thing on every list. System.out.println(new Date() + : + new Random().nextInt()); At least most of your code uses objects correctly I didn't find any instances of places you did a type conversion functionally. That is usually one of the lowest hanging fruits, because it creates additional objects, and is a memory hog. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
I feel like I need to keep following up just in case a Java newbie takes this advice seriously. On Mon, Jul 23, 2012 at 11:14 PM, Drake drak...@digerat.com wrote: Also None of your references are weak? Do you just hate garbage collectors? That won't help much with startup, but again it would reduce your need to startup as often. That makes absolutely no sense. The app doesn't cache data in instances; there's utterly no point to using weak references. You do a lot of String + String.. use stringbuffer instead this + that is compiler syntactic sugar for using a StringBuilder to assemble the string. There is no difference between the two. It's probable that I could do the same trick I did with imports and create with two classes that generate identical bytecodes, but figuring out the exact pattern is more effort than I care to spend at the moment. Also, never use StringBuffer; it's a legacy class from JDK 1.0 which unnecessarily synchronizes method calls. StringBuilder is the replacement. You do a lot of .Trim when you should use Vector I'm genuinely afraid to ask, but... huh? Vector? (!!!) Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
That makes absolutely no sense. The app doesn't cache data in instances; there's utterly no point to using weak references. Your app runs for multiple seconds, you get garbage collection in that time and since you also feel no need to destroy objects you aren't using... this would be a lazy way to deal with it. this + that is compiler syntactic sugar for using a StringBuilder to assemble the string. There is no difference between the two. It's probable that I could do the same trick I did with imports and create with two classes that generate identical bytecodes, but figuring out the exact pattern is more effort than I care to spend at the moment. No, the intermediary objects are different and you have a much larger memory foot print using loops with + than stringbuilder For Print this + that you are essentially correct, but for 1000+ loops the difference is huge. Typically for small stuff it isn't worth messing with but you have some long loops of this. Also, never use StringBuffer; it's a legacy class from JDK 1.0 which unnecessarily synchronizes method calls. StringBuilder is the replacement. Yes I misspoke. I have been up for 48 hours and would be sleeping except someone said I couldn't do hello world in 1.5 seconds and I missed it by 100ms on an F1, so I didn't want to be slow getting response to this. You do a lot of .Trim when you should use Vector I'm genuinely afraid to ask, but... huh? Vector? (!!!) Vector resizes an object without creating an intermediary or placeholder object, so there is nothing for garbage collection to deal with, and you don't have to create a new object which gets renamed to the old object. (which is what trim does) Again a Pain in the ass for doing a trim on something once, but when you have it in a for loop it adds up. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Oh, and you are right the biggest slow down is grabbing data via HTTP. (usually is) But that doesn't seem to be hurting concurrency. The obvious question. Why aren't you mirroring the site so that you have your own fast copy? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Tue, Jul 24, 2012 at 12:00 AM, Drake drak...@digerat.com wrote:
I like that you use +'s to concat strings, it shows a real lack of
experience doing optimizations since that is the very first thing on every
list.
System.out.println(new Date() + : + new
Random().nextInt());
You're either trolling me or have yet another wild misconception about
Java. This one is really easy to disprove though:
--- Stringy.java ---
public class Stringy {
public static void main(String[] args) {
String foo = def;
System.out.println(abc + foo);
}
}
$ javac Stringy.java
$ javap -c Stringy
Compiled from Stringy.java
public class Stringy extends java.lang.Object{
public Stringy();
Code:
0: aload_0
1: invokespecial #1; //Method java/lang/Object.init:()V
4: return
public static void main(java.lang.String[]);
Code:
0: ldc #2; //String def
2: astore_1
3: getstatic #3; //Field java/lang/System.out:Ljava/io/PrintStream;
6: new #4; //class java/lang/StringBuilder
9: dup
10: invokespecial #5; //Method java/lang/StringBuilder.init:()V
13: ldc #6; //String abc
15: invokevirtual #7; //Method
java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
18: aload_1
19: invokevirtual #7; //Method
java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
22: invokevirtual #8; //Method
java/lang/StringBuilder.toString:()Ljava/lang/String;
25: invokevirtual #9; //Method
java/io/PrintStream.println:(Ljava/lang/String;)V
28: return
}
I'm pretty sure you can read that.
Jeff
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I can't get over how much memory this thing uses before it does something. Like just warming up I'm at 248M of memory. On an F2 I can do like 5 places and it soft memories and restarts. On an F4 I do about 350 How many requests are you seeing. Why do you hate garbage collection so much? I don't live in objectify, is it putting everything in an instance cache? Each request seems to add about 1.5Megs to the used memory. I can't see any difference (300ms) between load speed on an f2 and an F4 That usually means you are object creation bound, but you aren't front loading the Object creation, so that seems unlikely. (though you aren't doing lazy classes so maybe you are creating every object in your entire code at startup?) Tell me again why you are against Lazy Classes. Like I am the 3 year old you seem to think I am? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
What does your usage profile look like? It looks like if I kill off all the places code I can get this to 86 megs after warm up. If you put places on a backend, it looks like you could serve this off of an always one F1 backend, and alwasys on F1 frontend. And your warmups would be about 2.5 seconds, and you would have a very minor code tweak. Like you would access places like it was an API. Places should scale well since it isn't CPU dense. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Tue, Jul 24, 2012 at 12:37 AM, Drake drak...@digerat.com wrote: I can't get over how much memory this thing uses before it does something. Like just warming up I'm at 248M of memory. I'd start looking at what you've changed, or some defect in your test environment. https://img.skitch.com/20120724-cwa7r2g9x774aj4rb6xgmbiywa.jpg This is the production system in normal use, even with appstats turned on. There's no instance cycling (other than normal load expansion and contraction). You're doing something wrong. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
That's on your stock code. No changes. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Ok so it works its way down to the same levels you are seeing it takes 3-5 minutes. That's a first for me as well. In your production can you hit shutdown an app, hit it and see what the memory is 30 seconds after start? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I am crashing now. Splitting out the app I can make it load much faster. That seems to work. You would have two apps one living only on a backend, and accessed via an api (places) This gets your warm up way down, and shouldn't cost much more money. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
https://img.skitch.com/20120724-xdwyfjbfe7pxeyh2aa98938yxd.jpg No memory spikes. Jeff On Tue, Jul 24, 2012 at 12:51 AM, Drake drak...@digerat.com wrote: Ok so it works its way down to the same levels you are seeing it takes 3-5 minutes. That's a first for me as well. In your production can you hit shutdown an app, hit it and see what the memory is 30 seconds after start? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Am Donnerstag, 12. Juli 2012 18:26:40 UTC+2 schrieb David Hardwick: Hello, I realize there's been a lot of discussion on startup times exceeded on this forum recently, but wanted needed to post this experience we had this morning to keep the attention on this important issue. We uploaded a point release of our app to a not-live version this morning and, of course, we were going to click around on that instance to make sure it's all kosher before making that version live. The warm-up requests for the not-live version were exceeding the deadline limit of 60s... __and__we__are__on__F4s__!_!. However, the LIVE version of the app crashed too, 500 server errors, instance counts went to zero, all sorts of whacky stuff was seen in the control panel. All that happened to our LIVE version without when all we did was upload another non-live version and hit it with a single request...did I mention we were on F4s? ;-) Does the failure of any instance to exceed the 60s limit take down all instances to include live one? We did a few things as quickly as possible since our live application was down, so clearly we didn't have the time to take the scientific approach of only changing one thing at a time and wait to see if it that did it. We... 1. Switched from F4s to F2 (i figured if this would least get us on some new servers/instances) 2. Increased max idle instances from 1 to 2 (with F4s running, I'm fine with having just 1 idle instance and not at all happy about paying for 2 idle instances, so maybe we'll just increase this prior to deployments and then back down again after the deployment succeeds until we know more) 3. Made the recently uploaded version live (hey, why not, the production app was down for 10 minutes, so how much more harm could we do?) We use GWT and Guice, we jar everything (as I have been paying attention to this startup time discussions for quite some time now. We are also considering switching our Guice libraries to a non-AOP version as we saw suggested in another blog since we just need the injection. Any insight, and I'm all ears! app_id=s~myflashpanel Regards, -Hardwick -- *We make Google Apps even better.* *David Hardwick* *CTO* david.hardw...@bettercloud.com *Signature by Flashpanel http://flashpanel.com/* *See us in Mashable: Growing Up Google: How Cloud Computing Is Changing a Generationhttp://mashable.com/2012/04/30/generation-growing-up-google/ * -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2M_6SmcBWdwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
You should really focus an general architecture of the app for optimizing...havn't looked into the provided example. Your micro-optimizing suggestions for Java ...I think all Java guys are like ROFL now. Star visa Full Imports, String Concatenations in this example visa StringBuilder Really, if I carefully craft line numbers or embed no debug info in classes I can generate byte-identical class files with either import and string style. Import statements are not stored at all in Java class code and this string+string in one expression automatically use the StringBuilder pattern. you can lazy load classes - I just fear you mean something different ;) and it's the same problem like separating end points into micro apps - complex business code doesn't work that way - you have to load all that stuff anyway. Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz: I like that you use +'s to concat strings, it shows a real lack of experience doing optimizations since that is the very first thing on every list. System.out.println(new Date() + : + new Random().nextInt()); At least most of your code uses objects correctly I didn't find any instances of places you did a type conversion functionally. That is usually one of the lowest hanging fruits, because it creates additional objects, and is a memory hog. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
On 5 hours now of sleep so I can form sentences: Your micro-optimizing suggestions for Java ...I think all Java guys are like ROFL now. Star visa Full Imports, String Concatenations in this example visa StringBuilder Y'all'd be right if this was 100 lines of code in Java 101. What you aren't looking at is that what is slow in this code set is object creation and object recycling. You are right that Import Io.* And Import Io.Blah Import Io.Blah2 Results in the same code, What doesn't result in the same code is doing import Io.Blah in classa and import io.blah2 in classb and io.blah3 in classC because it changes the order and timing of the Object and imported Class Creation. + doesn't matter if you do it once, but at one point this code is doing up to 15k +'s in a loop, and + doesn't do efficient recycling of temporary/intermediary objects. So laugh all you want, but there are serious gains in profiling the code and seeing where object and memory manipulation are sub optimal, especially when you have long loops. The problem I see with you guys is that you don't test, REAL application you test micro bits of code and don't look at the impact or history of how the code got to where it is. You say it doesn't matter in 4 lines of codes but you don't have THOSE 4 lines of codes in your app, you have 4 lines intermingled in with 80 other lines that change what is going on. There are times when just changing the order of your Imports can add or remove 200ms to your app. As to architecture changes. No one liked those. Don't calculate hashes if you are already in a session, use memcache and instance cache. Thread really long loops. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
With a shower so the sentences might be a little better. Lazy Loading classes is a variation on the same Object creation stuff. On warm up all you are doing is initializing Objects, and creating classes. If you lazy load those, whole branches of your code won’t do that until you need to use those things. As a result an “Empty” warm up can go from 15s to 2. your first request of type X will be 4s, and of type y will be 4s and Z will be 4s until you have paid 20% more than what the non-lazy version would be, but the advantage is that a “cold” request will be 2+4s not 15s This is especially useful when you have something like the “places” in this sample code that needs a whole bunch of objects that aren’t used anywhere else. And uses libraries no other part of the code shares. You aren’t making the code more efficient, and it sucks because you have to know what dependencies each bit of your code has or things explode (much like when you do threading) but when your goal is to reduce the maximum suckage for the end user and instead spread the suckage around. (like the walmart check out analogy) Lazy Object and Class creation can be a big win. Lastly, there are some really hierarchical objects in this code, which is great for Object oriented code writing and is easier on the dev, but I think that functionalizing some of this would speed up the code base pretty significantly as Objects with like 4 tiers in their hierarchy don’t store efficiently. If you do need those, create them with the hierarchy early rather than attaching to them later, because you get an intermediary copy of the old object in creating the new object. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz Sent: Tuesday, July 24, 2012 5:25 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! You should really focus an general architecture of the app for optimizing...havn't looked into the provided example. Your micro-optimizing suggestions for Java ...I think all Java guys are like ROFL now. Star visa Full Imports, String Concatenations in this example visa StringBuilder Really, if I carefully craft line numbers or embed no debug info in classes I can generate byte-identical class files with either import and string style. Import statements are not stored at all in Java class code and this string+string in one expression automatically use the StringBuilder pattern. you can lazy load classes - I just fear you mean something different ;) and it's the same problem like separating end points into micro apps - complex business code doesn't work that way - you have to load all that stuff anyway. Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz: I like that you use +'s to concat strings, it shows a real lack of experience doing optimizations since that is the very first thing on every list. System.out.println(new Date() + : + new Random().nextInt()); At least most of your code uses objects correctly I didn't find any instances of places you did a type conversion functionally. That is usually one of the lowest hanging fruits, because it creates additional objects, and is a memory hog. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
the same problem like separating end points into micro apps - complex business code doesn't work that way - you have to load all that stuff anyway. First, but not all at once. Which I addressed in Lazy Load, but since you are still attacking the micro-app / API model This sample app has a “Places” that is the slowest part of the code, and loads things nothing else uses. Spinning that off to an API handled by a backend would make a HUGE difference. Any functionalized code can be converted to standalone apps, and the overhead is that you have a wrapper, but the advantage is that you have huge memory and caching advantages. So any very slow memory intensive function can be converted to an API especially if it doesn’t need to modify globals. If it does need to modify globals you can do that on return. Say for example you have an imaginary app that does business stuff. Selling stuff and searching stuff. Searching works better the more in instance caching you can do. You might want an F4 just for the memory. And because of the shared memory 128 +128+128+128 doesn’t have the cache hit ratio of 1 512. Because of that an F4 with 100% of its available memory dedicated to caching search info your single purpose F4 could serve as many search requests as 8 F1s. but even when that isn’t the case if your “Idle” is 78 megs, an F2 will still idle at 78, but instead of having 50 megs for cache will have 178. Even better if that is the only place you need a wrapper for the complex queries to the datastore this is the only place you need Factory or Objectify. The selling stuff app is likely the only place you need Crypto libraries and because you want to cache sessions again having more memory in instance dedicated to the task of session management increases your cache hits by orders of magnitude. In both cases reducing the number of imports and the frame works required for both of these “microapps” is going to reduce Cold start time. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz Sent: Tuesday, July 24, 2012 5:25 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! You should really focus an general architecture of the app for optimizing...havn't looked into the provided example. Your micro-optimizing suggestions for Java ...I think all Java guys are like ROFL now. Star visa Full Imports, String Concatenations in this example visa StringBuilder Really, if I carefully craft line numbers or embed no debug info in classes I can generate byte-identical class files with either import and string style. Import statements are not stored at all in Java class code and this string+string in one expression automatically use the StringBuilder pattern. you can lazy load classes - I just fear you mean something different ;) and it's the same problem like separating end points into micro apps - complex business code doesn't work that way - you have to load all that stuff anyway. Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz: I like that you use +'s to concat strings, it shows a real lack of experience doing optimizations since that is the very first thing on every list. System.out.println(new Date() + : + new Random().nextInt()); At least most of your code uses objects correctly I didn't find any instances of places you did a type conversion functionally. That is usually one of the lowest hanging fruits, because it creates additional objects, and is a memory hog. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
But back to the roots...long loops have nothing to do with class loading in java ;) imports and loops - we would be hard pressed to find a _relevant_ example. really... I don't believe you the Smart-Java-Guy story. 200 ms through import-reordering? nope...no imports in bytecode...just straigt bytecode with ful types: GET com.bla.C fieldname, INVOKE com.bla.D methodname methoddescriptor You know the GAE very well and for sure python and your cache stuff...but very often you generalize this for all and that gets really annoying. You can lazy load classes if you don't use their functionallity, thats right - this classes are not fully loaded at the beginning. But many annotation-scanning frameworks (DI / O/R mapper etc.) in Java don't work this way. Am Dienstag, 24. Juli 2012 16:48:36 UTC+2 schrieb Brandon Wirtz: On 5 hours now of sleep so I can form sentences: Your micro-optimizing suggestions for Java ...I think all Java guys are like ROFL now. Star visa Full Imports, String Concatenations in this example visa StringBuilder Y’all’d be right if this was 100 lines of code in Java 101. What you aren’t looking at is that what is slow in this code set is object creation and object recycling. You are right that Import Io.* And Import Io.Blah Import Io.Blah2 Results in the same code, What doesn’t result in the same code is doing import Io.Blah in classa and import io.blah2 in classb and io.blah3 in classC because it changes the order and timing of the Object and imported Class Creation. + doesn’t matter if you do it once, but at one point this code is doing up to 15k +’s in a loop, and + doesn’t do efficient recycling of temporary/intermediary objects. So laugh all you want, but there are serious gains in profiling the code and seeing where object and memory manipulation are sub optimal, especially when you have long loops. The problem I see with you guys is that you don’t test, “REAL” application you test micro bits of code and don’t look at the impact or history of how the code got to where it is. You say “it doesn’t matter in 4 lines of codes” but you don’t have “THOSE” 4 lines of codes in your app, you have 4 lines intermingled in with 80 other lines that change what is going on. There are times when just changing the order of your Imports can add or remove 200ms to your app. As to architecture changes… No one liked those. Don’t calculate hashes if you are already in a session, use memcache and instance cache. Thread really long loops. Am Dienstag, 24. Juli 2012 16:48:36 UTC+2 schrieb Brandon Wirtz: On 5 hours now of sleep so I can form sentences: Your micro-optimizing suggestions for Java ...I think all Java guys are like ROFL now. Star visa Full Imports, String Concatenations in this example visa StringBuilder Y’all’d be right if this was 100 lines of code in Java 101. What you aren’t looking at is that what is slow in this code set is object creation and object recycling. You are right that Import Io.* And Import Io.Blah Import Io.Blah2 Results in the same code, What doesn’t result in the same code is doing import Io.Blah in classa and import io.blah2 in classb and io.blah3 in classC because it changes the order and timing of the Object and imported Class Creation. + doesn’t matter if you do it once, but at one point this code is doing up to 15k +’s in a loop, and + doesn’t do efficient recycling of temporary/intermediary objects. So laugh all you want, but there are serious gains in profiling the code and seeing where object and memory manipulation are sub optimal, especially when you have long loops. The problem I see with you guys is that you don’t test, “REAL” application you test micro bits of code and don’t look at the impact or history of how the code got to where it is. You say “it doesn’t matter in 4 lines of codes” but you don’t have “THOSE” 4 lines of codes in your app, you have 4 lines intermingled in with 80 other lines that change what is going on. There are times when just changing the order of your Imports can add or remove 200ms to your app. As to architecture changes… No one liked those. Don’t calculate hashes if you are already in a session, use memcache and instance cache. Thread really long loops. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mV8RrorNY2oJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Tue, Jul 24, 2012 at 7:48 AM, Drake drak...@digerat.com wrote:
+ doesn’t matter if you do it once, but at one point this code is doing up
to 15k +’s in a loop, and + doesn’t do efficient recycling of
temporary/intermediary objects.
Did you even read the bytecode output I posted? Seriously, I've never
met someone who is given all the right answers to the test but just
keeps insisting on the wrong answers anyways.
Try compiling these two classes. They produce *identical* class files:
File m1/Stringy.java
public class Stringy {
public static void main(String[] args) {
String foo = def;
System.out.println(abc + foo);
}
}
File m2/Stringy.java
public class Stringy {
public static void main(String[] args) {
String foo = def;
System.out.println(new
StringBuilder().append(abc).append(foo).toString());
}
}
If you don't believe me that the JVM lazy-loads classes, you can
either run some experiments with -XX:+TraceClassLoading or just spend
a few minutes with Google. I've wasted enough time with this
discussion.
The honorable thing to do is apologize and star my issue:
http://code.google.com/p/googleappengine/issues/detail?id=7865
Jeff
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
And you get a new string builder each loop. If you use string builder and
recycle it you don't have to re-create the object. Which doesn't create so
much garbage.
You have the answer and you don't understand it.
Try compiling these two classes. They produce *identical* class files:
File m1/Stringy.java
public class Stringy {
public static void main(String[] args) {
String foo = def;
System.out.println(abc + foo);
}
}
File m2/Stringy.java
public class Stringy {
public static void main(String[] args) {
String foo = def;
System.out.println(new
StringBuilder().append(abc).append(foo).toString());
}
}
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Tue, Jul 24, 2012 at 10:40 AM, Drake drak...@digerat.com wrote: And you get a new string builder each loop. If you use string builder and recycle it you don't have to re-create the object. Which doesn't create so much garbage. You have the answer and you don't understand it. Just give it up and star the issue, mkay? http://code.google.com/p/googleappengine/issues/detail?id=7865 Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
I didn't read in full everything here, but using StringBuilder is good old
optimization, strange many don't know it here...
http://www.venishjoe.net/2009/11/java-string-concatenation-and.html
Having said that it seems that both of you are kind of right, recent JDK's
compile '+' into StringBuilder, but not in all cases, so you have to be
carefull.
For example, this would be compiled into builder:
String ab = a + b;
but this will be compiled into 2 StringBuilder...
String ab = ab;
ab += a;
ab += b;
вторник, 24 июля 2012 г., 20:40:10 UTC+3 пользователь Brandon Wirtz написал:
And you get a new string builder each loop. If you use string builder and
recycle it you don't have to re-create the object. Which doesn't create so
much garbage.
You have the answer and you don't understand it.
Try compiling these two classes. They produce *identical* class files:
File m1/Stringy.java
public class Stringy {
public static void main(String[] args) {
String foo = def;
System.out.println(abc + foo);
}
}
File m2/Stringy.java
public class Stringy {
public static void main(String[] args) {
String foo = def;
System.out.println(new
StringBuilder().append(abc).append(foo).toString());
}
}
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/CDZcdLF0bIgJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Personally, I don't care much for the who's-a-better-guru argument, doesn't get us any closer to a solution. App Engine proves Joel Spolsky's all abstractions are leaky statement. An abstraction is nice when it works but you'd better know what's going on under the hood when it doesn't. Most of us are willing to put in extra effort to make our apps GAE-friendly upfront, but everything's about balancing available resources given our specific app and situation. If we weren't willing, we'd be using something else. Some have had more opportunity than others to refine their approach, and some apps suit the platform better than others. But we'll do better if we respect the opinions of the other participants a touch more, or at least try keep comments about the ball, not the man. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/NzEYDO9XK-gJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Well I love the who's-a-better-guru argument. I've learned a ton reading this thread, I hope they keep going at it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bXo-VqpW7LQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
It's not a who's a better guru. I'm sure Jeff writes better Java than I do. But This aint your grandmother's java. I participate in a HipHop (php) discussion group too, and people will say Well XY and Z aren't possible with out ZEND well, no they are just hard without it. If you are running Wordpress for your personal blog you can do PHP on DreamHost. If you are Facebook you use HipHop. Writing a small App with no performance requirements you can use PHP and Zend. But at scale you need that Simpler version of the language. Even Python on AppEngine isn't Python you don't get all the C stuff. The Fortran stuff. You are Accessing Google API's via a language. You can pick your language, Python, Java, Go, but you are writing in AppEngine it's there world we just play in it. The good news, if you write to those rules and wrap the API's in Class/functions of your own (which you need to make things work in the sandbox), your code will be scalable portable, and happy on anyone else's platform. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Paul v Sent: Monday, July 23, 2012 8:38 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! Well I love the who's-a-better-guru argument. I've learned a ton reading this thread, I hope they keep going at it. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bXo-VqpW7LQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Monday, July 23, 2012 10:38:12 AM UTC-5, Paul v wrote: Well I love the who's-a-better-guru argument. I've learned a ton reading this thread, I hope they keep going at it. Just out of sheer curiosity, what exactly have you learned from reading this read? Really, the entire discussion between Mr. Wirtz and Mr. Schnitzer distills down to optimizing your code as much as possible. Which is a really good idea regardless of what hosting platform you pick. And frankly, this discussion hinges entirely on instance cold start issues. If you have enough money to keep spare idle instances sitting around, or your traffic is steady and constant (as opposed to spiky and unpredictable) you really won't hit this problem very much. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/V4-frxZVYSEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
On GAE it's not just the startup.. Well it is, but for more reasons. You have less memory too. So someone's really optimized framework might be AMAZING for CPU optimizations, but the cost of adding it from a memory standpoint causes you to hit the soft limit instance death more often. Which causes you to cold start A LOT more because the scheduler didn't spin the instance, the instance ceased to exist. And frankly, this discussion hinges entirely on instance cold start issues. If you have enough money to keep spare idle instances sitting around, or your traffic is steady and constant (as opposed to spiky and unpredictable) you really won't hit this problem very much. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
I can't believe I'm keeping this thread going, but... On Sun, Jul 22, 2012 at 10:15 AM, Drake drak...@digerat.com wrote: And your absurd proposal that all 326 of my url endpoints should be separate applications... Just. Wow. If you had actually read what I posted I said that you should group by task type, and used class so that you had many smaller apps, that were optimized for like tasks and minimizing classes. Your exact words were Each of what you call and End point should be a micro app. A single purpose App that handles one type of request. But let's move on. I was a tech lead in EA's online platform group - hundreds of services among thousands of machines - so don't lecture me about service oriented architectures. And that's not even close to the most impressive thing on my resume. Now, can we put our dicks back in our pants? There is a proper scale for breaking down a monolithic application into separately deployed modules, but this isn't it. In real-world business apps (the kind that - unlike your CDN - represent 90% of GAE users), transactional boundaries place limits on how you can decompose an application since you can't (without getting *really* crazy) run a transaction across instances. I'm glad that you've figured out how to tightly optimize serving static content through GAE. I'm sure your CDN code is lovely and well suited to its painfully narrow task. But you're trying to generalize a hack - write at the lowest API level all the time - to projects that are simply too complicated for that kind of hack. With broader experience, you would understand this. If you want it to be constructive, post your code and I'll whack 30% off your start time just to get you to stop griping about how it is not possible to speed up start times because Google sucks. Cute. I'll make you an equally pointless offer: If you want to fly out to San Francisco, I'll show you the code for a modern business application and you will see for yourself why your optimization strategy is farcical. The difference between your pointless offer and my pointless offer is that if you actually took me up on it, I could deliver. However, if you want to play a game, try optimizing this demo app: http://www.motomapia.com/ The code is linked from the header. Typical instance startup time is 10-15s. Here's the rules of the game: * The @Path annotations must keep working * The @Transact(TxnType.REQUIRED) annotation must keep working * Datastore access must be through typed POJOs. Let me predict the response: Too many frameworks! Use low-level API! Use web.xml! Manipulate transactions by hand! All of that works in a toy demo but not in an app with hundreds of endpoints, dozens of polymorphic entity classes, and complicated transactional logic. Essentially, your optimization strategy is to write code in a way that does not scale to the complexity of a modern business application. Pure fail. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
August 25th I have an app launch. It is no small thing. My CDN is a Toy in the view you have of it. But the internal version is designed to make all the other stuff work. It is a load balanacing url handling, code modularizing tool. But you keep slamming my CDN because you like those other CDN guys but it is just an enabling technology for everything else I do. Because of that tool I can do www.myappp.com/user/12354 as an app in python, and www.myapp.com/products/12345?view=json in Java, and www.myapp.com/search/frogs as Go. And if I really want, /checkout/ can go to PHP on Dreamhost. That gives me infinite more control. It also lets me share databases because java.myapp.appspot.com can share data with python.myapp.appspot.com . So that comment that you can't convert your code to Python... You don't have to convert all of it, or all of it all at once. Maybe my bias is that because without such tech you have to do web.xml stuff or that it is hard to modularize your app that I'm so jaded by it's awesomeness that I can't see past, but... Typically when a company asks me to refactor code the first thing I do, is break their code in to groups that share classes/imports/frameworks and create 3-4 apps from those, and then use a load balancer or a front end proxy or a Templated API wrapper to make that app faster and more scalable. Step two is convert as much as possible to lowlevel api. Step 3 is to strip everything out of every framework that they aren't using. Step 4 is to create flatter versions of their data to make queries simpler, faster, and more friendly to the lowlevel API Step 5 is to prioritize writes and reads so that things can be delayed if they aren't for realtime data Step 6 is to optimize start ups using serialization of configuration strings, and other data. Pushing any file streams to Memcache Step 7 is to create a task database to create precaclulated versions of common requests, and to handle cron and steady state tasks. Step 8 is to make determinations about lazy loading. (I try hard to not lazy load because it makes your app harder to profile, because if you lazy load a large frame work you may over run the soft memory limit. If you do everything up front then you know what your steady state memory usage is) Step 9 is to self manage parts of the memory. Rather than relying on garbage collection start throwing away memory that you aren't using. Step 10 is to profile decision trees, to optimize order of operations. Step 11 is to manage threading (this is often late in life because it makes debugging hard, and the gains are often not amazing in apps with lots of simultaneous users) Step 12 is to strip all the debugging, logging and non-esential stuff from the code. Rinse and repeat. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Mon, Jul 23, 2012 at 5:26 PM, Drake drak...@digerat.com wrote: But you keep slamming my CDN because you like those other CDN guys but it is just an enabling technology for everything else I do. I really am not slamming your product; for all I know it's amazingly wonderful. What I object to is that you are taking optimizations peculiar to one very unusual, specialized app and declaring them as universal truths for everyone - oh, and if we object, we're being lazy or whiny. The technical challenges of your CDN are very different than the technical challenges of a business application. It may make sense for you to break your app into tiny independent pieces, but you can't do that when you have crosscutting concerns like transactions or elaborate data dependencies. It may make sense for you to model your domain with the low-level API, but it would be insanity to try this in an app with hundreds of entity types and hundreds of thousands of lines of code, not to mention multiple engineers coming on and off the project over years. All these frameworks we use have evolved for good reason, because they make it easier to write code and scale the complexity of an app. It's not a solution to say just don't use them simply because you have managed to build one particular type of app using bare metal programming. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. I don't even promote the damn thing it helps with my old SEO clients. Our analytics and Ad management platforms are full on business apps. We do real database stuff. We did 1million write ops in an hour the other day. ( I can post pictures) we do queries and reports, and we use the lowlevel API for all that. (we can do a Join in Lowlevel we don't even use GQL ) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Monday, July 23, 2012 6:37:15 PM UTC-7, Brandon Wirtz wrote: CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. I don't even promote the damn thing it helps with my old SEO clients. Our analytics and Ad management platforms are full on business apps. We do real database stuff. We did 1million write ops in an hour the other day. ( I can post pictures) we do queries and reports, and we use the lowlevel API for all that. (we can do a Join in Lowlevel we don't even use GQL ) Unfortunately, both Jeff and I have prior experience with working with people just like Brandon. All talk (mostly gibberish), but no listen and no do. Brandon, your next response on this impossibly long thread should be a link to a pull request showing how you've made motomapia start up 30% faster while playing by the rules Jeff sent out. Nothing more, nothing less. Any other response will be proof that you are all bogus talk and no action. It's time to put up or shut up. jon -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AwjWPmk8szMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Skipping the start time issue since there doesn’t appear to be a single lazy load in the whole project…. And not having time to sort through the duplicate imports which are definitely killing load time... And ignoring the fact that you are doing Dotted imports which add up to most if not the entire import which is slowing things down even more……. (Stars aren’t always bad because in GAE the IO is more of an issue than the compile time) And that those 3 things would likely speed load time by a lot, if not all of my 30% goal…. The PolyLines String builder isn’t threaded, and profiling the app that seems to be really frakking slow. Create 4 threads and have each take ¼ of the array. I haven’t implemented the code but that should be about a 45% increase in speed. Am I required to actually check the changes in? or can you guys handle threading the slowest part of the code? ( I assume I’m optimizing for low volume and reduced latency) Can I replace your Sha256 with something else? That is also eating a lot of CPU time and I can’t really tell from the 5 minutes looking at the code why you are doing that. Sha1 should suffice and would buy you 50% increase in speed on those requests that use it. And as far as I can tell that looks like all authenticated requests? It also doesn’t look like you are caching your HashCalculations? Based on privacy and such you may not want to memcache those… but at least store some number of them in instance memory. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Jon Stevens Sent: Monday, July 23, 2012 7:35 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! On Monday, July 23, 2012 6:37:15 PM UTC-7, Brandon Wirtz wrote: CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. I don't even promote the damn thing it helps with my old SEO clients. Our analytics and Ad management platforms are full on business apps. We do real database stuff. We did 1million write ops in an hour the other day. ( I can post pictures) we do queries and reports, and we use the lowlevel API for all that. (we can do a Join in Lowlevel we don't even use GQL ) Unfortunately, both Jeff and I have prior experience with working with people just like Brandon. All talk (mostly gibberish), but no listen and no do. Brandon, your next response on this impossibly long thread should be a link to a pull request showing how you've made motomapia start up 30% faster while playing by the rules Jeff sent out. Nothing more, nothing less. Any other response will be proof that you are all bogus talk and no action. It's time to put up or shut up. jon -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AwjWPmk8szMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Brandon, you have absolutely no idea how the JVM works. On Mon, Jul 23, 2012 at 8:41 PM, Drake drak...@digerat.com wrote: And not having time to sort through the duplicate imports which are definitely killing load time... And ignoring the fact that you are doing Dotted imports which add up to most if not the entire import which is slowing things down even more……. (Stars aren’t always bad because in GAE the IO is more of an issue than the compile time) And that those 3 things would likely speed load time by a lot, if not all of my 30% goal…. Try it. I was tempted wait and see what kind of results you got (not as much as I had hoped would have been hilarious), but that would just be mean. Those changes aren't going to help because they have no effect on compiled code. Literally, the Java compiler produces the *exact same output*. This isn't Python; Java imports are just syntactic sugar to the compiler. I wasn't quite sure what to make of not a single lazy load in the project so I'll just assume you don't have the faintest clue how classloading works. But go ahead and suggest some specific code changes that you think are going to make the app startup faster. The PolyLines String builder isn’t threaded, and profiling the app that seems to be really frakking slow. Create 4 threads and have each take ¼ of the array. I haven’t implemented the code but that should be about a 45% increase in speed. Just. Wow. First of all, the polylines string builder has no effect on instance startup time. Try hitting urls which don't render polylines... same 10-15s startup time. Second of all, you have a broken mental model of the relative performance of various operations. It might be different in Python, but I kinda doubt it. The computational cost of creating, synchronizing, and joining a thread (plus all the security checks that Google adds) are at least an order of magnitude more, and probably two or three orders of magnitude more expensive than the cost of assembling a polyline. Seriously, try it. Especially try it in production. Am I required to actually check the changes in? or can you guys handle threading the slowest part of the code? ( I assume I’m optimizing for low volume and reduced latency) You can submit a pull request if you actually come up with a real improvement to startup time. Can I replace your Sha256 with something else? That is also eating a lot of CPU time and I can’t really tell from the 5 minutes looking at the code why you are doing that. Sha1 should suffice and would buy you 50% increase in speed on those requests that use it. And as far as I can tell that looks like all authenticated requests? It also doesn’t look like you are caching your HashCalculations? Based on privacy and such you may not want to memcache those… but at least store some number of them in instance memory. This has nothing to do with startup time. And isn't a problem. And what you say here doesn't really make sense anyways. If you're wondering why /places requests are slow, I already know the answer: Wikimapia is slow. Not my problem: https://img.skitch.com/20120724-q517rqfg7mfhbndnbn4nfmwn89.jpg Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Really? You don't think that doing lazy class loads would speed up your startup? Did you try? Did you benchmark Dotted Imports vs full import, or less dotted? I knocked 1s of the startup just by * importing objectify, that you took all of but did so in 6 places. No, you never try anything. I think I made it clear that threading wouldn't change the startup, it would speed up the app. And since your issue was startup plus response time and it appears that can take 6-8 seconds and tests at 2-5 after threading that would appear that would make your QoS much better, and based on the load you were describing that would seemingly reduce your number of starts, and need for more instances. You aren't doing hash caching. That's a huge, easy speed up. Even if you don't downgrade to sha1, that will make a huge difference in your need to spin up more instances. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I didn't need task queues/defer because I was self managing queues and in fact do that more than using task Queue still because it lets me do more process shaping. So now short term tasks are defer, and long term tasks go in to a self managed bucket that fires based on current load as determined by the number of instances active. * How do you persist data? (low-level, jdo, objectify, etc) Actually I use a multi-approach based on the work being done. Python has NDB, CachePy, and a number of things that Java seems to missing good analogs for, but using the lowlevel API gets things up and running and does the majority of the heavy lifting. When we don't want to have to manage a bunch of code to handle stuff that is more complex or do things that are less time sensitive like deep queries for single users, we use JPA. (I think JDO is mostly dead.) * How many entity kinds/classes do you have? A shit ton. For the analytics app we built we found out that you are limited to 65535 and that Dynamic is not as dynamic as you might think. * How do you manage URL routing? (ie, jsp files? web.xml? JAX-RS) How quaint. Did you miss that I said one type of task per app/backent? Web.XML at the individual app, but path routing is handled by the caching/loadbalancing that is done using the python version of CDNInabox for most our apps, and Squid on another service for some of the other Apps. No App ever is exposed directly to a user request, they are all proxied. (ps this also allows us to do much of our own QoS manipulation because we can send a request that is taking too long to an F8 that has multiple apps installed on it and can be anything it needs at a moments notice. * How many URL endpoints do you have? Per app? Or In total? In many cases we have endpoints per domain in multi-tenant apps, and we are running 10k plus domains. * How many total java classes in your application? What is total size? * How many jars in your WEB-INF/lib? What is total size? Per app? Maybe 20. In an application deployment couple thousand. Last time I looked our code base was nearly 800 megs, with 80% of apps at less than 10 megs Jeff, my toys are bigger than your projects. My unique indexed pages in Google is over 1 Billion. I likely spend more in Datastore file costs than you spend hosting your entire client base. -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer Sent: Saturday, July 21, 2012 9:27 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! Brandon, you talk a lot of shit for a guy who only discovered the task queue 6 months ago. Everyone who thinks they have the secret to starting up a Java app in 5s, answer these questions and prove that you're running more than a toy: * How do you persist data? (low-level, jdo, objectify, etc) * How many entity kinds/classes do you have? * How do you manage URL routing? (ie, jsp files? web.xml? JAX-RS) * How many URL endpoints do you have? * How many total java classes in your application? What is total size? * How many jars in your WEB-INF/lib? What is total size? Having done a fair bit of experimentation around this issue, I know that you're talking out of your asses. Clever suggestions like rewrite app in Python aren't nearly as clever as you think they are. Suggestions that you can optimize startup time by some combination of caching, task queues, or threading merely indicate total lack of understanding of the problem in the first place. To the people paying attention to this thread who are actually trying to build real world apps on Appengine: Brandon does not speak for the GAE team. Neither do I, but I occasionally talk to people that do. I know that Google really does want Java apps to run well on GAE. They know about this issue and care about it - maybe even enough to do something about it. It *is* constructive to whine politely about startup time because there are many issues competing for the team's time, and our feedback helps them prioritize their efforts. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google- appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
I really don't care who has the shiniest toys in this playground, bottom line is that I consider every minute spent getting my app to load faster a complete waste of my time. Adding business logic, minimizing user request times and costs, that's what I care about, not jumping through hoops to get around some design flaw that Google could easily fix if they chose to. If you have a zillion apps and a huge budget and/or team, and it is cost effective to do a crapload of optimizations, good for you! but I suspect most of us just want to get our apps to work reasonably efficiently, in as little time, and with as many of the tools we already know and love as possible, so we can move on to our next project, client, idea, etc. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/gmXX3iD35ccJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
And this would be why we don't use contractors. Optimizations aren't for Google they are good practice. When you run on 600 mhz with 128 Megs of ram, you see the optimizations when you have 3.2 Ghz * 8 and 64 gigs of ram, you can't hardly profile an app. That doesn't mean you can't make it twice as efficient. The tools you know and love suck. You just don't know it because you are spoiled by big computers. I love my '50 Merc and my '66 Stang, but my MiniCooper has a lower cost of operation and has a faster 0-60 and top speed than either of the old Fords do. Because it is optimized. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Mauricio Aristizabal Sent: Sunday, July 22, 2012 12:28 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! I really don't care who has the shiniest toys in this playground, bottom line is that I consider every minute spent getting my app to load faster a complete waste of my time. Adding business logic, minimizing user request times and costs, that's what I care about, not jumping through hoops to get around some design flaw that Google could easily fix if they chose to. If you have a zillion apps and a huge budget and/or team, and it is cost effective to do a crapload of optimizations, good for you! but I suspect most of us just want to get our apps to work reasonably efficiently, in as little time, and with as many of the tools we already know and love as possible, so we can move on to our next project, client, idea, etc. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/gmXX3iD35ccJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Sat, Jul 21, 2012 at 11:32 PM, Drake drak...@digerat.com wrote: * How do you persist data? (low-level, jdo, objectify, etc) Actually I use a multi-approach based on the work being done. Python has NDB, CachePy, and a number of things that Java seems to missing good analogs for, but using the lowlevel API gets things up and running and does the majority of the heavy lifting. When we don't want to have to manage a bunch of code to handle stuff that is more complex or do things that are less time sensitive like deep queries for single users, we use JPA. (I think JDO is mostly dead.) This thread is about Java instance startup time. Enough about Python already. It's irrelevant. You've completely - and lamely - dodged ALL of my questions. You insinuate that you have sophisticated Java applications that start up quickly. I'm calling bullshit. Give me ACTUAL numbers for one ACTUAL Java application. * How many entity kinds/classes do you have? A shit ton. For the analytics app we built we found out that you are limited to 65535 and that Dynamic is not as dynamic as you might think. Generated kinds or classes don't count. I mean this literally, how many user-defined entity types does your Java app load? You're going to tell me you have Java apps that load 64k JPA classes? Not a chance. See, this is getting to the ACTUAL problem with the Java environment - classloading is slow. Entity classes must be loaded and introspected before datastore operations can start. Near as I can tell, the only way around this particular problem is to stay with the low-level api - which, like assembly programming, severely limits how sophisticated an application you can reasonably build. * How do you manage URL routing? (ie, jsp files? web.xml? JAX-RS) How quaint. Did you miss that I said one type of task per app/backent? I will be generous and assume for the moment that this makes sense for your particular application. At best you are arguing that you have a wacky application. You won't find too many people building business apps that way, especially when you have elaborate transactional logic. There are sometimes good reasons to compose an app out of more fundamental services, but this should be driven by application design - not a dumb limitation of how many classes you can pack into a JAR before instance startup times get out of control. Web.XML at the individual app, but path routing is handled by the If I went the web.xml route, it would map 326 servlets. Not really fun. There's a reason people started inventing web frameworks in the late 90s... because that kind of programming sucks when your app is even mildly complicated. * How many URL endpoints do you have? Per app? Or In total? In many cases we have endpoints per domain in multi-tenant apps, and we are running 10k plus domains. I was really asking about your unicorn java app that loads in 2 seconds. But I think you misunderstand what an 'endpoint' is. It's a particular code path; for example, http://example.com/thing/123, http://example.com/thing/456, and http://other.com/thing/789 are all the same endpoint because they all execute the same code. It's relevant because it is related to how many classes get loaded at startup, which has a strong effect on the startup latency. * How many total java classes in your application? What is total size? * How many jars in your WEB-INF/lib? What is total size? Per app? Maybe 20. In an application deployment couple thousand. Last time I looked our code base was nearly 800 megs, with 80% of apps at less than 10 megs I want those numbers for your unicorn sophisticated Java app that you claim starts in 2s. Jeff, my toys are bigger than your projects. My unique indexed pages in Google is over 1 Billion. I likely spend more in Datastore file costs than you spend hosting your entire client base. Yawn. The Royal Wedding site served more traffic and probably cost more than either of us will make in revenue all year, but it's 99.9% static content. Your application is highly atypical of business applications, and yet you myopically believe that we should all build our apps out of zillions tiny components that use the low-level persistence API and communicate through urlfetches and task queues. That's retarded, and when you eventually discover the transaction feature of the datastore you will understand why. Jeff P.S. One consequence of my particular career history is that I'm not impressed by people trying to prove their dick is bigger. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I will be generous and assume for the moment that this makes sense for your particular application. At best you are arguing that you have a wacky application. You won't find too many people building business apps that way, especially when you have elaborate transactional logic. There are sometimes good reasons to compose an app out of more fundamental services, but this should be driven by application design - not a dumb limitation of how many classes you can pack into a JAR before instance startup times get out of control. Web.XML at the individual app, but path routing is handled by the If I went the web.xml route, it would map 326 servlets. Not really fun. There's a reason people started inventing web frameworks in the late 90s... because that kind of programming sucks when your app is even mildly complicated. And this is where you prove your stupidity. This is how every large app on the planet works. A load balancer serves requests to the right app and each app is optimized for what it is supposed to do. Data is shared between apps using API's, Memcache, and Datastore. I know what an end point is. Apparently you don't. Each of what you call and End point should be a micro app. A single purpose App that handles one type of request. This lets you build my Unicorns. Fast little apps that do one thing, and do it fast. Also because they do only one thing they get to maximize their instance memory. Java gets REALLY slow and has more warm ups when you have 200 megs of code and 128 megs of ram. (as in it doesn't work, and if by some miracle it does make it past warm up typically you hit soft memory limit and it dies) You clearly are building Monolithic code in a world of Micro-instances. STOP IT. You will only have pain doing so. Build single purpose instances that do what they need to do. I'm sorry your app is too small for there to always be one instance doing each task. That's not my fault. Get more popular or something. In the mean time use lazy loads so that you can fake having instances that do one and only one thing by only loading enough stuff to do one thing until you need to do that other thing. I'm clearly not going to convince you to do anything, because you don't want to. You want somebody else to fix your problem so you won't even look at solutions. At which point I have to point out you are always going to be sad and miserable. PS Survivor Finale. And Comments aren't static. Nor are post to facebook. Nor are Oauth calls. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
200 1625ms 98kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6 70.162.197.131 - - [22/Jul/2012:01:38:53 -0700] GET /www.xyhd.tv HTTP/1.1 200 98752 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6 second.cdninabox-java.appspot.com ms=1626 cpu_ms=844 api_cpu_ms=0 cpm_usd=0.042899 loading_request=1 instance=00c61b117c382cd6d2d719025a767570ad95c0 I 2012-07-22 01:38:53.885 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application. I read that as 1.626 MS. What I did in that time on an F1. Initialized the App from nothing. Read the configuration out of Datastore (not even from Memcache) Did a URL Fetch to read the contents of XYHD.tv (like you doing your read from file) Regexed it to change all the paths from relative to absolute. Deferred a write to the datastore so I wouldn't have to do that for the next request that comes along in the next 30 seconds. Output the data. I still have 370 Ms to do something else. And that is on an F1. I think that counts as Hello world. -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer Sent: Sunday, July 22, 2012 1:18 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! On Sat, Jul 21, 2012 at 11:32 PM, Drake drak...@digerat.com wrote: * How do you persist data? (low-level, jdo, objectify, etc) Actually I use a multi-approach based on the work being done. Python has NDB, CachePy, and a number of things that Java seems to missing good analogs for, but using the lowlevel API gets things up and running and does the majority of the heavy lifting. When we don't want to have to manage a bunch of code to handle stuff that is more complex or do things that are less time sensitive like deep queries for single users, we use JPA. (I think JDO is mostly dead.) This thread is about Java instance startup time. Enough about Python already. It's irrelevant. You've completely - and lamely - dodged ALL of my questions. You insinuate that you have sophisticated Java applications that start up quickly. I'm calling bullshit. Give me ACTUAL numbers for one ACTUAL Java application. * How many entity kinds/classes do you have? A shit ton. For the analytics app we built we found out that you are limited to 65535 and that Dynamic is not as dynamic as you might think. Generated kinds or classes don't count. I mean this literally, how many user- defined entity types does your Java app load? You're going to tell me you have Java apps that load 64k JPA classes? Not a chance. See, this is getting to the ACTUAL problem with the Java environment - classloading is slow. Entity classes must be loaded and introspected before datastore operations can start. Near as I can tell, the only way around this particular problem is to stay with the low-level api - which, like assembly programming, severely limits how sophisticated an application you can reasonably build. * How do you manage URL routing? (ie, jsp files? web.xml? JAX-RS) How quaint. Did you miss that I said one type of task per app/backent? I will be generous and assume for the moment that this makes sense for your particular application. At best you are arguing that you have a wacky application. You won't find too many people building business apps that way, especially when you have elaborate transactional logic. There are sometimes good reasons to compose an app out of more fundamental services, but this should be driven by application design - not a dumb limitation of how many classes you can pack into a JAR before instance startup times get out of control. Web.XML at the individual app, but path routing is handled by the If I went the web.xml route, it would map 326 servlets. Not really fun. There's a reason people started inventing web frameworks in the late 90s... because that kind of programming sucks when your app is even mildly complicated. * How many URL endpoints do you have? Per app? Or In total? In many cases we have endpoints per domain in multi-tenant apps, and we are running 10k plus domains. I was really asking about your unicorn java app that loads in 2 seconds. But I think you misunderstand what an 'endpoint' is. It's a particular code path; for example, http://example.com/thing/123, http://example.com/thing/456, and http://other.com/thing/789 are all the same endpoint because they all execute the same code. It's relevant because it is related to how many classes get loaded at startup, which has a strong effect on the startup latency. * How many total java classes
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Sun, Jul 22, 2012 at 1:34 AM, Drake drak...@digerat.com wrote: I know what an end point is. Apparently you don't. Each of what you call and End point should be a micro app. A single purpose App that handles one type of request. All I have to say is wow. I'm really glad you're just a troll here and not actually responsible for anything I depend on. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
I can understand that GAE is more optimized for python than for java. Maybe it is a highly specialized tool for really tiny apps that use no frameworks. But I don't get one thing. How Google plans to compete for java apps by selling platform that forces you to not use frameworks? I simply refuse to go back into the 90's (in software development aspect, no problem otherwise :)). I simply do not want to drop dependency injection and other patterns I have been doing for many years already. If I must, then I think it just isn't worth it. Seriously Java is about frameworks. Saying not to use frameworks in java is same as use python instead. No thanks. On 22 июл, 11:45, Jeff Schnitzer j...@infohazard.org wrote: On Sun, Jul 22, 2012 at 1:34 AM, Drake drak...@digerat.com wrote: I know what an end point is. Apparently you don't. Each of what you call and End point should be a micro app. A single purpose App that handles one type of request. All I have to say is wow. I'm really glad you're just a troll here and not actually responsible for anything I depend on. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
All I have to say is wow. I'm really glad you're just a troll here and not actually responsible for anything I depend on. Let's see... You spend your life complaining how the platform sucks. I release tutorials on how to make it suck less. I'm the troll? Anyone who thinks the low level api is analogous to assembly probably doesn't want to depend on me because the laughter when they asked a question might be deafening. Read EVERY optimization guide on Java for GAE and that is the first thing they tell you to do. If that is too hard, then this is not the platform for you. I have vouchers for Azure, hit me up off list I'll send them along, you will be happier in .Net where you never have to do anything low level. J# is supported for 3 more years... So you should have plenty of time to learn basic principles of cloud development in that time. All animosity aside. I'm sure you are a good Java dev, but this isn't Java. The Python I write isn't Python either. I mean it is, but not how you would write it if you were on a real server. You have to get over that. You are going to suck at GAE until you do. NO FRAMEWORKS. NONE. Deal with it. Every email you beat me over the head with why you need PMF. Well Frak PMF it is holding you back. It is making you stupid. You don't need it. If you need it for one really hard thing once in a great while that is what a lazy load is for. Yes classes suck we went over that. Which is why you build micro apps that are purpose built and only use limited classes. Yes I have an F8 that can be anyone of my Micro Apps, it has 63k Entities. It lazy loads. So it doesn't break, It is used for tests and for emergency off loading of traffic. You don't have that kind of traffic so you don't need that, I get that. But you need to get over yourself and look at profiling your app, and determining where you can cut the fat, and how you can break a single monolithic app in to 4 smaller apps. I don't know what that looks like, maybe it is Customer authentication and session handling, Credit card processing and Order insertion, and Product Look up and shipping calculation. In so doing you can limit your classes, you can make better use of your instance memory, and you can likely keep from ever hitting warm ups because each app will run more efficiently than the monolith. Yeah that is a lot of work, and sure it sucks that this guy who called you stupid told you to do it, but it is the right thing to do. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I don't use frameworks in Python either. Ask my Dev team. They all but riot when they find a library they want to use and I say, figure out what it does and make your own, we don't have room for that thing. Java aint Ruby on Rails. (or Oh please don't let it be) I get that the appeal of Java is that it is older than dirt and there is a framework for everything. You can use 1 frame work if it is a small one :-). You are trading rapid elasticity for ease of development. You have to remember that when you are small Servers are cheap and humans are expensive. When you get to Google Scale that reverses. GAE is optimized for near Google scale. (not all of Google, but Small Google product) if you are 1/1000th that the dev resources for optimization will kick your ass. Just like the guys who say they wont' do it. Those are guys who don't spend $50k a month on hosting. When the amount you spend is double the cost of a dev, it is time to look at ways to cut that hosting in half using a dev, because you will break even, and give higher QoS. -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Aleksei Rovenski Sent: Sunday, July 22, 2012 2:05 AM To: Google App Engine Subject: [google-appengine] Re: Startup time exceeded...on F4?! I can understand that GAE is more optimized for python than for java. Maybe it is a highly specialized tool for really tiny apps that use no frameworks. But I don't get one thing. How Google plans to compete for java apps by selling platform that forces you to not use frameworks? I simply refuse to go back into the 90's (in software development aspect, no problem otherwise :)). I simply do not want to drop dependency injection and other patterns I have been doing for many years already. If I must, then I think it just isn't worth it. Seriously Java is about frameworks. Saying not to use frameworks in java is same as use python instead. No thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Sun, Jul 22, 2012 at 2:05 AM, Aleksei Rovenski aleksei.roven...@gmail.com wrote: I can understand that GAE is more optimized for python than for java. Maybe it is a highly specialized tool for really tiny apps that use no frameworks. But I don't get one thing. How Google plans to compete for java apps by selling platform that forces you to not use frameworks? I simply refuse to go back into the 90's (in software development aspect, no problem otherwise :)). I simply do not want to drop dependency injection and other patterns I have been doing for many years already. If I must, then I think it just isn't worth it. Seriously Java is about frameworks. Saying not to use frameworks in java is same as use python instead. No thanks. Don't listen to Brandon. By and large you can use frameworks and *should* use frameworks, because framework is just a word for code you would otherwise have to write yourself. Google doesn't want to send you back to the late 90s programming styles (even if Brandon does), and nobody sane will seriously suggest you break your app into hundreds of tiny deployments. It is a known issue with the GAE Java environment that instance starts for nontrivial applications take significant time -- enough to negatively impact the user experience if a user request gets routed to a cold start. Some frameworks make this problem worse than others, but ultimately it is a linear relationship with the amount of code in your app. Maybe Google will be able to radically improve instance startup time in the future, but probably not. There have been incremental improvements but it's a difficult problem because the JVM loads code very differently from Python or Go. The key is to prevent users from seeing cold starts in the first place. Takashi has posted one workaround: Keep min idle instances high enough that there is always enough capacity. This works but it's not ideal. It costs more and there is always the chance that a sudden burst of traffic will overload the idle instances and route a request to a cold start. You can decrease the chance of this by adding even more min idle instances but that costs even more. My hope is that the scheduler can be made smart enough to always keep requests in the pending queue until new instances are warmed up. I believe that GAE used to do this, but something recently changed so that it no longer does. If requests never see cold starts, it really doesn't matter how long your app takes to start - you can optimize capacity for time-to-service-each-request latency, instead of the % chance that a user request will get sent to an instance for 20+ seconds. So - don't panic. There is a solution that works right now, assuming you have reasonably smooth traffic patterns: Pay a bit more than you would otherwise for more idle instances. But also star this issue, which IMHO is a better long-term solution: http://code.google.com/p/googleappengine/issues/detail?id=7865 Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Ok, I get it, welcome to the world of elastic apps. But still there are few simple things google could do to ease the pain... If you are a small startup, with small traffic (just a few F1's) and a couple devs, you have money for few month to run, what you want to do is probably keep making features like hell (and consuming memory)... Don't really care for long startups (simply because not doing anything important and my client side handles timeouts gracefully) as long as google won't start killing my instances. I'm worried because I read a lot here about bad days when startup times are 3-4 times compared to normal and in that case this could happen... There are tons of projects like us and some of us can go big, so why not make small effort and make the life a little easier for conventional java apps. For example, would it be totally impossible to soften the 60s limit for the loading requests and make it double? Some of the comments above (by Jeff and others) regarding pending queue do not look too hard to implement either. It will make many project owners happiers (even if they won't go as big, but maybe they don't need to), so google wins also. On 22 июл, 12:18, Drake drak...@digerat.com wrote: I don't use frameworks in Python either. Ask my Dev team. They all but riot when they find a library they want to use and I say, figure out what it does and make your own, we don't have room for that thing. Java aint Ruby on Rails. (or Oh please don't let it be) I get that the appeal of Java is that it is older than dirt and there is a framework for everything. You can use 1 frame work if it is a small one :-). You are trading rapid elasticity for ease of development. You have to remember that when you are small Servers are cheap and humans are expensive. When you get to Google Scale that reverses. GAE is optimized for near Google scale. (not all of Google, but Small Google product) if you are 1/1000th that the dev resources for optimization will kick your ass. Just like the guys who say they wont' do it. Those are guys who don't spend $50k a month on hosting. When the amount you spend is double the cost of a dev, it is time to look at ways to cut that hosting in half using a dev, because you will break even, and give higher QoS. -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Aleksei Rovenski Sent: Sunday, July 22, 2012 2:05 AM To: Google App Engine Subject: [google-appengine] Re: Startup time exceeded...on F4?! I can understand that GAE is more optimized for python than for java. Maybe it is a highly specialized tool for really tiny apps that use no frameworks. But I don't get one thing. How Google plans to compete for java apps by selling platform that forces you to not use frameworks? I simply refuse to go back into the 90's (in software development aspect, no problem otherwise :)). I simply do not want to drop dependency injection and other patterns I have been doing for many years already. If I must, then I think it just isn't worth it. Seriously Java is about frameworks. Saying not to use frameworks in java is same as use python instead. No thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Starred and the one posted by Takashi also just in case... Regarding min idle instances, I must admit there is something strange going on in my Instance tab in Dashboard. I have settings like this: idle instances min=auto, max=1, pending queue min and max=15s. I have some working instances that I get charged for and then few more that are anyway created by google just waiting out there for hours and even days and I do not seem to pay for them. If I kill them, they are coming back anyway. This looks nice indeed, but could someone explain me what is behind that. I'm worried because it may be a drug that google will suddenly stop providing me :) Also although I'm using F1 type, google is playing soft on me here as well. For example even now one instance has already reached 143Mb and the other one 129Mb, but both are still alive and kicking... On 22 июл, 12:52, Jeff Schnitzer j...@infohazard.org wrote: On Sun, Jul 22, 2012 at 2:05 AM, Aleksei Rovenski aleksei.roven...@gmail.com wrote: I can understand that GAE is more optimized for python than for java. Maybe it is a highly specialized tool for really tiny apps that use no frameworks. But I don't get one thing. How Google plans to compete for java apps by selling platform that forces you to not use frameworks? I simply refuse to go back into the 90's (in software development aspect, no problem otherwise :)). I simply do not want to drop dependency injection and other patterns I have been doing for many years already. If I must, then I think it just isn't worth it. Seriously Java is about frameworks. Saying not to use frameworks in java is same as use python instead. No thanks. Don't listen to Brandon. By and large you can use frameworks and *should* use frameworks, because framework is just a word for code you would otherwise have to write yourself. Google doesn't want to send you back to the late 90s programming styles (even if Brandon does), and nobody sane will seriously suggest you break your app into hundreds of tiny deployments. It is a known issue with the GAE Java environment that instance starts for nontrivial applications take significant time -- enough to negatively impact the user experience if a user request gets routed to a cold start. Some frameworks make this problem worse than others, but ultimately it is a linear relationship with the amount of code in your app. Maybe Google will be able to radically improve instance startup time in the future, but probably not. There have been incremental improvements but it's a difficult problem because the JVM loads code very differently from Python or Go. The key is to prevent users from seeing cold starts in the first place. Takashi has posted one workaround: Keep min idle instances high enough that there is always enough capacity. This works but it's not ideal. It costs more and there is always the chance that a sudden burst of traffic will overload the idle instances and route a request to a cold start. You can decrease the chance of this by adding even more min idle instances but that costs even more. My hope is that the scheduler can be made smart enough to always keep requests in the pending queue until new instances are warmed up. I believe that GAE used to do this, but something recently changed so that it no longer does. If requests never see cold starts, it really doesn't matter how long your app takes to start - you can optimize capacity for time-to-service-each-request latency, instead of the % chance that a user request will get sent to an instance for 20+ seconds. So - don't panic. There is a solution that works right now, assuming you have reasonably smooth traffic patterns: Pay a bit more than you would otherwise for more idle instances. But also star this issue, which IMHO is a better long-term solution: http://code.google.com/p/googleappengine/issues/detail?id=7865 Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Sun, Jul 22, 2012 at 2:12 AM, Drake drak...@digerat.com wrote: Let's see... You spend your life complaining how the platform sucks. I release tutorials on how to make it suck less. I'm the troll? This conversation was constructive and mostly positive until you chimed in. It was a reasonable discussion of real problems that many people are having with scheduler behavior, including concrete suggestions for how to improve it. For the record, I have never, ever said that GAE sucks. Your advice, on the other hand... Anyone who thinks the low level api is analogous to assembly probably doesn't want to depend on me because the laughter when they asked a question might be deafening. Read EVERY optimization guide on Java for GAE and that is the first thing they tell you to do. You have this wrong. It's true that the JDO/JPA module gets a bad rap, but every optimization guide I've read recommends abandoning it in favor of lighter-weight layers like Objectify or Twig. It's true that these tools will give you a better startup time because they don't do classpath scanning. But it turns out that avoiding classpath scanning only gets you so far - the up-front cost of classloading and inspecting persistence classes becomes significant when you reach a critical mass of entity classes. As for writing complex business applications using strictly the low-level API... good luck. I can only explain your attitude about this as naïveté. And your absurd proposal that all 326 of my url endpoints should be separate applications... Just. Wow. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
And your absurd proposal that all 326 of my url endpoints should be separate applications... Just. Wow. If you had actually read what I posted I said that you should group by task type, and used class so that you had many smaller apps, that were optimized for like tasks and minimizing classes. Which is just good design because then you can run fewer instances likely move to f1s or f2s and better utilize in instance memory, hitting the soft memory limits less. And no it was not a constructive conversation it was a Gripe fest. If you want it to be constructive, post your code and I'll whack 30% off your start time just to get you to stop griping about how it is not possible to speed up start times because Google sucks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
On Jul 22, 5:12 am, Aleksei Rovenski aleksei.roven...@gmail.com wrote: Regarding min idle instances, I must admit there is something strange going on in my Instance tab in Dashboard. I have settings like this: idle instances min=auto, max=1, pending queue min and max=15s. I have some working instances that I get charged for and then few more that are anyway created by google just waiting out there for hours and even days and I do not seem to pay for them. If I kill them, they are coming back anyway. This looks nice indeed, but could someone explain me what is behind that. I'm worried because it may be a drug that google will suddenly stop providing me :) I get that as well. My theory is that at non-peak times, the Google scheduler opens up extra instances to be ready for spikes. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I have a similar theory. We have 9 Million instances, Let's look busy. :-) I think that they assign instances to somebody at all times, and if that somebody suddenly needs one it raises the QoS. Doesn't cost anything to idle instances that weren't doing anything. And you make the customer happy if you avoid a warm up. -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of hyperflame Sent: Sunday, July 22, 2012 10:28 AM To: Google App Engine Subject: [google-appengine] Re: Startup time exceeded...on F4?! On Jul 22, 5:12 am, Aleksei Rovenski aleksei.roven...@gmail.com wrote: Regarding min idle instances, I must admit there is something strange going on in my Instance tab in Dashboard. I have settings like this: idle instances min=auto, max=1, pending queue min and max=15s. I have some working instances that I get charged for and then few more that are anyway created by google just waiting out there for hours and even days and I do not seem to pay for them. If I kill them, they are coming back anyway. This looks nice indeed, but could someone explain me what is behind that. I'm worried because it may be a drug that google will suddenly stop providing me :) I get that as well. My theory is that at non-peak times, the Google scheduler opens up extra instances to be ready for spikes. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google- appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I just answer,1.5 seconds startup : yep, with java not even an empty hello world will manage this for us mortals. Unicorn land. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iT3DPOmS6toJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
I just answer,1.5 seconds startup : yep, with java not even an empty hello world will manage this for us mortals. Unicorn land. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/rbmZpe-pilEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Brandon, your comments are irrelevant and not constructive. The Python runtime has a completely different startup profile from the Java runtime. The long startup delays in Javaland occur prior to any service calls; no amount of caching, deferring, queueing, or serialization is going to help. Jeff On Fri, Jul 20, 2012 at 9:52 PM, Brandon Wirtz drak...@digerat.com wrote: Cache Incoming requests= The thing that I am always accused of. Use Edge Cache to make sure you don't need to serve people who are asking for the same thing. Fault Tolerant writes is about determining how race your race conditions are, and being smart about your writes. Common things I see, people want to increment something so they get count. Which is slow. Then they +1 and write. That is really slow. I use Defer for a lot of writes when I know I wont have a read for a bit of time. Google Bot just did something. Well Frak Google Bot, it's not a real user, delay that write until we have time for it. Defer. You have to look at your write types and determine the priority so you can decide how important it is to be fast, if you 100% need the write to complete, or if you are just doing something hoping people will use it later. Write to a write Cache with the Marshall/pickle, if you need the data broken out as well, do a follow up and read the Serialized data and write the individual bits when you have more time. Do reads in a similar way when ever possible use serialized data. On Friday, July 20, 2012 9:39:43 PM UTC-7, Kyle Finley wrote: Hi Brandon, The people who don't know how to build APIs so that apps are task specific. Piss me off. Build modular. Dump frame works. Defer often. Be your own scheduler by shaping internal ops. Cache incoming. Cache reads cache writes. Manage threads. Use warmups. This is not rocket science. OT, but how are you cacheing writes in a fault tolerant manner? Task queue? I've been trying to develop a strategy for that. Also, what do you mean by Cache incoming? Thanks, -Kyle -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/EVXSdF_xc1UJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Sat, Jul 21, 2012 at 1:12 AM, André Pankraz andrepank...@gmail.com wrote: I just answer,1.5 seconds startup : yep, with java not even an empty hello world will manage this for us mortals. Unicorn land. * How do you persist data? (low-level, jdo, objectify, etc) * How many entity kinds/classes do you have? * How do you manage URL routing? (ie, jsp files? web.xml? JAX-RS) * How many URL endpoints do you have? * How many total java classes in your application? What is total size? * How many jars in your WEB-INF/lib? What is total size? Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Ah, yes, and we come back to Brandon must only run one kind of app, cause I mean who runs more than one App? He can't know Java AND Python. I'm apparently not the only person living in Unicorn land, Dave Chandler also lives there http://turbomanage.wordpress.com/2010/03/26/appengine-cold-starts-considered / So does Meetu Maltiar http://thoughts.inphina.com/2010/10/09/managing-cold-start-on-google-app-eng ine/ Just because you are too lazy to optimize your code, and want to blame the platform don't take your anger out on me. Looking at the archives you have been singing the same song for 2 years. Not once have a I seen you say, Yeah my imports are now being lazy loaded or I serialized my configuration initialization variables and It saved me 1s but I need to find 5 more where should I look All I see from you and Andre is Wha-Wha, Wha-Wha, Wha-Wha-Wha. If Python is magic move to it. I run about 70% py 30% java because unlike you I profiled my apps and used which ever was better. And in the cases where Go Is Better. I'm Learning and building code in Go now too. So grow up. Quit whining read the docs on the internet and when you can post I did all those things and my app still doesn't load fast enough and when you tell me you modularized your code so that you can leverage backend for steady state and large code base latency tolerant apps, come back to me. I'll help you with a code review just to get you to stop complaining. It's ok, not every programmer and learn to think in the Google code construction methodology, or think in cloud. But there are clearly highly successful projects that have gotten around your issues. So stop saying those people live in a magical land not available to everyone else. Google doesn't look at my Email associated with the app and give me a performance bonus. (I Know I checked) -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer Sent: Saturday, July 21, 2012 1:19 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! Brandon, your comments are irrelevant and not constructive. The Python runtime has a completely different startup profile from the Java runtime. The long startup delays in Javaland occur prior to any service calls; no amount of caching, deferring, queueing, or serialization is going to help. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
What annoys me the most is that it's nowhere explicitly stated in the docs that only heavily optimized low-level applications are allowed to run on AppEngine. Google should clearly specify that most well known Java frameworks shouldn't be used on AppEngine. In my case I have tried to optimize my code as much as possible (no spring, only low-level datastore access, etc.) but I still need to load a bunch of libraries as my application is heavily connected to many external web services of all kinds (soap, rest, json, xml, etc.). I can't imagine to rewrite all these vendor specific librairies even if this include some kind of duplicate code. And for some of them, source code is not available. At the end my application takes between 10 to 15s to load. I really hope things will improve over time. So Jeff, you are not alone, I guess we are quite a few AppEngine users annoyed by these user facing loading requests. In fact, the problem exists since the beginning. We have seen some improvements over time, but things seems to go backward lately. I just hope it's temporary. Francois On Saturday, July 21, 2012 7:19:52 PM UTC+2, Brandon Wirtz wrote: Ah, yes, and we come back to Brandon must only run one kind of app, cause I mean who runs more than one App? He can't know Java AND Python. I'm apparently not the only person living in Unicorn land, Dave Chandler also lives there http://turbomanage.wordpress.com/2010/03/26/appengine-cold-starts-considered / So does Meetu Maltiar http://thoughts.inphina.com/2010/10/09/managing-cold-start-on-google-app-eng ine/http://thoughts.inphina.com/2010/10/09/managing-cold-start-on-google-app-engine/ Just because you are too lazy to optimize your code, and want to blame the platform don't take your anger out on me. Looking at the archives you have been singing the same song for 2 years. Not once have a I seen you say, Yeah my imports are now being lazy loaded or I serialized my configuration initialization variables and It saved me 1s but I need to find 5 more where should I look All I see from you and Andre is Wha-Wha, Wha-Wha, Wha-Wha-Wha. If Python is magic move to it. I run about 70% py 30% java because unlike you I profiled my apps and used which ever was better. And in the cases where Go Is Better. I'm Learning and building code in Go now too. So grow up. Quit whining read the docs on the internet and when you can post I did all those things and my app still doesn't load fast enough and when you tell me you modularized your code so that you can leverage backend for steady state and large code base latency tolerant apps, come back to me. I'll help you with a code review just to get you to stop complaining. It's ok, not every programmer and learn to think in the Google code construction methodology, or think in cloud. But there are clearly highly successful projects that have gotten around your issues. So stop saying those people live in a magical land not available to everyone else. Google doesn't look at my Email associated with the app and give me a performance bonus. (I Know I checked) -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer Sent: Saturday, July 21, 2012 1:19 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! Brandon, your comments are irrelevant and not constructive. The Python runtime has a completely different startup profile from the Java runtime. The long startup delays in Javaland occur prior to any service calls; no amount of caching, deferring, queueing, or serialization is going to help. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/L9oaqE2apt8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Brandon, your 1.5 seconds - I call this nonsense in Java world, thats all. Create empty Java Project with 1 Servlet, disable JDO/Datanucleus stuff, no Framework, nothing. 2012-07-21 12:14:33.275 / 200 3525ms 1kb Mozilla/5.0 (Windows NT 6.0; rv:14.0) Gecko/20100101 Firefox/14.0.1 May be there was some time where your mantioned link could reach 2.5 seconds - but it's not regular. Do I have problems with 5 seconds startup time? No - it's considered very fast in the Java world. I don't use any Frameworks, have learned that early. Just saying...1.5 seconds. And your regular posts that all is fine and the people are just doing it wrong are at least as annoying as mine. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/tv73lRqnY1MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
The intent of that line was that you would save 3.5s dropping factory for the GAE API But one of my main apps is 2.2s average spinup time, I am pretty sure I can get Hello world in under 1.5. Are you lazy loading? Are your lazy loads lazy loading? Have you threaded your Lazy Loads? Class 1 has dependencies ABC, Class 2 has depencies DEF, Warm up does basically nothing but defer Class1 and Class2 If Cold Start only do Called Class, If first call to instance defer other classes for warm up. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz Sent: Saturday, July 21, 2012 12:27 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! Brandon, your 1.5 seconds - I call this nonsense in Java world, thats all. Create empty Java Project with 1 Servlet, disable JDO/Datanucleus stuff, no Framework, nothing. 2012-07-21 12:14:33.275 / 200 3525ms 1kb Mozilla/5.0 (Windows NT 6.0; rv:14.0) Gecko/20100101 Firefox/14.0.1 May be there was some time where your mantioned link could reach 2.5 seconds - but it's not regular. Do I have problems with 5 seconds startup time? No - it's considered very fast in the Java world. I don't use any Frameworks, have learned that early. Just saying...1.5 seconds. And your regular posts that all is fine and the people are just doing it wrong are at least as annoying as mine. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/tv73lRqnY1MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Hi, I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a local ressource and writes it to the output stream, thats all. Hello World. Even the empty container does need some class loading - thats the Java world. And currently people have startup timing problems. 1.5 Seconds are possible if all your stuff is static and in Edge cache, nothing really starts up there. Best regards, André Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz: The intent of that line was that you would save 3.5s dropping factory for the GAE API But one of my main apps is 2.2s average spinup time, I am pretty sure I can get Hello world in under 1.5. Are you lazy loading? Are your lazy loads lazy loading? Have you threaded your Lazy Loads? Class 1 has dependencies ABC, Class 2 has depencies DEF, Warm up does basically nothing but defer Class1 and Class2 If Cold Start only do Called Class, If first call to instance defer other classes for warm up. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Read a local resource? You mean you do FileReader? Don’t do that. That’s bad. Put that in Memcache and read it from memcache. Keep it in memcache at all times. Never use FileReader. Ever. Unless it is to get an app booted for the very, very first time. It should be in the datastore, and in memcache. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz Sent: Saturday, July 21, 2012 1:00 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! Hi, I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a local ressource and writes it to the output stream, thats all. Hello World. Even the empty container does need some class loading - thats the Java world. And currently people have startup timing problems. 1.5 Seconds are possible if all your stuff is static and in Edge cache, nothing really starts up there. Best regards, André Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz: The intent of that line was that you would save 3.5s dropping factory for the GAE API But one of my main apps is 2.2s average spinup time, I am pretty sure I can get Hello world in under 1.5. Are you lazy loading? Are your lazy loads lazy loading? Have you threaded your Lazy Loads? Class 1 has dependencies ABC, Class 2 has depencies DEF, Warm up does basically nothing but defer Class1 and Class2 If Cold Start only do Called Class, If first call to instance defer other classes for warm up. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
Shit, even putting it in your static and Fetching by URL is faster than File reader. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz Sent: Saturday, July 21, 2012 1:00 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! Hi, I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a local ressource and writes it to the output stream, thats all. Hello World. Even the empty container does need some class loading - thats the Java world. And currently people have startup timing problems. 1.5 Seconds are possible if all your stuff is static and in Edge cache, nothing really starts up there. Best regards, André Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz: The intent of that line was that you would save 3.5s dropping factory for the GAE API But one of my main apps is 2.2s average spinup time, I am pretty sure I can get Hello world in under 1.5. Are you lazy loading? Are your lazy loads lazy loading? Have you threaded your Lazy Loads? Class 1 has dependencies ABC, Class 2 has depencies DEF, Warm up does basically nothing but defer Class1 and Class2 If Cold Start only do Called Class, If first call to instance defer other classes for warm up. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
I use getRessourceAsStream an Stream-copy - but you miss the point...it's a Hello World, done in 5 minutes as test for your post, stripping away all excuses. The calls after startup are answered in 80 ms. That mans that I'm still far above 3 seconds for initializing of an empty project. I could simply write Hello World into the Stream. I could use static...yes...but thats not the point. Some people have dynamic and user dependand data on each page. Am Samstag, 21. Juli 2012 22:09:11 UTC+2 schrieb Brandon Wirtz: Shit, even putting it in your static and Fetching by URL is faster than File reader. Hi, I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a local ressource and writes it to the output stream, thats all. Hello World. Even the empty container does need some class loading - thats the Java world. And currently people have startup timing problems. 1.5 Seconds are possible if all your stuff is static and in Edge cache, nothing really starts up there. Best regards, André Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz: The intent of that line was that you would save 3.5s dropping factory for the GAE API But one of my main apps is 2.2s average spinup time, I am pretty sure I can get Hello world in under 1.5. Are you lazy loading? Are your lazy loads lazy loading? Have you threaded your Lazy Loads? Class 1 has dependencies ABC, Class 2 has depencies DEF, Warm up does basically nothing but defer Class1 and Class2 If Cold Start only do Called Class, If first call to instance defer other classes for warm up. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cTL6IofY1ygJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
No the point is a major limitation of Appengine is that File IO is SLOW so you never do “open file” You want to do “dynamic” pull that from memory or datastore. NEVER EVER LOAD DATA FROM FILE OTHER THAN TO POPULATE THE DATASTORE FOR THE FIRST TIME IF YOU CARE ABOUT WARMUP Yes I have an exception where I do that with a 40 MB pickled file that it is faster than doing 40 reads from the datastore and doing a join because of the memory usage, but that is a weird case. But yes. You are doing it wrong. And if you would profile your apps you would know you are doing it wrong. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz Sent: Saturday, July 21, 2012 1:19 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! I use getRessourceAsStream an Stream-copy - but you miss the point...it's a Hello World, done in 5 minutes as test for your post, stripping away all excuses. The calls after startup are answered in 80 ms. That mans that I'm still far above 3 seconds for initializing of an empty project. I could simply write Hello World into the Stream. I could use static...yes...but thats not the point. Some people have dynamic and user dependand data on each page. Am Samstag, 21. Juli 2012 22:09:11 UTC+2 schrieb Brandon Wirtz: Shit, even putting it in your static and Fetching by URL is faster than File reader. Hi, I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a local ressource and writes it to the output stream, thats all. Hello World. Even the empty container does need some class loading - thats the Java world. And currently people have startup timing problems. 1.5 Seconds are possible if all your stuff is static and in Edge cache, nothing really starts up there. Best regards, André Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz: The intent of that line was that you would save 3.5s dropping factory for the GAE API But one of my main apps is 2.2s average spinup time, I am pretty sure I can get Hello world in under 1.5. Are you lazy loading? Are your lazy loads lazy loading? Have you threaded your Lazy Loads? Class 1 has dependencies ABC, Class 2 has depencies DEF, Warm up does basically nothing but defer Class1 and Class2 If Cold Start only do Called Class, If first call to instance defer other classes for warm up. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cTL6IofY1ygJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
I just looked over the logs of a corporate GAE application. It's a very simple heartbeat application; essentially our production applications have to send a message to it periodically. The message sent is about 500 characters; the application does some checksum-ing and sends back a JSON string about 100 characters long. The only external libraries are jars of twitter4j and the org.json parser. It's really not much more complex than a Hello World app. Looking over the logs, the fastest start time from cold-instance to first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5 seconds to cold-start) on a F1 instance. Once the instance is warm, subsequent requests take, on average, ~150ms to process and send back a reply. I could probably get that lower if I upgraded to F4. On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote: I use getRessourceAsStream an Stream-copy - but you miss the point...it's a Hello World, done in 5 minutes as test for your post, stripping away all excuses. The calls after startup are answered in 80 ms. That mans that I'm still far above 3 seconds for initializing of an empty project. I could simply write Hello World into the Stream. I could use static...yes...but thats not the point. Some people have dynamic and user dependand data on each page. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
May be it's my language barrier...don't know. Whats hard to understand in NewProject-HelloWorld-Servlet-3 seconds startup time-1,5 s isn't true and cannot be true If I take 500 ms away for the request because i'm so terribad it's still 3 s startup time, for HelloWorld. Even if it's 2.5 s in a good timeslice, it's far off 1.5 and it has to load 1 (!) servlet without depencies (besides Java-Servlet-Stuff). I'm done here - you win the internet. Am Samstag, 21. Juli 2012 22:39:51 UTC+2 schrieb Brandon Wirtz: No the point is a major limitation of Appengine is that File IO is SLOW so you never do “open file” You want to do “dynamic” pull that from memory or datastore. NEVER EVER LOAD DATA FROM FILE OTHER THAN TO POPULATE THE DATASTORE FOR THE FIRST TIME IF YOU CARE ABOUT WARMUP Yes I have an exception where I do that with a 40 MB pickled file that it is faster than doing 40 reads from the datastore and doing a join because of the memory usage, but that is a weird case. But yes. You are doing it wrong. And if you would profile your apps you would know you are doing it wrong. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2NbtQVmyFOYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
When I said 1.5 I meant PMF was adding 3.5 you don’t need. PMF takes 5s the API takes under 1s. I think 2.5s is probably the lowest you can get and do something truly useful, but it seems that I can do a LOT in 2.5s (assuming the F4 this thread was about when we started) (well actually assuming F2, F4’s seem to need 3s no matter what you do) But also you don’t have to do anything useful on warmup. So you shouldn’t serve a ColdStart very often. I have mentioned before I work to keep everything to under 4s. If my App Gets to 8s I start slicing it up I am also 100% confident that if all you do is say “I am up” your warm up will take under 1.5s for 95% of starts. And that should really do “I am up” and defer lazyloads of everything you need to be ready to do real work. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of hyperflame Sent: Saturday, July 21, 2012 1:47 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! I just looked over the logs of a corporate GAE application. It's a very simple heartbeat application; essentially our production applications have to send a message to it periodically. The message sent is about 500 characters; the application does some checksum-ing and sends back a JSON string about 100 characters long. The only external libraries are jars of twitter4j and the org.json parser. It's really not much more complex than a Hello World app. Looking over the logs, the fastest start time from cold-instance to first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5 seconds to cold-start) on a F1 instance. Once the instance is warm, subsequent requests take, on average, ~150ms to process and send back a reply. I could probably get that lower if I upgraded to F4. On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote: I use getRessourceAsStream an Stream-copy - but you miss the point...it's a Hello World, done in 5 minutes as test for your post, stripping away all excuses. The calls after startup are answered in 80 ms. That mans that I'm still far above 3 seconds for initializing of an empty project. I could simply write Hello World into the Stream. I could use static...yes...but thats not the point. Some people have dynamic and user dependand data on each page. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Yeah but where this whole discussion started was that when there's no available instance you can't defer anything, since a cold instance will apparently get the request and due to the change dynamic instances aren't warmed up like they used to be. You're bound to need to do some initialization at that point. And while you could build every single component you're app needs yourself to let it be super duper optimized for App Engine, that will just not justify the amount of work it will take to run a app on App Engine. IMHO the warm up requests serve their purpose; they warm up the app so that a slightly longer start up time of 10 secs or so isn't that bad. The issue is just that there are now situations where user facing requests end up warming up a instance which sounds like a bad thing, especially if the app is receiving spiky loads. On Saturday, July 21, 2012 11:00:53 PM UTC+2, Brandon Wirtz wrote: When I said 1.5 I meant PMF was adding 3.5 you don’t need. PMF takes 5s the API takes under 1s. I think 2.5s is probably the lowest you can get and do something truly useful, but it seems that I can do a LOT in 2.5s (assuming the F4 this thread was about when we started) (well actually assuming F2, F4’s seem to need 3s no matter what you do) But also you don’t have to do anything useful on warmup. So you shouldn’t serve a ColdStart very often. I have mentioned before I work to keep everything to under 4s. If my App Gets to 8s I start slicing it up I am also 100% confident that if all you do is say “I am up” your warm up will take under 1.5s for 95% of starts. And that should really do “I am up” and defer lazyloads of everything you need to be ready to do real work. *From:* google-appengine@googlegroups.com [mailto: google-appengine@googlegroups.com] *On Behalf Of *hyperflame *Sent:* Saturday, July 21, 2012 1:47 PM *To:* google-appengine@googlegroups.com *Subject:* Re: [google-appengine] Re: Startup time exceeded...on F4?! I just looked over the logs of a corporate GAE application. It's a very simple heartbeat application; essentially our production applications have to send a message to it periodically. The message sent is about 500 characters; the application does some checksum-ing and sends back a JSON string about 100 characters long. The only external libraries are jars of twitter4j and the org.json parser. It's really not much more complex than a Hello World app. Looking over the logs, the fastest start time from cold-instance to first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5 seconds to cold-start) on a F1 instance. Once the instance is warm, subsequent requests take, on average, ~150ms to process and send back a reply. I could probably get that lower if I upgraded to F4. On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote: I use getRessourceAsStream an Stream-copy - but you miss the point...it's a Hello World, done in 5 minutes as test for your post, stripping away all excuses. The calls after startup are answered in 80 ms. That mans that I'm still far above 3 seconds for initializing of an empty project. I could simply write Hello World into the Stream. I could use static...yes...but thats not the point. Some people have dynamic and user dependand data on each page. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/yM2ywgzd4PIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Are you using a free or paid app to do your testing? My understanding is that paid apps are treated preferentially by the scheduler. On Jul 21, 3:49 pm, André Pankraz andrepank...@gmail.com wrote: May be it's my language barrier...don't know. Whats hard to understand in NewProject-HelloWorld-Servlet-3 seconds startup time-1,5 s isn't true and cannot be true If I take 500 ms away for the request because i'm so terribad it's still 3 s startup time, for HelloWorld. Even if it's 2.5 s in a good timeslice, it's far off 1.5 and it has to load 1 (!) servlet without depencies -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Hi guys, I actually didn't want to reply to this thread originally even I was the one who opened the small thread 3 months ago regarding the very slow startups on gae with spring + velocity + objectify but after reading the email telling me I should write modular apps and don't use frameworks I have to write up something. I'm 10+ years java developer and I'm big fan of Spring as it saves a hundred hours of time - I originally developed the www.librarist.com 2 years ago as a pet project on app engine as a just a prototype using simple JSPs + Servlets and it works working fine (startups 3-5 secs). But when I wanted to add another features like cart, OAuth login, use templating, use proper caching - and I decided to go with Spring as this the reason why we have frameworks. Plugging in Velocity is just matter of 1 line of code in XML (plus some tweaks if you need them), the cache is again couple of lines in XML and you automatically get in memory + memcache with ehcache, authentication is 2 lines of code. And I can continue. I use spring in work for other projects which runs on full java stack (but thats just Tomcat/Jetty) and when I run the new librarist equivalent code on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be wrong here a little bit), the startup is 2-3 secs and I can serve all my traffic with it with load at max 0.5. When I run the similar code on GAE, I get 45+ secs startup (when I'm lucky, sometime it just go over 1 nib and got killed) and then GAE keep spinning another instances like crazy but very often use them for three request and then dying to get started again. Worth to mention - I do not use the full Spring stac, basically its just Spring core + Spring MVC, then objectify + Velocity - and I really tried to do all the possible optimizations (packing all the stuff into one big jar, used annotations + java confing, used XML, disable scanning of annotations) but these changes just save me like 4-5 secs from 45 secs. When I tried just Spring code + Spring MVC without any other library (and here we talk about ~2MB of jars), use one JSP, my best startup time is 15 secs. Okay thats cool, I can understand the GAE is different and basically it could be used just for small projects where you write everything by yourself and don't use any normal stuff like Dependency Injection / Proxying and similar. But my biggest question for GAE guys would be this - why the hell the startup takes so long simply I don't get what is actually the issue here: - is it loading of the jars from the network (virtual drive) when the gae loads each class as separate file - is it the scanning of the classes (annotations etc) - is it java reflections - is it creating of proxies (which Spring does for almost everything) - anything else I can't see I really REALLY liked the idea of the app engine since I hate to mess up with server configuration and it's great to focus on SW alone. But honestly speaking, I gave up on GAE for serious projects - it's good platform for doing some quick prototype to show to client (easy deploy, no server maintenance, you can have a simple app talking with a lot of apis up and running in half a day) - but then move the stuff the AWS. I made this decision 3 months ago and from the app engine google groups I can see it was right thing to do as there is more and more emails coming in regarding broken live apps, issues with deploys, issues with startups and stuff like that (the SSL cert price is just a joke for example). I really appreciate the work done on the GAE by Google but if I could give an advice - please stop adding new features with every release, invest one month of time of your team and do a proper maintenance release which will fix all that shit around Java apps (this really can't be so hard) or you will lose the war and all java devs will run away to AWS/Heroku/Cloud Foundry/Own servers. Cheers. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ozWj4QUDBDkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
If it makes you feel better I have this fight once a month. AWS that is easier to build for or GAE which doesn't require any sys admin and is infinitely elastic. Someone will put something on AWS show me how awesome it is, and how cheap. I will throw the load simulator at it, and watch it burn. They say but you didn't give it more instances. I will say fine every 4th 8 minutes when CNN runs a commercial for the site you need to handle 5000 uesrs, and the rest of the time you need to handle 8. Which is cheaper? They always lose. How do we manage keeping the versioning the same on all instances in AWS? How long is the down time on deploy of new version? (GAE you can deploy, warm up enough instances and change the Default version and not even lose sessions) Every service has plusses and minuses if they didn't then someone would be a monopoly. Edge Cache is amazing for most apps it will negate the AWS savings. Rapid Elasticity is amazing it will negate even RackSpace savings. Not taking off the shelf code. That will cost you a developer. Sorry. I would also say most the time, you should be rewriting that code anyway to be exactly what you need, but I get that sometimes what you want is to use Facebook Authorization, and there is no reason to strip stuff out. Except that you have to remember everyone who is not you writes sucky code. Generic code. Meant to run everywhere. Except GAE is not everywhere, it is in unicorn land where you traded having nice friendly donkey's who will put up with what ever shit you feed them and still pull your wagon, for a flying unicorn that shoots rainbows out of its ass but requires that you don't just give it a sugar cube every now and then, but that it is raw organic sugar fed to it by beautiful naked virgins. In exchange you can go farther, faster, and you don't have to clean up poo off the floor. These are the trades you make for living in Unicorn land. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Tomas Sent: Saturday, July 21, 2012 2:50 PM To: google-appengine@googlegroups.com Subject: [google-appengine] Re: Startup time exceeded...on F4?! Hi guys, I actually didn't want to reply to this thread originally even I was the one who opened the small thread 3 months ago regarding the very slow startups on gae with spring + velocity + objectify but after reading the email telling me I should write modular apps and don't use frameworks I have to write up something. I'm 10+ years java developer and I'm big fan of Spring as it saves a hundred hours of time - I originally developed the www.librarist.com 2 years ago as a pet project on app engine as a just a prototype using simple JSPs + Servlets and it works working fine (startups 3-5 secs). But when I wanted to add another features like cart, OAuth login, use templating, use proper caching - and I decided to go with Spring as this the reason why we have frameworks. Plugging in Velocity is just matter of 1 line of code in XML (plus some tweaks if you need them), the cache is again couple of lines in XML and you automatically get in memory + memcache with ehcache, authentication is 2 lines of code. And I can continue. I use spring in work for other projects which runs on full java stack (but thats just Tomcat/Jetty) and when I run the new librarist equivalent code on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be wrong here a little bit), the startup is 2-3 secs and I can serve all my traffic with it with load at max 0.5. When I run the similar code on GAE, I get 45+ secs startup (when I'm lucky, sometime it just go over 1 nib and got killed) and then GAE keep spinning another instances like crazy but very often use them for three request and then dying to get started again. Worth to mention - I do not use the full Spring stac, basically its just Spring core + Spring MVC, then objectify + Velocity - and I really tried to do all the possible optimizations (packing all the stuff into one big jar, used annotations + java confing, used XML, disable scanning of annotations) but these changes just save me like 4-5 secs from 45 secs. When I tried just Spring code + Spring MVC without any other library (and here we talk about ~2MB of jars), use one JSP, my best startup time is 15 secs. Okay thats cool, I can understand the GAE is different and basically it could be used just for small projects where you write everything by yourself and don't use any normal stuff like Dependency Injection / Proxying and similar. But my biggest question for GAE guys would be this - why the hell the startup takes so long simply I don't get what is actually the issue here: - is it loading of the jars from the network (virtual drive) when the gae loads each class as separate file - is it the scanning of the classes (annotations etc) - is it java reflections - is it creating of proxies (which Spring does for almost everything
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Here is a counter example. We are a small app with peak traffic of 1 request per second. We use java. No frameworks. We use jdo, guice. Startup time 20 seconds. Average request latency under 500 ms. We have been on appengine close to 2 years. Right now we are able to serve our users with one instance without exposing them to loading requests ( almost zero ) We use min = auto , max = 2 and pending latency = 7.5 s. we arrived at this configuration after some experimentation and it gives the best performance and cost so far... At least we are fine now, unless google decides to change something. We do have our share of problems with gae ( memcache errors, sudden increase in latency for brief periods..) But that's negligible relative to the advantages of the platform and cost. Learnings Avoid warmups/loading requests at all costs. Our goal was never let gae start extra instances, as that can lead to instability. We cannot reduce startup time, and gae cannot get rid of all the issues associated with startup. Min = 1 max = 1 gave us the worst performance as it creates a constraint that spins lot of instances esp if you have a bursts of traffic. Min = auto allows us to use pending latency config and it gives better control on the scheduler behaviour. Disclaimer Probably this worked for us because we have a steady request pattern during the day, and we don't normally see zero traffic for more than a few mins. if you are an app with different behaviour ( long idle times ) then min = auto may expose the user to loading requests..in this case min = 1 max = 2 or more should work better.. Again our $.02 cents...your mileage may vary Hope this helps. Prashant Sent from my iPad On 22-Jul-2012, at 3:20 AM, Tomas tomas.ada...@gmail.com wrote: Hi guys, I actually didn't want to reply to this thread originally even I was the one who opened the small thread 3 months ago regarding the very slow startups on gae with spring + velocity + objectify but after reading the email telling me I should write modular apps and don't use frameworks I have to write up something. I'm 10+ years java developer and I'm big fan of Spring as it saves a hundred hours of time - I originally developed the www.librarist.com 2 years ago as a pet project on app engine as a just a prototype using simple JSPs + Servlets and it works working fine (startups 3-5 secs). But when I wanted to add another features like cart, OAuth login, use templating, use proper caching - and I decided to go with Spring as this the reason why we have frameworks. Plugging in Velocity is just matter of 1 line of code in XML (plus some tweaks if you need them), the cache is again couple of lines in XML and you automatically get in memory + memcache with ehcache, authentication is 2 lines of code. And I can continue. I use spring in work for other projects which runs on full java stack (but thats just Tomcat/Jetty) and when I run the new librarist equivalent code on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be wrong here a little bit), the startup is 2-3 secs and I can serve all my traffic with it with load at max 0.5. When I run the similar code on GAE, I get 45+ secs startup (when I'm lucky, sometime it just go over 1 nib and got killed) and then GAE keep spinning another instances like crazy but very often use them for three request and then dying to get started again. Worth to mention - I do not use the full Spring stac, basically its just Spring core + Spring MVC, then objectify + Velocity - and I really tried to do all the possible optimizations (packing all the stuff into one big jar, used annotations + java confing, used XML, disable scanning of annotations) but these changes just save me like 4-5 secs from 45 secs. When I tried just Spring code + Spring MVC without any other library (and here we talk about ~2MB of jars), use one JSP, my best startup time is 15 secs. Okay thats cool, I can understand the GAE is different and basically it could be used just for small projects where you write everything by yourself and don't use any normal stuff like Dependency Injection / Proxying and similar. But my biggest question for GAE guys would be this - why the hell the startup takes so long simply I don't get what is actually the issue here: - is it loading of the jars from the network (virtual drive) when the gae loads each class as separate file - is it the scanning of the classes (annotations etc) - is it java reflections - is it creating of proxies (which Spring does for almost everything) - anything else I can't see I really REALLY liked the idea of the app engine since I hate to mess up with server configuration and it's great to focus on SW alone. But honestly speaking, I gave up on GAE for serious projects - it's good platform for doing some quick prototype to show to client (easy deploy, no server
[google-appengine] Re: Startup time exceeded...on F4?!
On Jul 21, 5:17 pm, Drake drak...@digerat.com wrote: unicorn land where you traded having nice friendly donkey's who will put up with what ever shit you feed them and still pull your wagon, for a flying unicorn that shoots rainbows out of its ass but requires that you don't just give it a sugar cube every now and then, but that it is raw organic sugar fed to it by beautiful naked virgins. In exchange you can go farther, faster, and you don't have to clean up poo off the floor. These are the trades you make for living in Unicorn land. I like this visual. If someone puts it on a coffee cup or t-shirt as a GAE promotion, I'd buy it. Seriously, someone do this. On Jul 21, 9:21 pm, Prashant Hegde prashant.he...@gmail.com wrote: Here is a counter example. We are a small app with peak traffic of 1 request per second. We use java. No frameworks. We use jdo, guice. Startup time 20 seconds. Average request latency under 500 ms. We have been on appengine close to 2 years. Right now we are able to serve our users with one instance without exposing them to loading requests ( almost zero ) This isn't a counterexample, for the reason that you mentioned in the first sentence: you can serve everything off one instance. The original poster needs multiple instances, and to be able to scale as load changes. If you're not loading new instances, then startup time is pretty much irrelevant; you can start an instance once, then your users will never see a cold instance. This entire discussion is about cold instances. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
On Jul 21, 4:50 pm, Tomas tomas.ada...@gmail.com wrote: I use spring in work for other projects which runs on full java stack (but thats just Tomcat/Jetty) and when I run the new librarist equivalent code on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be wrong here a little bit), the startup is 2-3 secs and I can serve all my traffic with it with load at max 0.5. When I run the similar code on GAE, I get 45+ secs startup... Okay thats cool, I can understand the GAE is different and basically it could be used just for small projects where you write everything by yourself and don't use any normal stuff like Dependency Injection / Proxying and similar. But my biggest question for GAE guys would be this - why the hell the startup takes so long simply I don't get what is actually the issue here: - is it loading of the jars from the network (virtual drive) when the gae loads each class as separate file - is it the scanning of the classes (annotations etc) - is it java reflections - is it creating of proxies One (small) reason why GAE is slower is just that instances are smaller than comparable AWS instances. An AWS Micro is 613mb RAM and can temporarily use 2 ECU of CPU (looking at Amazon docs, they claim that 1 ECU is equal to a 1 ghz 2007 opteron). A GAE F1 is 128 mb of RAM and 600mhz. I bet that the major reason is just network I/O, for the GAE servers to find an available server, transfer a copy of the application +libraries to that server, and start up the servlet runner. It would explain why even simple apps have long startup times: it's not that the frameworks and libraries are taking a long time, it's just the overhead of moving all those libraries into place initially. This also suggests a simple solution to the problem: just integrate a copy of commonly used libraries (spring, guice, twitter4j, etc) into the servlet runner. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Brandon, you talk a lot of shit for a guy who only discovered the task queue 6 months ago. Everyone who thinks they have the secret to starting up a Java app in 5s, answer these questions and prove that you're running more than a toy: * How do you persist data? (low-level, jdo, objectify, etc) * How many entity kinds/classes do you have? * How do you manage URL routing? (ie, jsp files? web.xml? JAX-RS) * How many URL endpoints do you have? * How many total java classes in your application? What is total size? * How many jars in your WEB-INF/lib? What is total size? Having done a fair bit of experimentation around this issue, I know that you're talking out of your asses. Clever suggestions like rewrite app in Python aren't nearly as clever as you think they are. Suggestions that you can optimize startup time by some combination of caching, task queues, or threading merely indicate total lack of understanding of the problem in the first place. To the people paying attention to this thread who are actually trying to build real world apps on Appengine: Brandon does not speak for the GAE team. Neither do I, but I occasionally talk to people that do. I know that Google really does want Java apps to run well on GAE. They know about this issue and care about it - maybe even enough to do something about it. It *is* constructive to whine politely about startup time because there are many issues competing for the team's time, and our feedback helps them prioritize their efforts. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Sat, Jul 21, 2012 at 9:24 PM, hyperflame hyperfl...@gmail.com wrote: I bet that the major reason is just network I/O, for the GAE servers to find an available server, transfer a copy of the application +libraries to that server, and start up the servlet runner. It would explain why even simple apps have long startup times: it's not that the frameworks and libraries are taking a long time, it's just the overhead of moving all those libraries into place initially. The problem specifically seems to be that classloading is slow. A significant part of that is explained by slow network I/O, but not all; for example, getting AOP proxies built and loaded seems to take significantly longer than one would expect by profiling locally. I speculate that the extra security measures of GAE's sandbox are also taking a toll. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
This isn't a counterexample, for the reason that you mentioned in the first sentence: you can serve everything off one instance. The original poster needs multiple instances, and to be able to scale as load changes. If you're not loading new instances, then startup time is pretty much irrelevant; you can start an instance once, then your users will never see a cold instance. This entire discussion is about cold instances. To clarify, we need more than one instance during peak hours, but our overall billing is limited to 1 instance because night time traffic is literally zero. We have seen 2 instances without user facing loading request during peak hours. Our analysis was that when pending latency comes into picture, the behavior is more predictable. While the pending queue is getting filled up, new instance is started without user facing request as we have specified 7 second as the pending latency. By the time previous request is served, pending request goes to the hot instance. A lot of depends on your traffic pattern and time taken to handle the request though. Atleast for us the overall goal was - how do we achieve 500 ms latency for our users for the traffic, with auto scaling, without repeated loading requests, without escalation in costs. We seemed to have achieved that, hence the post and as a counter example to our friend who decided to go off appengine to AWS. Prashant -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
I did some thinking about this problem, and it occurred to me that we might be going at this the completely wrong way. Could we reverse- bloom instance startups? Here's the basic idea. If apps need to load in/process a bunch of stuff at startup, why don't we give them the processing power to do that, but only at the time of startup? Suppose we start up an instance. For the first 30 seconds of that instance, we give it F16 ranking (that is, 4 times the power of a current F4 instance). For the next 30 seconds, the instance gets F12 ranking (3 times F4 power). The next 30 seconds, the instance gets F8 ranking (2 times F4 power). After that, the instance gets lowered to F4 and remains there for the rest of its lifespan. For the first 90 seconds of the instance's lifespan, it gets turbocharged, and should be able to handle all startup processes and a few initial requests. The scheduler doesn't need to change; it can continue feeding requests into cold instances, because even cold instances will tear through their startup and handle the requests. Even if this turboboosted time was billed at standard rate, it wouldn't cost that much; F4 costs 32 cents per instance hour, so F16 would (theoretically) cost 4 times as much, so an F16 instance hour would cost $1.28 per instance hour. We only need 90 seconds at that rate, so 3.2 cents. I think paying 3 cents extra per instance startup is very reasonable, especially if it prevents user-facing cold instance startups. It's entirely doable on a technical level too. Amazon AWS has a similar system where Micro servers can temporarily boost to 2 compute units, then get lowered to their allotted 1 ECU. Linux has the ability built-in: see nice/renice http://en.wikipedia.org/wiki/Nice_(Unix) I'd be interested in hearing if this system is possible within the GAE hosting system. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
I did, and used Objectify (Jeff is kinda familiar with that I think :)). A plain Hello World with only registering the entities will get you around the 4-5 sec mark, which is hardly a real application. I have a highly optimized app as a backend for my Android app, and that even takes around the 6-8 seconds mark to start up and it varies too. Add in some third party frameworks, like Jersey for JAXRS and Guice to keep your code sane and you're at the 12 secs. So kool aid or not, Java start up overhead will always be more than it is for python and I think the 5 seconds mentioned is far from typical if you are running a non-trivial app. On Friday, July 20, 2012 6:20:56 AM UTC+2, Brandon Wirtz wrote: Dump PMF use the low level API https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi ne/api/datastore/package-summaryhttps://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/datastore/package-summary Either drink the Kool Aid, or quit complaining :-) 5s will be 1.5 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4icNAFZtGKwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
That's just not true - I have an app which uses no third-party libraries at all, uses no persistence and in fact it uses no GAE services. It simply has one servlet which processes request headers and returns a response. My average start-up time for this app is 3 seconds, when it's running well. The simple fact is that as soon as you use any third-party libraries, or do anything vaguely useful, you're out of that 5 second magic window. On Friday, 20 July 2012 05:20:56 UTC+1, Brandon Wirtz wrote: Dump PMF use the low level API https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi ne/api/datastore/package-summaryhttps://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/datastore/package-summary Either drink the Kool Aid, or quit complaining :-) 5s will be 1.5 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/W1sBfl9Pu2IJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Sorry, but, please forget about 5 secs 'magic' window. There is no any hard/soft deadline/threshold like that in the current App Engine system. It was just a one example of well behaved apps. Let me rephrase what I meant to say. With app engine, it is always a good practice to keep the loading request faster. The current scheduler works much better with fast loading apps than with slow loading apps. So I always suggest developers fastening the loading request. On the other hand, we understand that Java apps tend to be slow at a level where it is not acceptable for you if there are many user-facing loading requests. For a time being, please use min idle instances settings to reduce the loading requests with slow loading apps. -- Takashi On Jul 20, 2012 5:16 PM, Simon Knott knott.si...@gmail.com wrote: That's just not true - I have an app which uses no third-party libraries at all, uses no persistence and in fact it uses no GAE services. It simply has one servlet which processes request headers and returns a response. My average start-up time for this app is 3 seconds, when it's running well. The simple fact is that as soon as you use any third-party libraries, or do anything vaguely useful, you're out of that 5 second magic window. On Friday, 20 July 2012 05:20:56 UTC+1, Brandon Wirtz wrote: Dump PMF use the low level API https://developers.google.com/**appengine/docs/java/javadoc/**com/google/appengi ne/api/datastore/package-**summaryhttps://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/datastore/package-summary Either drink the Kool Aid, or quit complaining :-) 5s will be 1.5 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/W1sBfl9Pu2IJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
If you follow the group longer you should know - Brandom lives in GAE unicorn land and all you ever need are proper Edge cache settings. ;) Am Freitag, 20. Juli 2012 10:16:28 UTC+2 schrieb Simon Knott: That's just not true - I have an app which uses no third-party libraries at all, uses no persistence and in fact it uses no GAE services. It simply has one servlet which processes request headers and returns a response. My average start-up time for this app is 3 seconds, when it's running well. The simple fact is that as soon as you use any third-party libraries, or do anything vaguely useful, you're out of that 5 second magic window. On Friday, 20 July 2012 05:20:56 UTC+1, Brandon Wirtz wrote: Dump PMF use the low level API https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi ne/api/datastore/package-summaryhttps://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/datastore/package-summary Either drink the Kool Aid, or quit complaining :-) 5s will be 1.5 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lYiT-qQBw6UJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Perfect, thanks; that is all I am asking for! Additionally, we started an internal discussion about reviving warmup requests for dynamic instances. If you want this feature, please star the following issue: http://code.google.com/p/googleappengine/issues/detail?id=7865 -- Takashi -- Takashi Matsuo -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jNAp6gCZ_4QJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Discussing theoretical startup times is great, but I'd like to see some real-world startups. Does anyone with high startup times ( say, 30+ seconds) want to share the results of a code profiler/appstats? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
On Fri, Jul 20, 2012 at 10:34 AM, hyperflame hyperfl...@gmail.com wrote: Discussing theoretical startup times is great, but I'd like to see some real-world startups. Does anyone with high startup times ( say, 30+ seconds) want to share the results of a code profiler/appstats? What code profiler works on GAE server-side? It's not very helpful to profile in the dev environment since the problem does not manifest there. Appstats is not helpful; my startup is long even with zero service layer calls. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
If you follow the group longer you should know - Brandom lives in GAE unicorn land and all you ever need are proper Edge cache settings. ;) No, I just don't write code using unnecessary frameworks and I do a ton of testing and architecture planning. Most everyoneelse never uses defer or threads or task queues. They don't manage steady state tasks, or profile their imports. It is easy to blame the platform, but if you write crap code it will run like crap. I don't think GAE is perfect. But most the people who complain about its failings don't have the coding skills to be an intern at my shop. I blame Google for not documenting everything well. I blame google for not telling you don't use frameworks they suck. But the number of people who load spring and the gripe that it is slow and are only using 3 things out of it make me want to punch them in the head. The people who don't know how to build APIs so that apps are task specific. Piss me off. Build modular. Dump frame works. Defer often. Be your own scheduler by shaping internal ops. Cache incoming. Cache reads cache writes. Manage threads. Use warmups. This is not rocket science. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Startup time exceeded...on F4?!
All your pages should have the ability to have an optional ?debug=true Then if you have set that in the url the first line in every one of your functions, and before your imports, should be Timer start and the last thing should be Log Timer -Original Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer Sent: Friday, July 20, 2012 11:00 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! On Fri, Jul 20, 2012 at 10:34 AM, hyperflame hyperfl...@gmail.com wrote: Discussing theoretical startup times is great, but I'd like to see some real-world startups. Does anyone with high startup times ( say, 30+ seconds) want to share the results of a code profiler/appstats? What code profiler works on GAE server-side? It's not very helpful to profile in the dev environment since the problem does not manifest there. Appstats is not helpful; my startup is long even with zero service layer calls. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google- appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Hi Brandon, The people who don't know how to build APIs so that apps are task specific. Piss me off. Build modular. Dump frame works. Defer often. Be your own scheduler by shaping internal ops. Cache incoming. Cache reads cache writes. Manage threads. Use warmups. This is not rocket science. OT, but how are you cacheing writes in a fault tolerant manner? Task queue? I've been trying to develop a strategy for that. Also, what do you mean by Cache incoming? Thanks, -Kyle -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vB55pDznx94J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Cache Incoming requests= The thing that I am always accused of. Use Edge Cache to make sure you don't need to serve people who are asking for the same thing. Fault Tolerant writes is about determining how race your race conditions are, and being smart about your writes. Common things I see, people want to increment something so they get count. Which is slow. Then they +1 and write. That is really slow. I use Defer for a lot of writes when I know I wont have a read for a bit of time. Google Bot just did something. Well Frak Google Bot, it's not a real user, delay that write until we have time for it. Defer. You have to look at your write types and determine the priority so you can decide how important it is to be fast, if you 100% need the write to complete, or if you are just doing something hoping people will use it later. Write to a write Cache with the Marshall/pickle, if you need the data broken out as well, do a follow up and read the Serialized data and write the individual bits when you have more time. Do reads in a similar way when ever possible use serialized data. On Friday, July 20, 2012 9:39:43 PM UTC-7, Kyle Finley wrote: Hi Brandon, The people who don't know how to build APIs so that apps are task specific. Piss me off. Build modular. Dump frame works. Defer often. Be your own scheduler by shaping internal ops. Cache incoming. Cache reads cache writes. Manage threads. Use warmups. This is not rocket science. OT, but how are you cacheing writes in a fault tolerant manner? Task queue? I've been trying to develop a strategy for that. Also, what do you mean by Cache incoming? Thanks, -Kyle -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/EVXSdF_xc1UJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Startup time exceeded...on F4?!
Oh, OK. Thank you Brandon. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/rvs4uS2r6LkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
