Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Michael Hermus]

Great news, everyone! I have replaced all String concatenation with 
StringBuilder, and extensively reordered all my import statements. My startup 
time has been reduced 50% from twice it's original value.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/kL6FzpNuUb4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Wed, Jul 25, 2012 at 8:42 PM, David Hardwick
 wrote:
>
> Amazing, Andrei.  If you can get those startup times going with GWT and the
> complexities you mentioned in your app, then clearly stripping out DI and
> using low-level API (and likely jar'ing everything) is a mega contributor to
> reducing startup time.  I would hate to have to spend the team's time
> stripping that stuff out, i would just pay for higher idle instances until
> request #7865 gets implemented.

I just want to put this in proper perspective - we should expect
Andrei's app to startup quickly.  GWT-RPC is actually great for
startup time because there is only a single backend servlet to start
up; there's no need to build a sitemap by loading classes and looking
for annotations (ie, JAX-RS).  Similarly, the low-level API has zero
startup overhead.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Aleksei Rovenski]

I didn't read in full everything here, but using StringBuilder is good old 
optimization, strange many don't know it here...
http://www.venishjoe.net/2009/11/java-string-concatenation-and.html
Having said that it seems that both of you are kind of right, recent JDK's 
compile '+' into StringBuilder, but not in all cases, so you have to be 
carefull. 
For example, this would be compiled into builder: 
String ab = a + b;
but this will be compiled into 2 StringBuilder... 
String ab = "ab";
ab += a;
ab += b;

вторник, 24 июля 2012 г., 20:40:10 UTC+3 пользователь Brandon Wirtz написал:
>
> And you get a new string builder each loop. If you use string builder and 
> recycle it you don't have to re-create the object. Which doesn't create so 
> much garbage. 
> You have the answer and you don't understand it. 
>
>
>
> > Try compiling these two classes.  They produce *identical* class files: 
> > 
> >  File m1/Stringy.java  
> > public class Stringy { 
> > public static void main(String[] args) { 
> > String foo = "def"; 
> > System.out.println("abc" + foo); 
> > } 
> > } 
> >  File m2/Stringy.java  
> > public class Stringy { 
> > public static void main(String[] args) { 
> > String foo = "def"; 
> > System.out.println(new 
> > StringBuilder().append("abc").append(foo).toString()); 
> > } 
> > } 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/CDZcdLF0bIgJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Tue, Jul 24, 2012 at 10:40 AM, Drake  wrote:
> And you get a new string builder each loop. If you use string builder and
> recycle it you don't have to re-create the object. Which doesn't create so
> much garbage.
> You have the answer and you don't understand it.

Just give it up and star the issue, mkay?

http://code.google.com/p/googleappengine/issues/detail?id=7865

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

And you get a new string builder each loop. If you use string builder and
recycle it you don't have to re-create the object. Which doesn't create so
much garbage.
You have the answer and you don't understand it.



> Try compiling these two classes.  They produce *identical* class files:
>
>  File m1/Stringy.java 
> public class Stringy {
> public static void main(String[] args) {
> String foo = "def";
> System.out.println("abc" + foo);
> }
> }
>  File m2/Stringy.java 
> public class Stringy {
> public static void main(String[] args) {
> String foo = "def";
> System.out.println(new
> StringBuilder().append("abc").append(foo).toString());
> }
> }


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Tue, Jul 24, 2012 at 7:48 AM, Drake  wrote:
>
> + doesn’t matter if you do it once, but at one point this code is doing up
> to 15k +’s in a loop, and + doesn’t do efficient recycling of
> temporary/intermediary objects.

Did you even read the bytecode output I posted?  Seriously, I've never
met someone who is given all the right answers to the test but just
keeps insisting on the wrong answers anyways.

Try compiling these two classes.  They produce *identical* class files:

 File m1/Stringy.java 
public class Stringy {
public static void main(String[] args) {
String foo = "def";
System.out.println("abc" + foo);
}
}
 File m2/Stringy.java 
public class Stringy {
public static void main(String[] args) {
String foo = "def";
System.out.println(new
StringBuilder().append("abc").append(foo).toString());
}
}


If you don't believe me that the JVM lazy-loads classes, you can
either run some experiments with -XX:+TraceClassLoading or just spend
a few minutes with Google.  I've wasted enough time with this
discussion.


The honorable thing to do is apologize and star my issue:

http://code.google.com/p/googleappengine/issues/detail?id=7865

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

But back to the roots...long loops have nothing to do with class loading in 
java ;) imports and loops - we would be hard pressed to find a _relevant_ 
example. really... I don't believe you the Smart-Java-Guy story.
200 ms through import-reordering? nope...no imports in bytecode...just 
straigt bytecode with ful types: GET com.bla.C fieldname, INVOKE com.bla.D 
methodname methoddescriptor
You know the GAE very well and for sure python and your cache stuff...but 
very often you generalize this for all and that gets really annoying.

You can "lazy load" classes if you don't use their functionallity, thats 
right - this classes are not fully loaded at the beginning. But many 
annotation-scanning frameworks (DI / O/R mapper etc.) in Java don't work 
this way.


Am Dienstag, 24. Juli 2012 16:48:36 UTC+2 schrieb Brandon Wirtz:
>
> On 5 hours now of sleep so I can form sentences:
>
>  
>
> >Your micro-optimizing suggestions for Java ...I think all Java guys are 
> like ROFL now.
>
> >Star visa Full Imports, String Concatenations in this example visa 
> StringBuilder
>
> Y’all’d be right if this was 100 lines of code in Java 101.
>
>  
>
> What you aren’t looking at is that what is slow in this code set is object 
> creation and object recycling.
>
>  
>
> You are right that
>
>  
>
> Import Io.*
>
>  
>
> And 
>
>  
>
> Import Io.Blah
>
> Import Io.Blah2
>
>  
>
> Results in the same code,
>
>  
>
> What doesn’t result in the same code is doing import Io.Blah in classa and 
> import io.blah2 in classb and io.blah3 in classC because it changes the 
> order and timing of the Object and imported Class Creation.
>
>  
>
> + doesn’t matter if you do it once, but at one point this code is doing up 
> to 15k +’s in a loop, and + doesn’t do efficient recycling of 
> temporary/intermediary objects.
>
>  
>
> So laugh all you want, but there are serious gains in profiling the code 
> and seeing where object and memory manipulation are sub optimal, especially 
> when you have long loops.
>
>  
>
> The problem I see with you guys is that you don’t test, “REAL” application 
> you test micro bits of code and don’t look at the impact or history of how 
> the code got to where it is.  You say “it doesn’t matter in 4 lines of 
> codes” but you don’t have “THOSE” 4 lines of codes in your app, you have 4 
> lines intermingled in with 80 other lines that change what is going on.
>
>  
>
> There are times when just changing the order of your Imports can add or 
> remove 200ms to your app.
>
>  
>
> As to architecture changes… No one liked those. Don’t calculate hashes if 
> you are already in a session, use memcache and instance cache. Thread 
> really long loops.
>
>  
>
>  
>
>  
>

Am Dienstag, 24. Juli 2012 16:48:36 UTC+2 schrieb Brandon Wirtz:
>
> On 5 hours now of sleep so I can form sentences:
>
>  
>
> >Your micro-optimizing suggestions for Java ...I think all Java guys are 
> like ROFL now.
>
> >Star visa Full Imports, String Concatenations in this example visa 
> StringBuilder
>
> Y’all’d be right if this was 100 lines of code in Java 101.
>
>  
>
> What you aren’t looking at is that what is slow in this code set is object 
> creation and object recycling.
>
>  
>
> You are right that
>
>  
>
> Import Io.*
>
>  
>
> And 
>
>  
>
> Import Io.Blah
>
> Import Io.Blah2
>
>  
>
> Results in the same code,
>
>  
>
> What doesn’t result in the same code is doing import Io.Blah in classa and 
> import io.blah2 in classb and io.blah3 in classC because it changes the 
> order and timing of the Object and imported Class Creation.
>
>  
>
> + doesn’t matter if you do it once, but at one point this code is doing up 
> to 15k +’s in a loop, and + doesn’t do efficient recycling of 
> temporary/intermediary objects.
>
>  
>
> So laugh all you want, but there are serious gains in profiling the code 
> and seeing where object and memory manipulation are sub optimal, especially 
> when you have long loops.
>
>  
>
> The problem I see with you guys is that you don’t test, “REAL” application 
> you test micro bits of code and don’t look at the impact or history of how 
> the code got to where it is.  You say “it doesn’t matter in 4 lines of 
> codes” but you don’t have “THOSE” 4 lines of codes in your app, you have 4 
> lines intermingled in with 80 other lines that change what is going on.
>
>  
>
> There are times when just changing the order of your Imports can add or 
> remove 200ms to your app.
>
>  
>
> As to architecture changes… No one liked those. Don’t calculate hashes if 
> you are already in a session, use memcache and instance cache. Thread 
> really long loops.
>
>  
>
>  
>
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/mV8RrorNY2oJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsub

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

>the same problem like separating end points into micro apps - complex
business code doesn't work that way - you have to load all that stuff
anyway.

First, but not all at once.  Which I addressed in Lazy Load, but since you
are still attacking the micro-app / API model

 

This sample app has a “Places” that is the slowest part of the code, and
loads things nothing else uses. Spinning that off to an API handled by a
backend would make a HUGE difference. Any functionalized code can be
converted to standalone apps, and the overhead is that you have a wrapper,
but the advantage is that you have huge memory and caching advantages.  So
any very slow memory intensive function can be converted to an API
especially if it doesn’t need to modify globals. If it does need to modify
globals you can do that on return.

 

Say for example you have an imaginary app that does business stuff.

Selling stuff and searching stuff.

 

Searching works better the more in instance caching you can do.  You might
want an F4 just for the memory.  And because of the shared memory 128
+128+128+128 doesn’t have the cache hit ratio of 1 512.  Because of that an
F4 with 100% of its available memory dedicated to caching search info your
single purpose F4 could serve as many search requests as 8 F1s. but even
when that isn’t the case if your “Idle” is 78 megs, an F2 will still idle at
78, but instead of having 50 megs for cache will have 178.   Even better if
that is the only place you need a wrapper for the complex queries to the
datastore this is the only place you need Factory or Objectify.

 

The selling stuff app is likely the only place you need Crypto libraries and
because you want to cache sessions again having more memory in instance
dedicated to the task of session management increases your cache hits by
orders of magnitude.

 

In both cases reducing the number of imports and the frame works required
for both of these “microapps” is going to reduce Cold start time.

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Tuesday, July 24, 2012 5:25 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

You should really focus an general architecture of the app for
optimizing...havn't looked into the provided example.

Your micro-optimizing suggestions for Java ...I think all Java guys are like
ROFL now.

Star visa Full Imports, String Concatenations in this example visa
StringBuilder

Really, if I carefully craft line numbers or embed no debug info in classes
I can generate byte-identical class files with either import and string
style.
Import statements are not stored at all in Java class code and this
string+string in one expression automatically use the StringBuilder pattern.


you can lazy load classes - I just fear you mean something different ;)
and it's the same problem like separating end points into micro apps -
complex business code doesn't work that way - you have to load all that
stuff anyway.




Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz:

I like that you use +'s to concat strings, it shows a real lack of 
experience doing optimizations since that is the very first thing on every 
list. 

> System.out.println(new Date() + ":" + new 
Random().nextInt()); 

At least most of your code uses objects correctly I didn't find any 
instances of places you did a type conversion functionally. That is usually 
one of the lowest hanging fruits, because it creates additional objects, and

is a memory hog. 




-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

With a shower so the sentences might be a little better.

Lazy Loading classes is a variation on the same Object creation stuff.

 

On warm up all you are doing is initializing Objects, and creating classes.
If you lazy load those, whole branches of your code won’t do that until you
need to use those things.

 

As a result an “Empty” warm up can go from 15s to 2. your first request of
type X will be 4s, and of type y will be 4s and Z will be 4s until you have
paid 20% more than what the non-lazy version would be, but the advantage is
that a “cold” request will be 2+4s not 15s

 

This is especially useful when you have something like the “places” in this
sample code that needs a whole bunch of objects that aren’t used anywhere
else. And uses libraries no other part of the code shares.

 

You aren’t making the code more efficient, and it sucks because you have to
know what dependencies each bit of your code has or things explode (much
like when you do threading) but when your goal is to reduce the maximum
suckage for the end user and instead spread the suckage around. (like the
walmart check out analogy) Lazy Object and Class creation can be a big win. 

 

Lastly, there are some really hierarchical objects in this code, which is
great for Object oriented code writing and is easier on the dev, but I think
that functionalizing some of this would speed up the code base pretty
significantly as Objects with like 4 tiers in their hierarchy don’t store
efficiently. 

 

If you do need those, create them with the hierarchy early rather than
attaching to them later, because you get an intermediary copy of the old
object in creating the new object.

 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Tuesday, July 24, 2012 5:25 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

You should really focus an general architecture of the app for
optimizing...havn't looked into the provided example.

Your micro-optimizing suggestions for Java ...I think all Java guys are like
ROFL now.

Star visa Full Imports, String Concatenations in this example visa
StringBuilder

Really, if I carefully craft line numbers or embed no debug info in classes
I can generate byte-identical class files with either import and string
style.
Import statements are not stored at all in Java class code and this
string+string in one expression automatically use the StringBuilder pattern.


you can lazy load classes - I just fear you mean something different ;)
and it's the same problem like separating end points into micro apps -
complex business code doesn't work that way - you have to load all that
stuff anyway.




Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz:

I like that you use +'s to concat strings, it shows a real lack of 
experience doing optimizations since that is the very first thing on every 
list. 

> System.out.println(new Date() + ":" + new 
Random().nextInt()); 

At least most of your code uses objects correctly I didn't find any 
instances of places you did a type conversion functionally. That is usually 
one of the lowest hanging fruits, because it creates additional objects, and

is a memory hog. 




-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

On 5 hours now of sleep so I can form sentences:

 

>Your micro-optimizing suggestions for Java ...I think all Java guys are
like ROFL now.

>Star visa Full Imports, String Concatenations in this example visa
StringBuilder



Y'all'd be right if this was 100 lines of code in Java 101.

 

What you aren't looking at is that what is slow in this code set is object
creation and object recycling.

 

You are right that

 

Import Io.*

 

And 

 

Import Io.Blah

Import Io.Blah2

 

Results in the same code,

 

What doesn't result in the same code is doing import Io.Blah in classa and
import io.blah2 in classb and io.blah3 in classC because it changes the
order and timing of the Object and imported Class Creation.

 

+ doesn't matter if you do it once, but at one point this code is doing up
to 15k +'s in a loop, and + doesn't do efficient recycling of
temporary/intermediary objects.

 

So laugh all you want, but there are serious gains in profiling the code and
seeing where object and memory manipulation are sub optimal, especially when
you have long loops.

 

The problem I see with you guys is that you don't test, "REAL" application
you test micro bits of code and don't look at the impact or history of how
the code got to where it is.  You say "it doesn't matter in 4 lines of
codes" but you don't have "THOSE" 4 lines of codes in your app, you have 4
lines intermingled in with 80 other lines that change what is going on.

 

There are times when just changing the order of your Imports can add or
remove 200ms to your app.

 

As to architecture changes. No one liked those. Don't calculate hashes if
you are already in a session, use memcache and instance cache. Thread really
long loops.

 

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

You should really focus an general architecture of the app for 
optimizing...havn't looked into the provided example.

Your micro-optimizing suggestions for Java ...I think all Java guys are 
like ROFL now.

Star visa Full Imports, String Concatenations in this example visa 
StringBuilder

Really, if I carefully craft line numbers or embed no debug info in classes 
I can generate byte-identical class files with either import and string 
style.
Import statements are not stored at all in Java class code and this 
string+string in one expression automatically use the StringBuilder pattern.


you can lazy load classes - I just fear you mean something different ;)
and it's the same problem like separating end points into micro apps - 
complex business code doesn't work that way - you have to load all that 
stuff anyway.




Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz:
>
> I like that you use +'s to concat strings, it shows a real lack of 
> experience doing optimizations since that is the very first thing on every 
> list. 
>
> > System.out.println(new Date() + ":" + new 
> Random().nextInt()); 
>
> At least most of your code uses objects correctly I didn't find any 
> instances of places you did a type conversion functionally. That is 
> usually 
> one of the lowest hanging fruits, because it creates additional objects, 
> and 
> is a memory hog. 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

https://img.skitch.com/20120724-xdwyfjbfe7pxeyh2aa98938yxd.jpg

No memory spikes.

Jeff

On Tue, Jul 24, 2012 at 12:51 AM, Drake  wrote:
> Ok so it works its way down to the same levels you are seeing it takes 3-5
> minutes.
>
> That's a first for me as well.
>
> In your production can you hit shutdown an app, hit it and see what the
> memory is 30 seconds after start?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I am crashing now.

Splitting out the app I can make it load much faster. That seems to work.
You would have two "apps" one living only on a backend, and accessed via an
"api" (places)  This gets your warm up way down, and shouldn't cost much
more money.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Ok so it works its way down to the same levels you are seeing it takes 3-5
minutes.

That's a first for me as well.

In your production can you hit shutdown an app, hit it and see what the
memory is 30 seconds after start?



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

That's on your stock code. No changes.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Tue, Jul 24, 2012 at 12:37 AM, Drake  wrote:
> I can't get over how much memory this thing uses before it does something.
> Like just warming up I'm at 248M of memory.

I'd start looking at what you've changed, or some defect in your test
environment.

https://img.skitch.com/20120724-cwa7r2g9x774aj4rb6xgmbiywa.jpg

This is the "production" system in normal use, even with appstats
turned on.  There's no instance cycling (other than normal load
expansion and contraction).

You're doing something wrong.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

What does your usage profile look like?  It looks like if I kill off all the
places code I can get this to 86 megs after warm up.

If you put places on a backend, it looks like you could serve this off of an
always one F1 backend, and alwasys on F1 frontend. And your warmups would be
about 2.5 seconds, and you would have a very minor code tweak. Like you
would access places like it was an API.  Places should scale well since it
isn't CPU dense.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I can't get over how much memory this thing uses before it does something.
Like just warming up I'm at 248M of memory.

On an F2 I can do like 5 places and it soft memories and restarts.

On an F4 I do about 350

How many requests are you seeing.

Why do you hate garbage collection so much?

I don't live in objectify, is it putting everything in an instance cache?
Each request seems to add about 1.5Megs to the used memory.

I can't see any difference (300ms) between load speed on an f2 and an F4
That usually means you are object creation bound, but you aren't front
loading the Object creation, so that seems unlikely. (though you aren't
doing lazy classes so maybe you are creating every object in your entire
code at startup?)

Tell me again why you are against Lazy Classes. Like I am the 3 year old you
seem to think I am?



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Tue, Jul 24, 2012 at 12:00 AM, Drake  wrote:
> I like that you use +'s to concat strings, it shows a real lack of
> experience doing optimizations since that is the very first thing on every
> list.
>
>> System.out.println(new Date() + ":" + new
> Random().nextInt());

You're either trolling me or have yet another wild misconception about
Java.  This one is really easy to disprove though:

--- Stringy.java ---
public class Stringy {
public static void main(String[] args) {
String foo = "def";
System.out.println("abc" + foo);
}
}

$ javac Stringy.java
$ javap -c Stringy

Compiled from "Stringy.java"
public class Stringy extends java.lang.Object{
public Stringy();
  Code:
   0:   aload_0
   1:   invokespecial   #1; //Method java/lang/Object."":()V
   4:   return

public static void main(java.lang.String[]);
  Code:
   0:   ldc #2; //String def
   2:   astore_1
   3:   getstatic   #3; //Field java/lang/System.out:Ljava/io/PrintStream;
   6:   new #4; //class java/lang/StringBuilder
   9:   dup
   10:  invokespecial   #5; //Method java/lang/StringBuilder."":()V
   13:  ldc #6; //String abc
   15:  invokevirtual   #7; //Method
java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
   18:  aload_1
   19:  invokevirtual   #7; //Method
java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
   22:  invokevirtual   #8; //Method
java/lang/StringBuilder.toString:()Ljava/lang/String;
   25:  invokevirtual   #9; //Method
java/io/PrintStream.println:(Ljava/lang/String;)V
   28:  return

}

I'm pretty sure you can read that.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Oh, and you are right the biggest slow down is grabbing data via HTTP.
(usually is)
But that doesn't seem to be hurting concurrency.

The obvious question.
Why aren't you mirroring the site so that you have your own fast copy?



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

> That makes absolutely no sense.  The app doesn't cache data in instances;
> there's utterly no point to using weak references.

Your app runs for multiple seconds, you get garbage collection in that time
and since you also feel no need to destroy objects you aren't using... this
would be a lazy way to deal with it.

> "this" + "that" is compiler syntactic sugar for using a StringBuilder to
assemble
> the string.  There is no difference between the two.  It's probable that I
could
> do the same trick I did with imports and create with two classes that
> generate identical bytecodes, but figuring out the exact pattern is more
> effort than I care to spend at the moment.

No, the intermediary objects are different and you have a much larger memory
foot print using loops with + than stringbuilder
For Print "this" + "that" you are essentially correct, but for 1000+ loops
the difference is huge.  Typically for small stuff it isn't worth messing
with but you have some long loops of this.


>
> Also, never use StringBuffer; it's a legacy class from JDK 1.0 which
> unnecessarily synchronizes method calls.  StringBuilder is the
replacement.
>

Yes I misspoke. I have been up for 48 hours and would be sleeping except
someone said I couldn't do hello world in 1.5 seconds and I missed it by
100ms on an F1, so I didn't want to be slow getting response to this.


> > You do a lot of .Trim when you should use Vector
>
> I'm genuinely afraid to ask, but... huh?  Vector? (!!!)

Vector resizes an object without creating an intermediary or placeholder
object, so there is nothing for garbage collection to deal with, and you
don't have to create a new object which gets renamed to the old object.
(which is what trim does)

Again a Pain in the ass for doing a trim on something once, but when you
have it in a for loop it adds up.





-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

I feel like I need to keep following up just in case a Java newbie
takes this advice seriously.

On Mon, Jul 23, 2012 at 11:14 PM, Drake  wrote:
> Also None of your references are weak? Do you just hate garbage collectors?
> That won't help much with startup, but again it would reduce your need to
> startup as often.

That makes absolutely no sense.  The app doesn't cache data in
instances; there's utterly no point to using weak references.

> You do a lot of String + String..  use stringbuffer instead

"this" + "that" is compiler syntactic sugar for using a StringBuilder
to assemble the string.  There is no difference between the two.  It's
probable that I could do the same trick I did with imports and create
with two classes that generate identical bytecodes, but figuring out
the exact pattern is more effort than I care to spend at the moment.

Also, never use StringBuffer; it's a legacy class from JDK 1.0 which
unnecessarily synchronizes method calls.  StringBuilder is the
replacement.

> You do a lot of .Trim when you should use Vector

I'm genuinely afraid to ask, but... huh?  Vector? (!!!)

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I like that you use +'s to concat strings, it shows a real lack of
experience doing optimizations since that is the very first thing on every
list.

> System.out.println(new Date() + ":" + new
Random().nextInt());

At least most of your code uses objects correctly I didn't find any
instances of places you did a type conversion functionally. That is usually
one of the lowest hanging fruits, because it creates additional objects, and
is a memory hog.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Odd. I get different results than you do.

Quite different actually.

.* in the sandbox is slower (by a lot)

.* on f4 is faster

Also, I think you are very wrong about dynamically/lazily loading classes.
Since my version warms up in 2.5s on an F2. (though I have an initialization
missing somewhere cause when I do places I get an error about an object)







-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

Brandon.  This pit is just getting deeper and deeper.  When are you
going to give up?

On Mon, Jul 23, 2012 at 10:55 PM, Drake  wrote:
> Really?
>
> You don't think that doing lazy class loads would speed up your startup?

The JVM lazy loads classes automatically.  This is just how it works.

> Did you benchmark Dotted Imports vs full import, or "less dotted"?  I
> knocked 1s of the startup just by * importing objectify, that you took all
> of but did so in 6 places.

Wow, I even gave you the answer ahead of time, and you *still* jumped
into the trap.

Here's a little experiment for you:

-- File m1/Main.java -
import java.util.*;

public class Main {
public static void main(String[] args) {
System.out.println(new Date() + ":" + new Random().nextInt());
}
}
-- File m2/Main.java -
import java.util.Date; import java.util.Random;  // on same line so
line# info doesn't change

public class Main {
public static void main(String[] args) {
System.out.println(new Date() + ":" + new Random().nextInt());
}
}

Notice that the only difference is the wildcard import vs the
fully-qualified import.

Compile each.  Diff the resulting classfiles.

Good work on that startup time improvement.

I'm pretty much done here.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I blew up or created some circular reference because it's late and I had a
lousy week... I'm not familiar enough with the code, but you have several
objects whose creation should be moved to the top of your code. And a few
places you should swap to mutable returns.

The mutable returns won't speed up startup, but will drop your memory usage
significantly.

Though it looks like a crap ton of memory is being allocated, more than
should be needed, for what I can tell is going on.

Do your logs have a lot of Soft Memory limit errors?   There is what I would
call a bug in GAE where if your only instance is killed by the Soft memory
limit that the replacement instance loads really slow. I think because the
pending request queue is transferred to the new instance.

I think your biggest win would be in lazy creating the objects and classes
that you don't use much.

Next is
Threading your large loops. Caching your Hashes. Changing your trims to
vectors, and your Concats with + to use stringbuffer. Weak the stuff that is
temporary variables so they don't get clung to.





-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Also None of your references are weak? Do you just hate garbage collectors?
That won't help much with startup, but again it would reduce your need to
startup as often.

You do a lot of String + String..  use stringbuffer instead

You do a lot of .Trim when you should use Vector

And yes in a few spots that would change your warm up. (not as much as it
would drop your memory foot print)


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Really?

You don't think that doing lazy class loads would speed up your startup?

Did you try?

Did you benchmark Dotted Imports vs full import, or "less dotted"?  I
knocked 1s of the startup just by * importing objectify, that you took all
of but did so in 6 places.

No, you never try anything.

I think I made it clear that threading wouldn't change the startup, it would
speed up the app. And since your issue was startup plus response time and it
appears that can take 6-8 seconds and tests at 2-5 after threading that
would appear that would make your QoS much better, and based on the load you
were describing that would seemingly reduce your number of starts, and need
for more instances.

You aren't doing hash caching. That's a huge, easy speed up. Even if you
don't "downgrade" to sha1, that will make a huge difference in your need to
spin up more instances.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

Brandon, you have absolutely no idea how the JVM works.

On Mon, Jul 23, 2012 at 8:41 PM, Drake  wrote:
>
> And not having time to sort through the duplicate imports which are
> definitely killing load time...
>
> And ignoring the fact that you are doing Dotted imports which add up to most
> if not the entire import which is slowing things down even more…….  (Stars
> aren’t always bad because in GAE the IO is more of an issue than the compile
> time)
>
> And that those 3 things would likely speed load time by a lot, if not all of
> my 30% goal….

Try it.

I was tempted wait and see what kind of results you got ("not as much
as I had hoped" would have been hilarious), but that would just be
mean.  Those changes aren't going to help because they have no effect
on compiled code.  Literally, the Java compiler produces the *exact
same output*.  This isn't Python; Java imports are just syntactic
sugar to the compiler.

I wasn't quite sure what to make of "not a single lazy load in the
project" so I'll just assume you don't have the faintest clue how
classloading works.  But go ahead and suggest some specific code
changes that you think are going to make the app startup faster.

> The PolyLines String builder isn’t threaded, and profiling the app that
> seems to be really frakking slow.
>
> Create 4 threads and have each take ¼ of the array.
>
> I haven’t implemented the code but that should be about a 45% increase in
> speed.

Just.  Wow.

First of all, the polylines string builder has no effect on instance
startup time.  Try hitting urls which don't render polylines... same
10-15s startup time.

Second of all, you have a broken mental model of the relative
performance of various operations.  It might be different in Python,
but I kinda doubt it.  The computational cost of creating,
synchronizing, and joining a thread (plus all the security checks that
Google adds) are at least an order of magnitude more, and probably two
or three orders of magnitude more expensive than the cost of
assembling a polyline.

Seriously, try it.  Especially try it in production.

> Am I required to actually check the changes in? or can you guys handle
> threading the slowest part of the code? ( I assume I’m optimizing for low
> volume and reduced latency)

You can submit a pull request if you actually come up with a real
improvement to startup time.

> Can I replace your Sha256 with something else? That is also eating a lot of
> CPU time and I can’t really tell from the 5 minutes looking at the code why
> you are doing that. Sha1 should suffice and would buy you 50% increase in
> speed on those requests that use it. And as far as I can tell that looks
> like all authenticated requests?  It also doesn’t look like you are caching
> your HashCalculations? Based on privacy and such you may not want to
> memcache those… but at least store some number of them in instance memory.

This has nothing to do with startup time.  And isn't a problem.  And
what you say here doesn't really make sense anyways.

If you're wondering why /places requests are slow, I already know the
answer:  Wikimapia is slow.  Not my problem:

https://img.skitch.com/20120724-q517rqfg7mfhbndnbn4nfmwn89.jpg

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

PS, sorry I was driving home between when you through down the gauntlet, and
10 minutes before I posted this. And I had to pee, which slowed me down.

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Skipping the start time issue since there doesn’t appear to be a single lazy
load in the whole project….

 

And not having time to sort through the duplicate imports which are
definitely killing load time...

 

And ignoring the fact that you are doing Dotted imports which add up to most
if not the entire import which is slowing things down even more…….  (Stars
aren’t always bad because in GAE the IO is more of an issue than the compile
time)

 

And that those 3 things would likely speed load time by a lot, if not all of
my 30% goal….

 

The PolyLines String builder isn’t threaded, and profiling the app that
seems to be really frakking slow.  

 

Create 4 threads and have each take ¼ of the array.

I haven’t implemented the code but that should be about a 45% increase in
speed.

 

Am I required to actually check the changes in? or can you guys handle
threading the slowest part of the code? ( I assume I’m optimizing for low
volume and reduced latency)

 

Can I replace your Sha256 with something else? That is also eating a lot of
CPU time and I can’t really tell from the 5 minutes looking at the code why
you are doing that. Sha1 should suffice and would buy you 50% increase in
speed on those requests that use it. And as far as I can tell that looks
like all authenticated requests?  It also doesn’t look like you are caching
your HashCalculations? Based on privacy and such you may not want to
memcache those… but at least store some number of them in instance memory.

 

 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Jon Stevens
Sent: Monday, July 23, 2012 7:35 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

On Monday, July 23, 2012 6:37:15 PM UTC-7, Brandon Wirtz wrote:

CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. I 
don't even promote the damn thing it helps with my old SEO clients. Our 
analytics and Ad management platforms are full on business apps. We do real 
database stuff. We did 1million write ops in an hour the other day. ( I can 
post pictures) we do queries and reports, and we use the lowlevel API for 
all that. 

(we can do a Join in Lowlevel we don't even use GQL ) 

 

Unfortunately, both Jeff and I have prior experience with working with
people just like Brandon.

 

All talk (mostly gibberish), but no listen and no do.

 

Brandon, your next response on this impossibly long thread should be a link
to a pull request showing how you've made motomapia start up 30% faster
while playing by the rules Jeff sent out. Nothing more, nothing less. Any
other response will be proof that you are all bogus talk and no action.

 

It's time to put up or shut up.

 

jon

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/AwjWPmk8szMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jon Stevens]

On Monday, July 23, 2012 6:37:15 PM UTC-7, Brandon Wirtz wrote:
>
> CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. 
> I 
> don't even promote the damn thing it helps with my old SEO clients. Our 
> analytics and Ad management platforms are full on business apps. We do 
> real 
> database stuff. We did 1million write ops in an hour the other day. ( I 
> can 
> post pictures) we do queries and reports, and we use the lowlevel API for 
> all that. 
>
> (we can do a Join in Lowlevel we don't even use GQL ) 
>

Unfortunately, both Jeff and I have prior experience with working with 
people just like Brandon.

All talk (mostly gibberish), but no listen and no do.

Brandon, your next response on this impossibly long thread should be a link 
to a pull request showing how you've made motomapia start up 30% faster 
while playing by the rules Jeff sent out. Nothing more, nothing less. Any 
other response will be proof that you are all bogus talk and no action.

It's time to put up or shut up.

jon

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/AwjWPmk8szMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. I
don't even promote the damn thing it helps with my old SEO clients. Our
analytics and Ad management platforms are full on business apps. We do real
database stuff. We did 1million write ops in an hour the other day. ( I can
post pictures) we do queries and reports, and we use the lowlevel API for
all that.

(we can do a Join in Lowlevel we don't even use GQL )



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Mon, Jul 23, 2012 at 5:26 PM, Drake  wrote:
> But you keep slamming my CDN because you like those
> other "CDN guys" but it is just an enabling technology for everything else I
> do.

I really am not slamming your product; for all I know it's amazingly
wonderful.  What I object to is that you are taking optimizations
peculiar to one very unusual, specialized app and declaring them as
universal truths for everyone - oh, and if we object, we're being lazy
or whiny.

The technical challenges of your CDN are very different than the
technical challenges of a business application.  It may make sense for
you to break your app into tiny independent pieces, but you can't do
that when you have crosscutting concerns like transactions or
elaborate data dependencies.  It may make sense for you to model your
domain with the low-level API, but it would be insanity to try this in
an app with hundreds of entity types and hundreds of thousands of
lines of code, not to mention multiple engineers coming on and off the
project over years.

All these frameworks we use have evolved for good reason, because they
make it easier to write code and scale the complexity of an app.  It's
not a solution to say "just don't use them" simply because you have
managed to build one particular type of app using bare metal
programming.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

August 25th I have an app launch. It is no small thing. My CDN is a Toy in
the view you have of it.  But the internal version is designed to make all
the other stuff work. It is a load balanacing url handling, code
modularizing tool.  But you keep slamming my CDN because you like those
other "CDN guys" but it is just an enabling technology for everything else I
do.

Because of that tool I can do www.myappp.com/user/12354 as an app in python,
and www.myapp.com/products/12345?view=json in Java, and
www.myapp.com/search/frogs as Go.  And if I really want, /checkout/ can go
to PHP on Dreamhost.  That gives me infinite more control.  It also lets me
share databases because java.myapp.appspot.com can share data with
python.myapp.appspot.com  . So that comment that you can't convert your code
to Python... You don't have to convert all of it, or all of it all at once.

Maybe my bias is that because without such tech you have to do web.xml stuff
or that it is hard to modularize your app that I'm so jaded by it's
awesomeness that I can't see past, but...

Typically when a company asks me to refactor code the first thing I do, is
break their code in to groups that share classes/imports/frameworks and
create 3-4 apps from those, and then use a "load balancer" or a "front end
proxy" or a "Templated API wrapper" to make that app faster and more
scalable.

Step two is convert as much as possible to lowlevel api.

Step 3 is to strip everything out of every framework that they aren't using.

Step 4 is to create flatter versions of their data to make queries simpler,
faster, and more friendly to the lowlevel API

Step 5 is to prioritize writes and reads so that things can be delayed if
they aren't for realtime data

Step 6 is to optimize start ups using serialization of configuration
strings, and other data. Pushing any file streams to Memcache

Step 7 is to create a task database to create precaclulated versions of
common requests, and to handle cron and steady state tasks.

Step 8 is to make determinations about lazy loading. (I try hard to not lazy
load because it makes your app harder to profile, because if you lazy load a
large frame work you may over run the soft memory limit. If you do
everything up front then you know what your steady state memory usage is)

Step 9 is to self manage parts of the memory. Rather than relying on garbage
collection start throwing away memory that you aren't using.

Step 10 is to profile decision trees, to optimize order of operations.

Step 11 is to manage threading (this is often late in life because it makes
debugging hard, and the gains are often not amazing in apps with lots of
simultaneous users)

Step 12 is to strip all the debugging, logging and non-esential stuff from
the code.

Rinse and repeat.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

I can't believe I'm keeping this thread going, but...

On Sun, Jul 22, 2012 at 10:15 AM, Drake  wrote:
>   And your
>> absurd proposal that all 326 of my url endpoints should be separate
>> applications...  Just.  Wow.
>
> If you had actually read what I posted I said that you should group by task
> type, and used class so that you had many smaller apps, that were optimized
> for like tasks and minimizing classes.

Your exact words were "Each of what you call and End point should be a
micro app. A single purpose App that handles one type of request."
But let's move on.

I was a tech lead in EA's online platform group - hundreds of services
among thousands of machines - so don't lecture me about service
oriented architectures.  And that's not even close to the most
impressive thing on my resume.  Now, can we put our dicks back in our
pants?

There is a proper scale for breaking down a monolithic application
into separately deployed modules, but this isn't it.  In real-world
business apps (the kind that - unlike your CDN - represent 90% of GAE
users), transactional boundaries place limits on how you can decompose
an application since you can't (without getting *really* crazy) run a
transaction across instances.

I'm glad that you've figured out how to tightly optimize serving
static content through GAE.  I'm sure your CDN code is lovely and well
suited to its painfully narrow task.  But you're trying to generalize
a hack - "write at the lowest API level all the time" - to projects
that are simply too complicated for that kind of hack.  With broader
experience, you would understand this.

> If you want it to be constructive, post your code and I'll whack 30% off
> your start time just to get you to stop griping about how it is not possible
> to speed up start times because Google sucks.

Cute.  I'll make you an equally pointless offer:  If you want to fly
out to San Francisco, I'll show you the code for a modern business
application and you will see for yourself why your optimization
strategy is farcical.  The difference between your pointless offer and
my pointless offer is that if you actually took me up on it, I could
deliver.

However, if you want to play a game, try optimizing this demo app:

http://www.motomapia.com/

The code is linked from the header.  Typical instance startup time is
10-15s.  Here's the rules of the game:

 * The @Path annotations must keep working
 * The @Transact(TxnType.REQUIRED) annotation must keep working
 * Datastore access must be through typed POJOs.

Let me predict the response:  "Too many frameworks!  Use low-level
API!  Use web.xml!  Manipulate transactions by hand!"

All of that works in a toy demo but not in an app with hundreds of
endpoints, dozens of polymorphic entity classes, and complicated
transactional logic.  Essentially, your optimization strategy is to
write code in a way that does not scale to the complexity of a modern
business application.  Pure fail.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

On GAE it's not just the startup.. Well it is, but for more reasons. You
have less memory too. So someone's really optimized framework might be
AMAZING for CPU optimizations, but the cost of adding it from a memory
standpoint causes you to hit the soft limit instance death more often. Which
causes you to cold start A LOT more because the scheduler didn't spin the
instance, the instance ceased to exist.

 

And frankly, this discussion hinges entirely on instance cold start issues.
If you have enough money to keep spare idle instances sitting around, or
your traffic is steady and constant (as opposed to spiky and unpredictable)
you really won't hit this problem very much.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

On Monday, July 23, 2012 10:38:12 AM UTC-5, Paul v wrote:
>
> Well I love the who's-a-better-guru argument.  I've learned a ton reading 
> this thread, I hope they keep going at it.


Just out of sheer curiosity, what exactly have you learned from reading 
this read? Really, the entire discussion between Mr. Wirtz and Mr. 
Schnitzer distills down to optimizing your code as much as possible. Which 
is a really good idea regardless of what hosting platform you pick. 

And frankly, this discussion hinges entirely on instance cold start issues. 
If you have enough money to keep spare idle instances sitting around, or 
your traffic is steady and constant (as opposed to spiky and unpredictable) 
you really won't hit this problem very much.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/V4-frxZVYSEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

It's not a who's a better guru.  I'm sure Jeff writes better Java than I do.
But "This aint your grandmother's java".   I participate in a HipHop (php)
discussion group too, and people will say "Well XY and Z aren't possible
with out ZEND"  well, no they are just hard without it.

If you are running Wordpress for your personal blog you can do PHP on
DreamHost. If you are Facebook you use HipHop. Writing a small App with no
performance requirements you can use PHP and Zend. But at scale you need
that "Simpler" version of the language.

 

Even Python on AppEngine isn't "Python" you don't get all the C stuff. The
Fortran stuff. 

 

You are Accessing Google API's via a language. You can pick your language,
Python, Java, Go, but you are writing in "AppEngine" it's there world we
just play in it.

The good news, if you write to those rules and wrap the API's in
Class/functions of your own (which you need to make things work in the
sandbox), your code will be scalable portable, and happy on anyone else's
platform.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Paul v
Sent: Monday, July 23, 2012 8:38 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Well I love the who's-a-better-guru argument.  I've learned a ton reading
this thread, I hope they keep going at it. 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/bXo-VqpW7LQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Paul v]

Well I love the who's-a-better-guru argument.  I've learned a ton reading 
this thread, I hope they keep going at it.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/bXo-VqpW7LQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Richard Watson]

Personally, I don't care much for the who's-a-better-guru argument, doesn't 
get us any closer to a solution.

App Engine proves Joel Spolsky's "all abstractions are leaky" statement. An 
abstraction is nice when it works but you'd better know what's going on 
under the hood when it doesn't. Most of us are willing to put in extra 
effort to make our apps GAE-friendly upfront, but everything's about 
balancing available resources given our specific app and situation. If we 
weren't willing, we'd be using something else. Some have had more 
opportunity than others to refine their approach, and some apps suit the 
platform better than others. But we'll do better if we respect the opinions 
of the other participants a touch more, or at least try keep comments about 
the ball, not the man.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/NzEYDO9XK-gJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I have a similar theory.

We have 9 Million instances, Let's look busy.  :-)

I think that they assign instances to somebody at all times, and if that
somebody suddenly needs one it raises the QoS.  Doesn't cost anything to
idle instances that weren't doing anything. And you make the customer happy
if you avoid a warm up.



> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of hyperflame
> Sent: Sunday, July 22, 2012 10:28 AM
> To: Google App Engine
> Subject: [google-appengine] Re: Startup time exceeded...on F4?!
>
> On Jul 22, 5:12 am, Aleksei Rovenski 
> wrote:
> > Regarding min idle instances, I must admit there is something strange
> > going on in my Instance tab in Dashboard.
> > I have settings like this: idle instances min=auto, max=1, pending
> > queue min and max=15s. I have some working instances that I get
> > charged for and then few more that are anyway created by google just
> > waiting out there for hours and even days and I do not seem to pay for
> > them. If I kill them, they are coming back anyway. This looks nice
> > indeed, but could someone explain me what is behind that. I'm worried
> > because it may be a drug that google will suddenly stop providing me
> > :)
>
> I get that as well. My theory is that at non-peak times, the Google
scheduler
> opens up extra instances to be ready for spikes.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

  And your
> absurd proposal that all 326 of my url endpoints should be separate
> applications...  Just.  Wow.

If you had actually read what I posted I said that you should group by task
type, and used class so that you had many smaller apps, that were optimized
for like tasks and minimizing classes. Which is just good design because
then you can run fewer instances likely move to f1s or f2s and better
utilize in instance memory, hitting the soft memory limits less.

And no it was not a constructive conversation it was a Gripe fest.

If you want it to be constructive, post your code and I'll whack 30% off
your start time just to get you to stop griping about how it is not possible
to speed up start times because Google sucks.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Sun, Jul 22, 2012 at 2:12 AM, Drake  wrote:
>
> Let's see... You spend your life complaining how the platform sucks. I
> release tutorials on how to make it suck less.  I'm the troll?

This conversation was constructive and mostly positive until you
chimed in.  It was a reasonable discussion of real problems that many
people are having with scheduler behavior, including concrete
suggestions for how to improve it.

For the record, I have never, ever said that GAE sucks.  Your advice,
on the other hand...

> Anyone who thinks the low level api is analogous to assembly probably
> doesn't want to depend on me because the laughter when they asked a question
> might be deafening.
>
> Read EVERY optimization guide on Java for GAE and that is the first thing
> they tell you to do.

You have this wrong.  It's true that the JDO/JPA module gets a bad
rap, but every optimization guide I've read recommends abandoning it
in favor of lighter-weight layers like Objectify or Twig.  It's true
that these tools will give you a better startup time because they
don't do classpath scanning.  But it turns out that avoiding classpath
scanning only gets you so far - the up-front cost of classloading and
inspecting persistence classes becomes significant when you reach a
critical mass of entity classes.

As for writing complex business applications using strictly the
low-level API... good luck.  I can only explain your attitude about
this as naïveté.  And your absurd proposal that all 326 of my url
endpoints should be separate applications...  Just.  Wow.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Sun, Jul 22, 2012 at 2:05 AM, Aleksei Rovenski
 wrote:
> I can understand that GAE is more optimized for python than for java.
> Maybe it is a highly specialized tool for really tiny apps that use no
> frameworks. But I don't get one thing. How Google plans to compete for
> java apps by selling platform that forces you to not use frameworks? I
> simply refuse to go back into the 90's (in software development
> aspect, no problem otherwise :)). I simply do not want to drop
> dependency injection and other patterns I have been doing for many
> years already. If I must, then I think it just isn't worth it.
> Seriously Java is about frameworks. Saying not to use frameworks in
> java is same as use python instead. No thanks.

Don't listen to Brandon.  By and large you can use frameworks and
*should* use frameworks, because "framework" is just a word for code
you would otherwise have to write yourself.  Google doesn't want to
send you back to the late 90s programming styles (even if Brandon
does), and nobody sane will seriously suggest you break your app into
hundreds of tiny deployments.

It is a known issue with the GAE Java environment that instance starts
for nontrivial applications take significant time -- enough to
negatively impact the user experience if a user request gets routed to
a cold start.  Some frameworks make this problem worse than others,
but ultimately it is a linear relationship with the amount of code in
your app.

Maybe Google will be able to radically improve instance startup time
in the future, but probably not.  There have been incremental
improvements but it's a difficult problem because the JVM loads code
very differently from Python or Go.

The key is to prevent users from seeing cold starts in the first
place.  Takashi has posted one workaround:  Keep "min idle instances"
high enough that there is always enough capacity.  This works but it's
not ideal.  It costs more and there is always the chance that a sudden
burst of traffic will overload the idle instances and route a request
to a cold start.  You can decrease the chance of this by adding even
more min idle instances but that costs even more.

My hope is that the scheduler can be made smart enough to always keep
requests in the pending queue until new instances are warmed up.  I
believe that GAE used to do this, but something recently changed so
that it no longer does.  If requests never see cold starts, it really
doesn't matter how long your app takes to start - you can optimize
capacity for time-to-service-each-request latency, instead of the %
chance that a user request will get sent to an instance for 20+
seconds.

So - don't panic.  There is a solution that works right now, assuming
you have reasonably smooth traffic patterns:  Pay a bit more than you
would otherwise for more idle instances.  But also star this issue,
which IMHO is a better long-term solution:

http://code.google.com/p/googleappengine/issues/detail?id=7865

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I don't use frameworks in Python either. Ask my Dev team. They all but riot 
when they find a library they want to use and I say, figure out what it does 
and make your own, we don't have room for that thing.

Java aint Ruby on Rails. (or Oh please don't let it be) I get that the appeal 
of Java is that it is older than dirt and there is a framework for everything. 
You can use 1 frame work if it is a small one :-).

You are trading rapid elasticity for ease of development. You have to remember 
that when you are small Servers are cheap and humans are expensive. When you 
get to Google Scale that reverses.  GAE is optimized for near Google scale. 
(not all of Google, but Small Google product) if you are 1/1000th that the dev 
resources for optimization will kick your ass. Just like the guys who say they 
wont' do it.  Those are guys who don't spend $50k a month on hosting. When the 
amount you spend is double the cost of a dev, it is time to look at ways to 
cut that hosting in half using a dev, because you will break even, and give 
higher QoS.




> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Aleksei Rovenski
> Sent: Sunday, July 22, 2012 2:05 AM
> To: Google App Engine
> Subject: [google-appengine] Re: Startup time exceeded...on F4?!
>
> I can understand that GAE is more optimized for python than for java.
> Maybe it is a highly specialized tool for really tiny apps that use no
> frameworks. But I don't get one thing. How Google plans to compete for java
> apps by selling platform that forces you to not use frameworks? I simply
> refuse to go back into the 90's (in software development aspect, no problem
> otherwise :)). I simply do not want to drop dependency injection and other
> patterns I have been doing for many years already. If I must, then I think 
> it
> just isn't worth it.
> Seriously Java is about frameworks. Saying not to use frameworks in java is
> same as use python instead. No thanks.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

> All I have to say is "wow".  I'm really glad you're just a troll here and
not
> actually responsible for anything I depend on.

Let's see... You spend your life complaining how the platform sucks. I
release tutorials on how to make it suck less.  I'm the troll?

Anyone who thinks the low level api is analogous to assembly probably
doesn't want to depend on me because the laughter when they asked a question
might be deafening.

Read EVERY optimization guide on Java for GAE and that is the first thing
they tell you to do.  If that is too hard, then this is not the platform for
you.  I have vouchers for Azure, hit me up off list I'll send them along,
you will be happier in .Net where you never have to do anything low level.
J# is supported for 3 more years... So you should have plenty of time to
learn basic principles of cloud development in that time.


All animosity aside.  I'm sure you are a good Java dev, but this isn't Java.
The Python I write isn't Python either.  I mean it is, but not how you would
write it if you were on a "real" server. You have to get over that. You are
going to suck at GAE until you do. NO FRAMEWORKS. NONE. Deal with it. Every
email you beat me over the head with why you need PMF. Well Frak PMF it is
holding you back. It is making you stupid. You don't need it. If you need it
for one really hard thing once in a great while that is what a lazy load is
for.

Yes classes suck we went over that. Which is why you build micro apps that
are purpose built and only use limited classes.

Yes I have an F8 that can be anyone of my Micro Apps, it has 63k Entities.
It lazy loads. So it doesn't break, It is used for tests and for emergency
off loading of traffic.
You don't have that kind of traffic so you don't need that, I get that.

But you need to get over yourself and look at profiling your app, and
determining where you can cut the fat, and how you can break a single
monolithic app in to 4 smaller apps.  I don't know what that looks like,
maybe it is Customer authentication and session handling, Credit card
processing and Order insertion, and Product Look up and shipping
calculation.

In so doing you can limit your classes, you can make better use of your
instance memory, and you can likely keep from ever hitting warm ups because
each app will run more efficiently than the monolith.

Yeah that is a lot of work, and sure it sucks that this guy who called you
stupid told you to do it, but it is the right thing to do.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Sun, Jul 22, 2012 at 1:34 AM, Drake  wrote:
>
> I know what an "end point" is. Apparently you don't.  Each of what you call
> and End point should be a micro app. A single purpose App that handles one
> type of request.

All I have to say is "wow".  I'm really glad you're just a troll here
and not actually responsible for anything I depend on.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

200 1625ms 98kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4)
AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6
70.162.197.131 - - [22/Jul/2012:01:38:53 -0700] "GET /www.xyhd.tv HTTP/1.1"
200 98752 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4)
AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6"
"second.cdninabox-java.appspot.com" ms=1626 cpu_ms=844 api_cpu_ms=0
cpm_usd=0.042899 loading_request=1
instance=00c61b117c382cd6d2d719025a767570ad95c0
I 2012-07-22 01:38:53.885
This request caused a new process to be started for your application, and
thus caused your application code to be loaded for the first time. This
request may thus take longer and use more CPU than a typical request for
your application.

I read that as 1.626 MS.

What I did in that time on an F1.
Initialized the App from nothing.
Read the configuration out of Datastore (not even from Memcache)
Did a URL Fetch to read the contents of XYHD.tv (like you doing your read
from file)
Regexed it to change all the paths from relative to absolute.
Deferred a write to the datastore so I wouldn't have to do that for the next
request that comes along in the next 30 seconds.
Output the data.

I still have 370 Ms to do something else.  And that is on an F1.

I think that counts as Hello world.

> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Sunday, July 22, 2012 1:18 AM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> On Sat, Jul 21, 2012 at 11:32 PM, Drake  wrote:
> >>  * How do you persist data?  (low-level, jdo, objectify, etc)
> > Actually I use a multi-approach based on the work being done.  Python
> > has NDB, CachePy, and a number of things that Java seems to missing
> > good analogs for, but using the lowlevel API gets things up and
> > running and does the majority of the heavy lifting. When we don't want
> > to have to manage a bunch of code to handle stuff that is more complex
> > or do things that are less time sensitive like deep queries for single
> > users, we use JPA. (I think JDO is mostly dead.)
>
> This thread is about Java instance startup time.  Enough about Python
> already.  It's irrelevant.
>
> You've completely - and lamely - dodged ALL of my questions.  You
insinuate
> that you have sophisticated Java applications that start up quickly.  I'm
calling
> bullshit.  Give me ACTUAL numbers for one ACTUAL Java application.
>
> >>  * How many entity kinds/classes do you have?
> > A shit ton. For the analytics app we built we found out that you are
> > limited to 65535 and that Dynamic is not as dynamic as you might think.
>
> Generated kinds or classes don't count.  I mean this literally, how many
user-
> defined entity types does your Java app load?  You're going to tell me you
> have Java apps that load 64k JPA classes?  Not a chance.
>
> See, this is getting to the ACTUAL problem with the Java environment -
> classloading is slow.  Entity classes must be loaded and introspected
before
> datastore operations can start.  Near as I can tell, the only way around
this
> particular problem is to stay with the low-level api - which, like
assembly
> programming, severely limits how sophisticated an application you can
> reasonably build.
>
> >>  * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
> > How quaint.  Did you miss that I said one type of task per app/backent?
>
> I will be generous and assume for the moment that this makes sense for
> your particular application.  At best you are arguing that you have a
wacky
> application.  You won't find too many people building business apps that
way,
> especially when you have elaborate transactional logic.
>  There are sometimes good reasons to compose an app out of more
> fundamental services, but this should be driven by application design
> - not a dumb limitation of how many classes you can pack into a JAR before
> instance startup times get out of control.
>
> > Web.XML at the individual app, but path routing is handled by the
>
> If I went the web.xml route, it would map 326 servlets.  Not really fun.
> There's a reason people started inventing "web frameworks" in the late
> 90s... because that kind of programming sucks when your app is even mildly
> complicated.
>
> >>  * How many URL endpoints do you have?
> > Per app? Or In total? In many cases we have endpoints per domain in
> > multi-tenant apps, and we are running 10k plus domains.
>
> I was really asking about your "unicorn" java app that loads in 2 seco

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

> I will be generous and assume for the moment that this makes sense for
> your particular application.  At best you are arguing that you have a
wacky
> application.  You won't find too many people building business apps that
way,
> especially when you have elaborate transactional logic.
>  There are sometimes good reasons to compose an app out of more
> fundamental services, but this should be driven by application design
> - not a dumb limitation of how many classes you can pack into a JAR before
> instance startup times get out of control.
>
> > Web.XML at the individual app, but path routing is handled by the
>
> If I went the web.xml route, it would map 326 servlets.  Not really fun.
> There's a reason people started inventing "web frameworks" in the late
> 90s... because that kind of programming sucks when your app is even mildly
> complicated.
>

And this is where you prove your stupidity.

This is how every large "app" on the planet works.
A load balancer serves requests to the right "app" and each app is optimized
for what it is supposed to do. Data is shared between apps using API's,
Memcache, and Datastore.

I know what an "end point" is. Apparently you don't.  Each of what you call
and End point should be a micro app. A single purpose App that handles one
type of request.  This lets you build my Unicorns. Fast little apps that do
one thing, and do it fast. Also because they do only one thing they get to
maximize their instance memory.

Java gets REALLY slow and has more warm ups when you have 200 megs of code
and 128 megs of ram. (as in it doesn't work, and if by some miracle it does
make it past warm up typically you hit soft memory limit and it dies)

You clearly are building Monolithic code in a world of Micro-instances. STOP
IT. You will only have pain doing so. Build single purpose instances that do
what they need to do. I'm sorry your app is too small for there to always be
one instance doing each task.  That's not my fault. Get more popular or
something. In the mean time use lazy loads so that you can fake having
instances that do one and only one thing by only loading enough stuff to do
one thing until you need to do that other thing.

I'm clearly not going to convince you to do anything, because you don't want
to.  You want somebody else to fix your problem so you won't even look at
solutions.
At which point I have to point out you are always going to be sad and
miserable.

PS
Survivor Finale. And Comments aren't static. Nor are "post to facebook". Nor
are Oauth calls.






-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Sat, Jul 21, 2012 at 11:32 PM, Drake  wrote:
>>  * How do you persist data?  (low-level, jdo, objectify, etc)
> Actually I use a multi-approach based on the work being done.  Python has
> NDB, CachePy, and a number of things that Java seems to missing good analogs
> for, but using the lowlevel API gets things up and running and does the
> majority of the heavy lifting. When we don't want to have to manage a bunch
> of code to handle stuff that is more complex or do things that are less time
> sensitive like deep queries for single users, we use JPA. (I think JDO is
> mostly dead.)

This thread is about Java instance startup time.  Enough about Python
already.  It's irrelevant.

You've completely - and lamely - dodged ALL of my questions.  You
insinuate that you have sophisticated Java applications that start up
quickly.  I'm calling bullshit.  Give me ACTUAL numbers for one ACTUAL
Java application.

>>  * How many entity kinds/classes do you have?
> A shit ton. For the analytics app we built we found out that you are limited
> to 65535 and that Dynamic is not as dynamic as you might think.

Generated kinds or classes don't count.  I mean this literally, how
many user-defined entity types does your Java app load?  You're going
to tell me you have Java apps that load 64k JPA classes?  Not a
chance.

See, this is getting to the ACTUAL problem with the Java environment -
classloading is slow.  Entity classes must be loaded and introspected
before datastore operations can start.  Near as I can tell, the only
way around this particular problem is to stay with the low-level api -
which, like assembly programming, severely limits how sophisticated an
application you can reasonably build.

>>  * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
> How quaint.  Did you miss that I said one type of task per app/backent?

I will be generous and assume for the moment that this makes sense for
your particular application.  At best you are arguing that you have a
wacky application.  You won't find too many people building business
apps that way, especially when you have elaborate transactional logic.
 There are sometimes good reasons to compose an app out of more
fundamental services, but this should be driven by application design
- not a dumb limitation of how many classes you can pack into a JAR
before instance startup times get out of control.

> Web.XML at the individual app, but path routing is handled by the

If I went the web.xml route, it would map 326 servlets.  Not really
fun.  There's a reason people started inventing "web frameworks" in
the late 90s... because that kind of programming sucks when your app
is even mildly complicated.

>>  * How many URL endpoints do you have?
> Per app? Or In total? In many cases we have endpoints per domain in
> multi-tenant apps, and we are running 10k plus domains.

I was really asking about your "unicorn" java app that loads in 2
seconds.  But I think you misunderstand what an 'endpoint' is.  It's a
particular code path; for example, http://example.com/thing/123,
http://example.com/thing/456, and http://other.com/thing/789 are all
the same endpoint because they all execute the same code.

It's relevant because it is related to how many classes get loaded at
startup, which has a strong effect on the startup latency.

>>  * How many total java classes in your application?  What is total size?
>>  * How many jars in your WEB-INF/lib?  What is total size?
> Per app? Maybe 20. In an application deployment couple thousand.   Last time
> I looked our code base was nearly 800 megs, with 80% of apps at less than 10
> megs

I want those numbers for your "unicorn" sophisticated Java app that
you claim starts in 2s.

> Jeff, my "toys" are bigger than your "projects".   My unique indexed pages
> in Google is over 1 Billion. I likely spend more in Datastore file costs
> than you spend hosting your entire client base.

Yawn.  The Royal Wedding site served more traffic and probably cost
more than either of us will make in revenue all year, but it's 99.9%
static content.  Your application is highly atypical of business
applications, and yet you myopically believe that we should all build
our apps out of zillions tiny components that use the low-level
persistence API and communicate through urlfetches and task queues.
That's retarded, and when you eventually discover the "transaction"
feature of the datastore you will understand why.

Jeff
P.S. One consequence of my particular career history is that I'm not
impressed by people trying to prove their dick is bigger.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

And this would be why we don't use contractors.  Optimizations aren't for
"Google" they are good practice.  When you run on 600 mhz with 128 Megs of
ram, you see the optimizations when you have 3.2 Ghz * 8 and 64 gigs of ram,
you can't hardly profile an app. That doesn't mean you can't make it twice
as efficient.

 

The tools you know and love suck. You just don't know it because you are
spoiled by big computers.  I love my '50 Merc and my '66 Stang, but my
MiniCooper has a lower cost of operation and has a faster 0-60 and top speed
than either of the old Fords do.  Because it is optimized. 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Mauricio Aristizabal
Sent: Sunday, July 22, 2012 12:28 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

I really don't care who has the shiniest toys in this playground, bottom
line is that I consider every minute spent getting my app to load faster a
complete waste of my time.  Adding business logic, minimizing user request
times and costs, that's what I care about, not jumping through hoops to get
around some design flaw that Google could easily fix if they chose to.

 

If you have a zillion apps and a huge budget and/or team, and it is cost
effective to do a crapload of optimizations, good for you! but I suspect
most of us just want to get our apps to work reasonably efficiently, in as
little time, and with as many of the tools we already know and love as
possible, so we can move on to our next project, client, idea, etc.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/gmXX3iD35ccJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Mauricio Aristizabal]

I really don't care who has the shiniest toys in this playground, bottom 
line is that I consider every minute spent getting my app to load faster a 
complete waste of my time.  Adding business logic, minimizing user request 
times and costs, that's what I care about, not jumping through hoops to get 
around some design flaw that Google could easily fix if they chose to.

If you have a zillion apps and a huge budget and/or team, and it is cost 
effective to do a crapload of optimizations, good for you! but I suspect 
most of us just want to get our apps to work reasonably efficiently, in as 
little time, and with as many of the tools we already know and love as 
possible, so we can move on to our next project, client, idea, etc.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/gmXX3iD35ccJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I didn't need task queues/defer because I was self managing queues and in
fact do that more than using task Queue still because it lets me do more
process shaping. So now short term tasks are defer, and long term tasks go
in to a self managed bucket that fires based on current load as determined
by the number of instances active.

>  * How do you persist data?  (low-level, jdo, objectify, etc)
Actually I use a multi-approach based on the work being done.  Python has
NDB, CachePy, and a number of things that Java seems to missing good analogs
for, but using the lowlevel API gets things up and running and does the
majority of the heavy lifting. When we don't want to have to manage a bunch
of code to handle stuff that is more complex or do things that are less time
sensitive like deep queries for single users, we use JPA. (I think JDO is
mostly dead.)

>  * How many entity kinds/classes do you have?
A shit ton. For the analytics app we built we found out that you are limited
to 65535 and that Dynamic is not as dynamic as you might think.

>  * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
How quaint.  Did you miss that I said one type of task per app/backent?
Web.XML at the individual app, but path routing is handled by the
caching/loadbalancing that is done using the python version of CDNInabox for
most our apps, and Squid on another service for some of the other Apps.  No
"App" ever is exposed directly to a user request, they are all proxied. (ps
this also allows us to do much of our own QoS manipulation because we can
send a request that is taking too long to an F8 that has multiple apps
installed on it and can be anything it needs at a moments notice.

>  * How many URL endpoints do you have?
Per app? Or In total? In many cases we have endpoints per domain in
multi-tenant apps, and we are running 10k plus domains.

>  * How many total java classes in your application?  What is total size?
>  * How many jars in your WEB-INF/lib?  What is total size?
Per app? Maybe 20. In an application deployment couple thousand.   Last time
I looked our code base was nearly 800 megs, with 80% of apps at less than 10
megs


Jeff, my "toys" are bigger than your "projects".   My unique indexed pages
in Google is over 1 Billion. I likely spend more in Datastore file costs
than you spend hosting your entire client base.



> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Saturday, July 21, 2012 9:27 PM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> Brandon, you talk a lot of shit for a guy who only discovered the task
queue 6
> months ago.
>
> Everyone who thinks they have the secret to starting up a Java app in <5s,
> answer these questions and prove that you're running more than a
> toy:
>
>  * How do you persist data?  (low-level, jdo, objectify, etc)
>  * How many entity kinds/classes do you have?
>  * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
>  * How many URL endpoints do you have?
>  * How many total java classes in your application?  What is total size?
>  * How many jars in your WEB-INF/lib?  What is total size?
>
> Having done a fair bit of experimentation around this issue, I know that
> you're talking out of your asses.  Clever suggestions like "rewrite app in
> Python" aren't nearly as clever as you think they are.
>  Suggestions that you can optimize startup time by some combination of
> caching, task queues, or threading merely indicate total lack of
understanding
> of the problem in the first place.
>
> To the people paying attention to this thread who are actually trying to
build
> real world apps on Appengine:  Brandon does not speak for the GAE team.
> Neither do I, but I occasionally talk to people that do.  I know that
Google
> really does want Java apps to run well on GAE.  They know about this issue
> and care about it - maybe even enough to do something about it.  It *is*
> constructive to "whine" politely about startup time because there are many
> issues competing for the team's time, and our feedback helps them
prioritize
> their efforts.
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Prashant Hegde]

> This isn't a counterexample, for the reason that you mentioned in the
> first sentence: you can serve everything off one instance. The
> original poster needs multiple instances, and to be able to scale as
> load changes. If you're not loading new instances, then startup time
> is pretty much irrelevant; you can start an instance once, then your
> users will never see a cold instance. This entire discussion is about
> cold instances.
>

To clarify,  we need more than one instance during peak hours, but our
overall billing is limited to 1 instance because night time traffic is
literally zero. We have seen > 2 instances without user facing loading
request during peak hours. Our analysis was that when pending latency comes
into picture, the behavior is more predictable. While the pending queue is
getting filled up, new instance is started without user facing request as
we have specified 7 second as the pending latency. By the time previous
request is served, pending request goes to the hot instance. A lot of
depends on your traffic pattern and time taken to handle the request
though.

Atleast for us the overall goal was - how do we achieve < 500 ms latency
for our users for the traffic, with auto scaling, without repeated loading
requests, without escalation in costs. We seemed to have achieved that,
hence the post and as a counter example to our friend  who decided to go
off appengine to AWS.


Prashant


> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Sat, Jul 21, 2012 at 9:24 PM, hyperflame  wrote:
>
> I bet that the major reason is just network I/O, for the GAE servers
> to find an available server, transfer a copy of the application
> +libraries to that server, and start up the servlet runner. It would
> explain why even simple apps have long startup times: it's not that
> the frameworks and libraries are taking a long time, it's just the
> overhead of moving all those libraries into place initially.

The problem specifically seems to be that classloading is slow.  A
significant part of that is explained by slow network I/O, but not
all; for example, getting AOP proxies built and loaded seems to take
significantly longer than one would expect by profiling locally.  I
speculate that the extra security measures of GAE's sandbox are also
taking a toll.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

Brandon, you talk a lot of shit for a guy who only discovered the task
queue 6 months ago.

Everyone who thinks they have the secret to starting up a Java app in
<5s, answer these questions and prove that you're running more than a
toy:

 * How do you persist data?  (low-level, jdo, objectify, etc)
 * How many entity kinds/classes do you have?
 * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
 * How many URL endpoints do you have?
 * How many total java classes in your application?  What is total size?
 * How many jars in your WEB-INF/lib?  What is total size?

Having done a fair bit of experimentation around this issue, I know
that you're talking out of your asses.  Clever suggestions like
"rewrite app in Python" aren't nearly as clever as you think they are.
 Suggestions that you can optimize startup time by some combination of
caching, task queues, or threading merely indicate total lack of
understanding of the problem in the first place.

To the people paying attention to this thread who are actually trying
to build real world apps on Appengine:  Brandon does not speak for the
GAE team.  Neither do I, but I occasionally talk to people that do.  I
know that Google really does want Java apps to run well on GAE.  They
know about this issue and care about it - maybe even enough to do
something about it.  It *is* constructive to "whine" politely about
startup time because there are many issues competing for the team's
time, and our feedback helps them prioritize their efforts.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Prashant Hegde]

Here is a counter example. We are a small app with peak traffic of 1 request 
per second. We use java. No frameworks. We use jdo, guice. Startup time 20 
seconds. Average request latency under 500 ms. We have been on appengine close 
to 2 years. Right now we are able to serve our users with one instance without 
exposing them to loading requests ( almost zero ) We use min = auto , max = 2 
and pending latency = 7.5 s. we arrived at this configuration after some 
experimentation and it gives the best performance and cost so far... At least 
we are fine now, unless google decides to change something.

We do have our share of problems with gae ( memcache errors, sudden increase in 
latency for brief periods..) But that's negligible relative to the advantages 
of the platform and cost.

Learnings 

Avoid warmups/loading requests at all costs. Our goal was never let gae start 
extra instances, as that can lead to instability. We cannot reduce startup 
time, and gae cannot get rid of all the issues associated with startup. Min = 1 
max = 1 gave us the worst performance as it creates a constraint that spins lot 
of instances esp if you have a bursts of traffic. Min = auto allows us to use 
pending latency config and it gives better control on the scheduler behaviour. 

Disclaimer

Probably this worked for us because  we have a steady request pattern during 
the day, and we don't normally see zero traffic for more than a few mins. if 
you are an app with different behaviour ( long idle times ) then min = auto may 
expose the user to loading requests..in this case min = 1 max = 2 or more 
should work better.. Again our $.02 cents...your mileage may vary


Hope this helps.


Prashant




Sent from my iPad

On 22-Jul-2012, at 3:20 AM, Tomas  wrote:

> Hi guys,
> 
> I actually didn't want to reply to this thread originally even I was the one 
> who opened the small thread 3 months ago regarding the very slow startups on 
> gae with spring + velocity + objectify but after reading the email telling me 
> I should "write modular apps and don't use frameworks" I have to write up 
> something.
> 
> I'm 10+ years java developer and I'm big fan of Spring as it saves a hundred 
> hours of time - I originally developed the www.librarist.com 2 years ago as a 
> pet project on app engine as a just a prototype using simple JSPs + Servlets 
> and it works working fine (startups 3-5 secs).
> 
> But when I wanted to add another features like cart, OAuth login, use 
> templating, use proper caching - and I decided to go with Spring as this the 
> reason why we have frameworks. Plugging in Velocity is just matter of 1 line 
> of code in XML (plus some tweaks if you need them), the cache is again couple 
> of lines in XML and you automatically get in memory + memcache with ehcache, 
> authentication is 2 lines of code. And I can continue.
> 
> I use spring in work for other projects which runs on full java stack (but 
> thats just Tomcat/Jetty) and when I run the "new" librarist equivalent code 
> on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be 
> wrong here a little bit), the startup is 2-3 secs and I can serve all my 
> traffic with it with load at max 0.5.
> 
> When I run the similar code on GAE, I get 45+ secs startup (when I'm lucky, 
> sometime it just go over 1 nib and got killed) and then GAE keep spinning 
> another instances like crazy but very often use them for three request and 
> then dying to get started again. Worth to mention - I do not  use the full 
> Spring stac, basically its just Spring core + Spring MVC, then objectify + 
> Velocity - and I really tried to do all the possible optimizations (packing 
> all the stuff into one big jar, used annotations + java confing, used XML, 
> disable scanning of annotations) but these changes just save me like 4-5 secs 
> from 45 secs.
> 
> When I tried just Spring code + Spring MVC without any other library (and 
> here we talk about ~2MB of jars), use one JSP, my best startup time is 15 
> secs.
> 
> Okay thats cool, I can understand the  GAE is different and basically it 
> could be used just for small projects where you write everything by yourself 
> and don't use any normal stuff like Dependency Injection / Proxying and 
> similar. But my biggest question for GAE guys would be this - why the hell 
> the startup takes so long simply I don't get what is actually the issue here:
> 
> - is it loading of the jars from the network (virtual drive) when the gae 
> loads each class as separate file
> - is it the scanning of the classes (annotations etc)
> - is it java reflections
> - is it creating of proxies (which Spring does for almost everything)
> - anything else I can't see
> 
> I really REALLY liked the idea of the app engine since I hate to mess up with 
> server configuration and it's great to focus on SW alone. But honestly 
> speaking, I gave up on GAE for serious projects - it's good platform for 
> doing some quick prototype to show to cl

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

If it makes you feel better I have this fight once a month. AWS that is
easier to build for or GAE which doesn't require any sys admin and is
infinitely elastic.

 

Someone will put something on AWS show me how awesome it is, and how cheap.
I will throw the load simulator at it, and watch it burn.  

They say but you didn't give it more instances.  

 

I will say fine every 4th  8 minutes when CNN runs a commercial for the site
you need to handle  5000 uesrs, and the rest of the time you need to handle
8. 

Which is cheaper?

 

They always lose.

 

How do we manage keeping the versioning the same on all instances in AWS?
How long is the down time on deploy of new version?  (GAE you can deploy,
warm up enough instances and change the "Default" version and not even lose
sessions)

 

Every service has plusses and minuses if they didn't then someone would be a
monopoly.

 

Edge Cache is amazing for most apps it will negate the AWS "savings". Rapid
Elasticity is amazing it will negate even RackSpace savings.

 

Not taking off the shelf code. That will cost you a developer. Sorry. I
would also say most the time, you should be rewriting that code anyway to be
exactly what you need, but I get that sometimes what you want is to use
Facebook Authorization, and there is no reason to strip stuff out.  Except
that you have to remember everyone who is not you writes sucky code. Generic
code. Meant to run everywhere. Except GAE is not everywhere, it is in
unicorn land where you traded having nice friendly donkey's who will put up
with what ever shit you feed them and still pull your wagon, for a flying
unicorn that shoots rainbows out of its ass but requires that you don't just
give it a sugar cube every now and then, but that it is raw organic sugar
fed to it by beautiful naked virgins.  In exchange you can go farther,
faster, and you don't have to clean up poo off the floor.  These are the
trades you make for living in Unicorn land.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Tomas
Sent: Saturday, July 21, 2012 2:50 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: Startup time exceeded...on F4?!

 

Hi guys,

I actually didn't want to reply to this thread originally even I was the one
who opened the small thread 3 months ago regarding the very slow startups on
gae with spring + velocity + objectify but after reading the email telling
me I should "write modular apps and don't use frameworks" I have to write up
something.

I'm 10+ years java developer and I'm big fan of Spring as it saves a hundred
hours of time - I originally developed the www.librarist.com 2 years ago as
a pet project on app engine as a just a prototype using simple JSPs +
Servlets and it works working fine (startups 3-5 secs).

But when I wanted to add another features like cart, OAuth login, use
templating, use proper caching - and I decided to go with Spring as this the
reason why we have frameworks. Plugging in Velocity is just matter of 1 line
of code in XML (plus some tweaks if you need them), the cache is again
couple of lines in XML and you automatically get in memory + memcache with
ehcache, authentication is 2 lines of code. And I can continue.

I use spring in work for other projects which runs on full java stack (but
thats just Tomcat/Jetty) and when I run the "new" librarist equivalent code
on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be
wrong here a little bit), the startup is 2-3 secs and I can serve all my
traffic with it with load at max 0.5.

When I run the similar code on GAE, I get 45+ secs startup (when I'm lucky,
sometime it just go over 1 nib and got killed) and then GAE keep spinning
another instances like crazy but very often use them for three request and
then dying to get started again. Worth to mention - I do not use the full
Spring stac, basically its just Spring core + Spring MVC, then objectify +
Velocity - and I really tried to do all the possible optimizations (packing
all the stuff into one big jar, used annotations + java confing, used XML,
disable scanning of annotations) but these changes just save me like 4-5
secs from 45 secs.

When I tried just Spring code + Spring MVC without any other library (and
here we talk about ~2MB of jars), use one JSP, my best startup time is 15
secs.

Okay thats cool, I can understand the  GAE is different and basically it
could be used just for small projects where you write everything by yourself
and don't use any normal stuff like Dependency Injection / Proxying and
similar. But my biggest question for GAE guys would be this - why the hell
the startup takes so long simply I don't get what is actually the issue
here:

- is it loading of the jars from the network (virtual drive) when the gae
loads each class as separate file
- is it the scanning of the classes (annotations etc)
- is it java reflections
- is it creating of proxies (which Spring does for almost everything)

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Hugo Visser]

Yeah but where this whole discussion started was that when there's no 
available instance you can't defer anything, since a cold instance will 
apparently get the request and due to the change dynamic instances aren't 
warmed up like they used to be. You're bound to need to do 
some initialization at that point. And while you could build every single 
component you're app needs yourself to let it be super duper optimized for 
App Engine, that will just not justify the amount of work it will take to 
run a app on App Engine.

IMHO the warm up requests serve their purpose; they warm up the app so that 
a slightly longer start up time of 10 secs or so isn't that bad. The issue 
is just that there are now situations where user facing requests end up 
warming up a instance which sounds like a bad thing, especially if the app 
is receiving spiky loads.


On Saturday, July 21, 2012 11:00:53 PM UTC+2, Brandon Wirtz wrote:
>
> When I said 1.5 I meant PMF was adding 3.5 you don’t need. PMF takes 5s 
> the API takes under 1s.  I think 2.5s is probably the lowest you can get 
> and do something truly useful, but it seems that I can do a LOT in 2.5s  
> (assuming the F4 this thread was about when we started) (well actually 
> assuming F2, F4’s seem to need 3s no matter what you do)
>
> But also you don’t have to do anything useful on warmup. So you shouldn’t 
> serve a ColdStart very often.
>
>  
>
> I have mentioned before I work to keep everything to under 4s. If my App 
> Gets to 8s I start slicing it up 
>
>  
>
> I am also 100% confident that if all you do is say “I am up” your warm up 
> will take under 1.5s  for 95% of starts. And that should really do “I am 
> up” and defer lazyloads of everything you need to be ready to do real work.
>
>  
>
>  
>
> *From:* google-appengine@googlegroups.com [mailto:
> google-appengine@googlegroups.com] *On Behalf Of *hyperflame
> *Sent:* Saturday, July 21, 2012 1:47 PM
> *To:* google-appengine@googlegroups.com
> *Subject:* Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
>  
>
> I just looked over the logs of a corporate GAE application. It's a very 
> simple "heartbeat" application; essentially our production applications 
> have to send a message to it periodically. The message sent is about 500 
> characters; the application does some checksum-ing and sends back a JSON 
> string about 100 characters long. The only external libraries are jars of 
> twitter4j and the org.json parser. It's really not much more complex than a 
> Hello World app.
>
> Looking over the logs, the fastest start time from cold-instance to 
> first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5 
> seconds to cold-start) on a F1 instance. Once the instance is warm, 
> subsequent requests take, on average, ~150ms to process and send back a 
> reply. I could probably get that lower if I upgraded to F4.
>
> On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote:
>
> I use getRessourceAsStream an Stream-copy - but you miss the point...it's 
> a Hello World, done in 5 minutes as test for your post, stripping away all 
> excuses.
> The calls after startup are answered in <80 ms.
> That mans that I'm still far above 3 seconds for initializing of an empty 
> project. I could simply write Hello World into the Stream.
>
> I could use static...yes...but thats not the point. Some people have 
> dynamic and user dependand data on each page.
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/yM2ywgzd4PIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

When I said 1.5 I meant PMF was adding 3.5 you don’t need. PMF takes 5s the
API takes under 1s.  I think 2.5s is probably the lowest you can get and do
something truly useful, but it seems that I can do a LOT in 2.5s  (assuming
the F4 this thread was about when we started) (well actually assuming F2,
F4’s seem to need 3s no matter what you do)

But also you don’t have to do anything useful on warmup. So you shouldn’t
serve a ColdStart very often.

 

I have mentioned before I work to keep everything to under 4s. If my App
Gets to 8s I start slicing it up 

 

I am also 100% confident that if all you do is say “I am up” your warm up
will take under 1.5s  for 95% of starts. And that should really do “I am up”
and defer lazyloads of everything you need to be ready to do real work.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of hyperflame
Sent: Saturday, July 21, 2012 1:47 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

I just looked over the logs of a corporate GAE application. It's a very
simple "heartbeat" application; essentially our production applications have
to send a message to it periodically. The message sent is about 500
characters; the application does some checksum-ing and sends back a JSON
string about 100 characters long. The only external libraries are jars of
twitter4j and the org.json parser. It's really not much more complex than a
Hello World app.

Looking over the logs, the fastest start time from cold-instance to
first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5
seconds to cold-start) on a F1 instance. Once the instance is warm,
subsequent requests take, on average, ~150ms to process and send back a
reply. I could probably get that lower if I upgraded to F4.

On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote:

I use getRessourceAsStream an Stream-copy - but you miss the point...it's a
Hello World, done in 5 minutes as test for your post, stripping away all
excuses.
The calls after startup are answered in <80 ms.
That mans that I'm still far above 3 seconds for initializing of an empty
project. I could simply write Hello World into the Stream.

I could use static...yes...but thats not the point. Some people have dynamic
and user dependand data on each page.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

May be it's my language barrier...don't know. Whats hard to understand in
NewProject->HelloWorld-Servlet->3 seconds startup time->1,5 s isn't true 
and cannot be true
If I take 500 ms away for the request because i'm so terribad it's still >3 
s startup time, for HelloWorld. Even if it's 2.5 s in a good timeslice, 
it's far off 1.5 and it has to load 1 (!) servlet without depencies 
(besides Java-Servlet-Stuff).
I'm done here - you win the internet.

Am Samstag, 21. Juli 2012 22:39:51 UTC+2 schrieb Brandon Wirtz:
>
> No the point is a major limitation of Appengine is that File IO is SLOW so 
> you never do “open file” 
>
> You want to do “dynamic” pull that from memory or datastore.
>
>  
>
> NEVER EVER LOAD DATA FROM FILE OTHER THAN TO POPULATE THE DATASTORE FOR 
> THE FIRST TIME IF YOU CARE ABOUT WARMUP
>
> Yes I have an exception where I do that with a 40 MB pickled file that it 
> is faster than doing 40 reads from the datastore and doing a join because 
> of the memory usage, but that is a weird case.
>
>  
>
> But yes. You are doing it wrong. And if you would profile your apps you 
> would know you are doing it wrong.  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/2NbtQVmyFOYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[hyperflame]

I just looked over the logs of a corporate GAE application. It's a very 
simple "heartbeat" application; essentially our production applications 
have to send a message to it periodically. The message sent is about 500 
characters; the application does some checksum-ing and sends back a JSON 
string about 100 characters long. The only external libraries are jars of 
twitter4j and the org.json parser. It's really not much more complex than a 
Hello World app.

Looking over the logs, the fastest start time from cold-instance to 
first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5 
seconds to cold-start) on a F1 instance. Once the instance is warm, 
subsequent requests take, on average, ~150ms to process and send back a 
reply. I could probably get that lower if I upgraded to F4.

On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote:
>
> I use getRessourceAsStream an Stream-copy - but you miss the point...it's 
> a Hello World, done in 5 minutes as test for your post, stripping away all 
> excuses.
> The calls after startup are answered in <80 ms.
> That mans that I'm still far above 3 seconds for initializing of an empty 
> project. I could simply write Hello World into the Stream.
>
> I could use static...yes...but thats not the point. Some people have 
> dynamic and user dependand data on each page.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

No the point is a major limitation of Appengine is that File IO is SLOW so
you never do “open file” 

You want to do “dynamic” pull that from memory or datastore.

 

NEVER EVER LOAD DATA FROM FILE OTHER THAN TO POPULATE THE DATASTORE FOR THE
FIRST TIME IF YOU CARE ABOUT WARMUP

Yes I have an exception where I do that with a 40 MB pickled file that it is
faster than doing 40 reads from the datastore and doing a join because of
the memory usage, but that is a weird case.

 

But yes. You are doing it wrong. And if you would profile your apps you
would know you are doing it wrong.  

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 1:19 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

I use getRessourceAsStream an Stream-copy - but you miss the point...it's a
Hello World, done in 5 minutes as test for your post, stripping away all
excuses.
The calls after startup are answered in <80 ms.
That mans that I'm still far above 3 seconds for initializing of an empty
project. I could simply write Hello World into the Stream.

I could use static...yes...but thats not the point. Some people have dynamic
and user dependand data on each page.


Am Samstag, 21. Juli 2012 22:09:11 UTC+2 schrieb Brandon Wirtz:

Shit, even putting it in your static and Fetching by URL is faster than File
reader. 

 

 

 

Hi,

I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a
local ressource and writes it to the output stream, thats all.
Hello World.
Even the empty container does need some class loading - thats the Java
world. And currently people have startup timing problems.

1.5 Seconds are possible if all your stuff is static and in Edge cache,
nothing really starts up there.

Best regards,
André


Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 

Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/cTL6IofY1ygJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

I use getRessourceAsStream an Stream-copy - but you miss the point...it's a 
Hello World, done in 5 minutes as test for your post, stripping away all 
excuses.
The calls after startup are answered in <80 ms.
That mans that I'm still far above 3 seconds for initializing of an empty 
project. I could simply write Hello World into the Stream.

I could use static...yes...but thats not the point. Some people have 
dynamic and user dependand data on each page.


Am Samstag, 21. Juli 2012 22:09:11 UTC+2 schrieb Brandon Wirtz:
>
> Shit, even putting it in your static and Fetching by URL is faster than 
> File reader. 
>
>  
>
>  
>
> Hi,
>
> I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a 
> local ressource and writes it to the output stream, thats all.
> Hello World.
> Even the empty container does need some class loading - thats the Java 
> world. And currently people have startup timing problems.
>
> 1.5 Seconds are possible if all your stuff is static and in Edge cache, 
> nothing really starts up there.
>
> Best regards,
> André
>
>
> Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:
>
> The intent of that line was that you would save 3.5s dropping factory for 
> the GAE API
>
>  
>
> But one of my main apps is 2.2s average spinup time, I am pretty sure I 
> can get Hello world in under 1.5.  
>
>  
>
> Are you lazy loading? Are your lazy loads lazy loading? Have you threaded 
> your Lazy Loads?
>
> Class 1 has dependencies ABC,  Class 2 has depencies DEF, 
>
> Warm up does basically nothing but defer Class1 and Class2
>
>  
>
> If Cold Start only do Called Class,  If first call to instance defer other 
> classes for warm up.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/cTL6IofY1ygJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Shit, even putting it in your static and Fetching by URL is faster than File
reader. 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 1:00 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Hi,

I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a
local ressource and writes it to the output stream, thats all.
Hello World.
Even the empty container does need some class loading - thats the Java
world. And currently people have startup timing problems.

1.5 Seconds are possible if all your stuff is static and in Edge cache,
nothing really starts up there.

Best regards,
André


Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 

Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Read a local resource? You mean you do FileReader? 

 

Don’t do that. That’s bad. Put that in Memcache and read it from memcache.
Keep it in memcache at all times.

 

Never use FileReader.  Ever. Unless it is to get an app booted for the very,
very first time. It should be in the datastore, and in memcache.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 1:00 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Hi,

I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a
local ressource and writes it to the output stream, thats all.
Hello World.
Even the empty container does need some class loading - thats the Java
world. And currently people have startup timing problems.

1.5 Seconds are possible if all your stuff is static and in Edge cache,
nothing really starts up there.

Best regards,
André


Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 

Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

Hi,

I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a 
local ressource and writes it to the output stream, thats all.
Hello World.
Even the empty container does need some class loading - thats the Java 
world. And currently people have startup timing problems.

1.5 Seconds are possible if all your stuff is static and in Edge cache, 
nothing really starts up there.

Best regards,
André


Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:
>
> The intent of that line was that you would save 3.5s dropping factory for 
> the GAE API
>
>  
>
> But one of my main apps is 2.2s average spinup time, I am pretty sure I 
> can get Hello world in under 1.5.  
>
>  
>
> Are you lazy loading? Are your lazy loads lazy loading? Have you threaded 
> your Lazy Loads?
>
> Class 1 has dependencies ABC,  Class 2 has depencies DEF, 
>
> Warm up does basically nothing but defer Class1 and Class2
>
>  
>
> If Cold Start only do Called Class,  If first call to instance defer other 
> classes for warm up.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 



Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 12:27 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Brandon, your 1.5 seconds - I call this nonsense in Java world, thats all.
Create empty Java Project with 1 Servlet, disable JDO/Datanucleus stuff, no
Framework, nothing.


2012-07-21 12:14:33.275 / 200 3525ms 1kb Mozilla/5.0 (Windows NT 6.0;
rv:14.0) Gecko/20100101 Firefox/14.0.1 



May be there was some time where your mantioned link could reach 2.5 seconds
- but it's not regular.
Do I have problems with 5 seconds startup time? No - it's considered very
fast in the Java world. I don't use any Frameworks, have learned that early.
Just saying...1.5 seconds. And your regular posts that all is fine and the
people are just doing it wrong are at least as annoying as mine.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/tv73lRqnY1MJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

Brandon, your 1.5 seconds - I call this nonsense in Java world, thats all. 
Create empty Java Project with 1 Servlet, disable JDO/Datanucleus stuff, no 
Framework, nothing.
2012-07-21 12:14:33.275 / 200 3525ms 1kb Mozilla/5.0 (Windows NT 6.0; 
rv:14.0) Gecko/20100101 Firefox/14.0.1 
May be there was some time where your mantioned link could reach 2.5 
seconds - but it's not regular.
Do I have problems with 5 seconds startup time? No - it's considered very 
fast in the Java world. I don't use any Frameworks, have learned that early.
Just saying...1.5 seconds. And your regular posts that all is fine and the 
people are just doing it wrong are at least as annoying as mine.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/tv73lRqnY1MJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Francois Masurel]

What annoys me the most is that it's nowhere explicitly stated in the docs 
that only heavily optimized low-level applications are "allowed" to run on 
AppEngine.

Google should clearly specify that most well known Java frameworks 
shouldn't be used on AppEngine.

In my case I have tried to optimize my code as much as possible (no spring, 
only low-level datastore access, etc.) but I still need to load a bunch of 
libraries as my application is heavily connected to many external web 
services of all kinds (soap, rest, json, xml, etc.).

I can't imagine to rewrite all these vendor specific librairies even if 
this include some kind of duplicate code.  And for some of them, source 
code is not available.

At the end my application takes between 10 to 15s to load.

I really hope things will improve over time.

So Jeff, you are not alone, I guess we are quite a few AppEngine users 
annoyed by these user facing loading requests.

In fact, the problem exists since the beginning.  We have seen some 
improvements over time, but things seems to go backward lately.  I just 
hope it's temporary.

Francois



On Saturday, July 21, 2012 7:19:52 PM UTC+2, Brandon Wirtz wrote:
>
> Ah, yes, and we come back to Brandon must only run one kind of app, cause 
> I 
> mean who runs more than one App? He can't know Java AND Python. 
>
> I'm apparently not the only person living in Unicorn land, Dave Chandler 
> also lives there 
>
> http://turbomanage.wordpress.com/2010/03/26/appengine-cold-starts-considered 
> / So does Meetu Maltiar 
> http://thoughts.inphina.com/2010/10/09/managing-cold-start-on-google-app-eng 
>
> ine/<http://thoughts.inphina.com/2010/10/09/managing-cold-start-on-google-app-engine/>
>  
>
> Just because you are too lazy to optimize your code, and want to blame the 
> platform don't take your anger out on me. 
>
> Looking at the archives you have been singing the same song for 2 years. 
> Not once have a I seen you say, "Yeah my imports are now being lazy 
> loaded" 
> or " I serialized my configuration initialization variables" and "It saved 
> me 1s but I need to find 5 more where should I look" 
>
> All I see from you and Andre is Wha-Wha, Wha-Wha, Wha-Wha-Wha. 
>
> If Python is magic move to it. I run about 70% py 30% java because unlike 
> you I profiled my apps and used which ever was better. And in the cases 
> where Go Is Better. I'm Learning and  building code in Go now too. 
>
> So grow up. Quit whining read the docs on the internet and when you can 
> post 
> "I did all those things and my app still doesn't load fast enough" and 
> when 
> you tell me you modularized your code so that you can leverage backend for 
> steady state and large code base latency tolerant apps, come back to me. 
> I'll help you with a code review just to get you to stop complaining. 
>
> It's ok, not every programmer and learn to think in the Google code 
> construction methodology, or think in "cloud". But there are clearly 
> highly 
> successful projects that have gotten around your issues. So stop saying 
> those people live in a magical land not available to everyone else. 
>  Google 
> doesn't look at my Email associated with the app and give me a performance 
> bonus. (I Know I checked) 
>
>
>
>
>
>
>
>
>
>
> > -----Original Message- 
> > From: google-appengine@googlegroups.com [mailto:google- 
> > appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer 
> > Sent: Saturday, July 21, 2012 1:19 AM 
> > To: google-appengine@googlegroups.com 
> > Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?! 
> > 
> > Brandon, your comments are irrelevant and not constructive.  The Python 
> > runtime has a completely different startup profile from the Java 
> runtime. 
> > The long startup delays in Javaland occur prior to any service calls; no 
> amount 
> > of caching, deferring, queueing, or serialization is going to help. 
> > 
> > Jeff 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/L9oaqE2apt8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Ah, yes, and we come back to Brandon must only run one kind of app, cause I
mean who runs more than one App? He can't know Java AND Python.

I'm apparently not the only person living in Unicorn land, Dave Chandler
also lives there
http://turbomanage.wordpress.com/2010/03/26/appengine-cold-starts-considered
/ So does Meetu Maltiar
http://thoughts.inphina.com/2010/10/09/managing-cold-start-on-google-app-eng
ine/

Just because you are too lazy to optimize your code, and want to blame the
platform don't take your anger out on me.

Looking at the archives you have been singing the same song for 2 years.
Not once have a I seen you say, "Yeah my imports are now being lazy loaded"
or " I serialized my configuration initialization variables" and "It saved
me 1s but I need to find 5 more where should I look"

All I see from you and Andre is Wha-Wha, Wha-Wha, Wha-Wha-Wha.

If Python is magic move to it. I run about 70% py 30% java because unlike
you I profiled my apps and used which ever was better. And in the cases
where Go Is Better. I'm Learning and  building code in Go now too.

So grow up. Quit whining read the docs on the internet and when you can post
"I did all those things and my app still doesn't load fast enough" and when
you tell me you modularized your code so that you can leverage backend for
steady state and large code base latency tolerant apps, come back to me.
I'll help you with a code review just to get you to stop complaining.

It's ok, not every programmer and learn to think in the Google code
construction methodology, or think in "cloud". But there are clearly highly
successful projects that have gotten around your issues. So stop saying
those people live in a magical land not available to everyone else.  Google
doesn't look at my Email associated with the app and give me a performance
bonus. (I Know I checked)










> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Saturday, July 21, 2012 1:19 AM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> Brandon, your comments are irrelevant and not constructive.  The Python
> runtime has a completely different startup profile from the Java runtime.
> The long startup delays in Javaland occur prior to any service calls; no
amount
> of caching, deferring, queueing, or serialization is going to help.
>
> Jeff



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Sat, Jul 21, 2012 at 1:12 AM, André Pankraz  wrote:
> I just answer,1.5 seconds startup : yep, with java not even an empty hello 
> world will manage this for us mortals.
> Unicorn land.

 * How do you persist data?  (low-level, jdo, objectify, etc)
 * How many entity kinds/classes do you have?
 * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
 * How many URL endpoints do you have?
 * How many total java classes in your application?  What is total size?
 * How many jars in your WEB-INF/lib?  What is total size?

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

Brandon, your comments are irrelevant and not constructive.  The
Python runtime has a completely different startup profile from the
Java runtime.  The long startup delays in Javaland occur prior to any
service calls; no amount of caching, deferring, queueing, or
serialization is going to help.

Jeff

On Fri, Jul 20, 2012 at 9:52 PM, Brandon Wirtz  wrote:
> Cache Incoming requests= The thing that I am always accused of. Use Edge
> Cache to make sure you don't need to serve people who are asking for the
> same thing.
>
> Fault Tolerant writes is about determining how "race" your race conditions
> are, and being smart about your writes.
>
> Common things I see, people want to increment something so they get count.
> Which is slow. Then they +1 and write. That is really slow.
>
> I use Defer for a lot of writes when I know I wont have a read for a bit of
> time.  Google Bot just did something. Well Frak Google Bot, it's not a real
> user, delay that write until we have time for it. Defer. You have to look at
> your write types and determine the priority so you can decide how important
> it is to be fast, if you 100% need the write to complete, or if you are just
> doing something hoping people will use it later.
>
> Write to a write Cache with the Marshall/pickle, if you need the data broken
> out as well, do a follow up and read the Serialized data and write the
> individual bits when you have more time.
>
> Do reads in a similar way when ever possible use serialized data.
>
>
>
> On Friday, July 20, 2012 9:39:43 PM UTC-7, Kyle Finley wrote:
>>
>> Hi Brandon,
>>
>>>
>>> The people who don't know how to build APIs so that apps are task
>>> specific. Piss me off. Build modular. Dump frame works. Defer often. Be your
>>> own scheduler by shaping internal ops. Cache incoming. Cache reads cache
>>> writes. Manage threads. Use warmups. This is not rocket science.
>>
>> OT, but how are you cacheing writes in a fault tolerant manner? Task
>> queue? I've been trying to develop a strategy for that. Also, what do you
>> mean by "Cache incoming"?
>>
>> Thanks,
>>
>> -Kyle
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/EVXSdF_xc1UJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

I just answer,1.5 seconds startup : yep, with java not even an empty hello 
world will manage this for us mortals.
Unicorn land.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/rbmZpe-pilEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

I just answer,1.5 seconds startup : yep, with java not even an empty hello 
world will manage this for us mortals.
Unicorn land.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/iT3DPOmS6toJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Kyle Finley]

Oh, OK. Thank you Brandon.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/rvs4uS2r6LkJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Brandon Wirtz]

Cache Incoming requests= The thing that I am always accused of. Use Edge 
Cache to make sure you don't need to serve people who are asking for the 
same thing.

Fault Tolerant writes is about determining how "race" your race conditions 
are, and being smart about your writes.

Common things I see, people want to increment something so they get count. 
Which is slow. Then they +1 and write. That is really slow.

I use Defer for a lot of writes when I know I wont have a read for a bit of 
time.  Google Bot just did something. Well Frak Google Bot, it's not a real 
user, delay that write until we have time for it. Defer. You have to look 
at your write types and determine the priority so you can decide how 
important it is to be fast, if you 100% need the write to complete, or if 
you are just doing something hoping people will use it later.

Write to a write Cache with the Marshall/pickle, if you need the data 
broken out as well, do a follow up and read the Serialized data and write 
the individual bits when you have more time.

Do reads in a similar way when ever possible use serialized data.



On Friday, July 20, 2012 9:39:43 PM UTC-7, Kyle Finley wrote:
>
> Hi Brandon,
>  
>
>> The people who don't know how to build APIs so that apps are task 
>> specific. Piss me off. Build modular. Dump frame works. Defer often. Be 
>> your own scheduler by shaping internal ops. Cache incoming. Cache reads 
>> cache writes. Manage threads. Use warmups. This is not rocket science.
>>
> OT, but how are you cacheing writes in a fault tolerant manner? Task 
> queue? I've been trying to develop a strategy for that. Also, what do you 
> mean by "Cache incoming"?
>
> Thanks,
>
> -Kyle
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/EVXSdF_xc1UJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Kyle Finley]

Hi Brandon,
 

> The people who don't know how to build APIs so that apps are task 
> specific. Piss me off. Build modular. Dump frame works. Defer often. Be 
> your own scheduler by shaping internal ops. Cache incoming. Cache reads 
> cache writes. Manage threads. Use warmups. This is not rocket science.
>
OT, but how are you cacheing writes in a fault tolerant manner? Task queue? 
I've been trying to develop a strategy for that. Also, what do you mean by 
"Cache incoming"?

Thanks,

-Kyle
 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/vB55pDznx94J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

All your pages should have the ability to have an optional ?debug=true

Then if you have set that in the url the first line in every one of your
functions, and before your imports,  should be "Timer start" and the last
thing should be "Log Timer"


> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Friday, July 20, 2012 11:00 AM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> On Fri, Jul 20, 2012 at 10:34 AM, hyperflame 
> wrote:
> > Discussing theoretical startup times is great, but I'd like to see
> > some real-world startups. Does anyone with high startup times ( say,
> > 30+ seconds) want to share the results of a code profiler/appstats?
>
> What code profiler works on GAE server-side?  It's not very helpful to
profile
> in the dev environment since the problem does not manifest there.
>
> Appstats is not helpful; my startup is long even with zero service layer
calls.
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

 

>If you follow the group longer you should know - Brandom lives in GAE
unicorn land and all you ever need are proper Edge cache settings. ;)




No, I just don't write code using unnecessary frameworks and I do a ton of
testing and architecture planning. 

Most everyoneelse never uses defer or threads or task queues. They don't
manage steady state tasks, or profile their imports.

It is easy to blame the platform, but if you write crap code it will run
like crap.

I don't think GAE is perfect. But most the people who complain about its
failings don't have the coding skills to be an intern at my shop. 

I blame Google for not documenting everything well. I blame google for not
telling you "don't use frameworks they suck".

But the number of people who load spring and the gripe that it is slow and
are only using 3 things out of it make me want to punch them in the head.

The people who don't know how to build APIs so that apps are task specific.
Piss me off. Build modular. Dump frame works. Defer often. Be your own
scheduler by shaping internal ops. Cache incoming. Cache reads cache writes.
Manage threads. Use warmups. This is not rocket science.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Fri, Jul 20, 2012 at 10:34 AM, hyperflame  wrote:
> Discussing theoretical startup times is great, but I'd like to see
> some real-world startups. Does anyone with high startup times ( say,
> 30+ seconds) want to share the results of a code profiler/appstats?

What code profiler works on GAE server-side?  It's not very helpful to
profile in the dev environment since the problem does not manifest
there.

Appstats is not helpful; my startup is long even with zero service layer calls.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Michael Hermus]

>
> Perfect, thanks; that is all I am asking for! 
>
> Additionally, we started an internal discussion about reviving warmup 
> requests for dynamic instances. If you want this feature, please star the 
> following issue:
> http://code.google.com/p/googleappengine/issues/detail?id=7865
>
> -- Takashi
>
>>
>> -- 
> Takashi Matsuo
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/jNAp6gCZ_4QJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[André Pankraz]

If you follow the group longer you should know - Brandom lives in GAE 
unicorn land and all you ever need are proper Edge cache settings. ;)


Am Freitag, 20. Juli 2012 10:16:28 UTC+2 schrieb Simon Knott:
>
> That's just not true - I have an app which uses no third-party libraries 
> at all, uses no persistence and in fact it uses no GAE services.  It simply 
> has one servlet which processes request headers and returns a response.  My 
> average start-up time for this app is 3 seconds, when it's running well. 
>  The simple fact is that as soon as you use any third-party libraries, or 
> do anything vaguely useful, you're out of that 5 second "magic" window.
>
> On Friday, 20 July 2012 05:20:56 UTC+1, Brandon Wirtz wrote:
>>
>> Dump PMF use the low level API 
>> https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi 
>>
>> ne/api/datastore/package-summary
>>  
>>
>> Either drink the Kool Aid, or quit complaining :-) 
>>
>> 5s will be 1.5 
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/lYiT-qQBw6UJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Takashi Matsuo]

Sorry, but, please forget about 5 secs 'magic' window. There is no any
hard/soft deadline/threshold like that in the current App Engine system.

It was just a one example of well behaved apps. Let me rephrase what I
meant to say.

With app engine, it is always a good practice to keep the loading request
faster. The current scheduler works much better with fast loading apps than
with slow loading apps. So I always suggest developers fastening the
loading request.

On the other hand, we understand that Java apps tend to be slow at a level
where it is not acceptable for you if there are many user-facing loading
requests. For a time being, please use min idle instances settings to
reduce the loading requests with slow loading apps.

-- Takashi
On Jul 20, 2012 5:16 PM, "Simon Knott"  wrote:

> That's just not true - I have an app which uses no third-party libraries
> at all, uses no persistence and in fact it uses no GAE services.  It simply
> has one servlet which processes request headers and returns a response.  My
> average start-up time for this app is 3 seconds, when it's running well.
>  The simple fact is that as soon as you use any third-party libraries, or
> do anything vaguely useful, you're out of that 5 second "magic" window.
>
> On Friday, 20 July 2012 05:20:56 UTC+1, Brandon Wirtz wrote:
>>
>> Dump PMF use the low level API
>> https://developers.google.com/**appengine/docs/java/javadoc/**com/google/appengi
>>
>> ne/api/datastore/package-**summary
>>
>> Either drink the Kool Aid, or quit complaining :-)
>>
>> 5s will be 1.5
>>
>>
>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/W1sBfl9Pu2IJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Simon Knott]

That's just not true - I have an app which uses no third-party libraries at 
all, uses no persistence and in fact it uses no GAE services.  It simply 
has one servlet which processes request headers and returns a response.  My 
average start-up time for this app is 3 seconds, when it's running well. 
 The simple fact is that as soon as you use any third-party libraries, or 
do anything vaguely useful, you're out of that 5 second "magic" window.

On Friday, 20 July 2012 05:20:56 UTC+1, Brandon Wirtz wrote:
>
> Dump PMF use the low level API 
> https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi 
>
> ne/api/datastore/package-summary
>  
>
> Either drink the Kool Aid, or quit complaining :-) 
>
> 5s will be 1.5 
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/W1sBfl9Pu2IJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Hugo Visser]

I did, and used Objectify (Jeff is kinda familiar with that I think :)). A 
plain Hello World with only registering the entities will get you around 
the 4-5 sec mark, which is hardly a real application. 

I have a highly optimized app as a backend for my Android app, and that 
even takes around the 6-8 seconds mark to start up and it varies too.
Add in some third party frameworks, like Jersey for JAXRS and Guice to keep 
your code sane and you're at the 12 secs.

So kool aid or not, Java start up overhead will always be more than it is 
for python and I think the 5 seconds mentioned is far from typical if you 
are running a non-trivial app.

On Friday, July 20, 2012 6:20:56 AM UTC+2, Brandon Wirtz wrote:
>
> Dump PMF use the low level API 
> https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi 
>
> ne/api/datastore/package-summary
>  
>
> Either drink the Kool Aid, or quit complaining :-) 
>
> 5s will be 1.5 
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/4icNAFZtGKwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Dump PMF use the low level API
https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi
ne/api/datastore/package-summary

Either drink the Kool Aid, or quit complaining :-)

5s will be 1.5




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Thu, Jul 19, 2012 at 3:37 PM, Takashi Matsuo  wrote:
>
> To clarify, there is no any hard limit on the cold startup time except for
> the hard deadline(60secs for online, 600secs for offline). The 5secs just
> came from Jeff's e-mail I think. That said, it is one of the most important
> best practice with App Engine to reduce the loading time.

Just to be clear, the 5s number came from your comment "...there are
also many well-behaved apps with <5 secs loading/warming time".

My reply is that if you create a "Hello, World" Java app that does
nothing but create a PersistenceManagerFactory for a single entity, it
takes 5-6 seconds to startup even when GAE is perfectly healthy.  In
practice, is impossible to create a "well-behaved" Java app on GAE.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Takashi Matsuo]

On Fri, Jul 20, 2012 at 7:10 AM, hyperflame  wrote:

> On Thursday, July 19, 2012 2:57:28 AM UTC-5, Richard Watson wrote:
>>
>>
>> That's fine, but all we're asking for is "don't dump those requests into
>> an instance that isn't able to serve the request immediately".  If there
>> are three instances running and one being started up, don't consider the
>> latter in the population of instances to send requests to. Dump it into an
>> instance level queue that is already running. Everything else remains
>> constant.
>>
>
> Let's back up a minute here. To the GAE scheduler, as it is right now,
> there is no concept of "starting up." Either there is an instance, or there
> is no instance. A "cold" instance is just as good as an instance that has
> been active for hours.
>
> If you read Mr. Matsuo of the GAE team's posting, he wrote, "If you have
> an app with average +50s loading time, I totally understand that you
> strongly want to avoid sending requests to cold instances. On the other
> hand, there are also many well-behaved apps with <5 secs loading/warming
> time. Please understand that for those apps, it is still acceptable if we
> send requests to cold instances, so it's likely we can not prioritize the
> feature bellow over other things, however..."
>
> I read that as (and correct me if I'm wrong) that the GAE scheduler
> expects cold instances to be ready in less than 5 seconds (notice the
> "well-behaved" remark). If that is so, there is no reason to categorize
> instances as being in "starting up" phase. An instance is an instance is an
> instance, regardless of whether or not it was just cold-started.
>
> Let's go back to the MegaWalmart example. MegaWalmart is guaranteeing that
> if it opens a new checkout lane, that checkout lane will be open quickly,
> that the clerk will not be drunk, the scanner will be working, etc. Is that
> guarantee reasonable? That's the point of this thread.
>
>
> On Thursday, July 19, 2012 2:57:28 AM UTC-5, Richard Watson wrote:
>> I stand to be corrected, but I doubt that Google searches are dumped onto
>> cold instances.
>>
>
> The GAE scheduler expects hosted apps to cold start in less than 5
> seconds. If the GAE scheduler expects that of us, I don't see why Google
> wouldn't hold itself to that same commitment.
>

To clarify, there is no any hard limit on the cold startup time except for
the hard deadline(60secs for online, 600secs for offline). The 5secs just
came from Jeff's e-mail I think. That said, it is one of the most important
best practice with App Engine to reduce the loading time.

At the same time, we understand that some people want to use popular Java
libraries which take longer to load. We're trying hard to mitigate the pain
by fastening the loading time, it's underway, but no ETA yet. So for a time
being, I would suggest setting sufficient number of min idle instances to
reduce the number of user-facing loading requests, especially if you think
your app is important and you strongly want to eliminate the user-facing
loading requests.

Additionally, we started an internal discussion about reviving warmup
requests for dynamic instances. If you want this feature, please star the
following issue:
http://code.google.com/p/googleappengine/issues/detail?id=7865

-- Takashi

>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/nwb8Q4Bxx7cJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Takashi Matsuo

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I can make searches that are slow. You just have to pick things no one has
searched before.

That said, the 1% of bad check outs would only apply if you only serve 1
request from the slow server. If each instance serves 100k requests in its
lifetime, then you are talking about 1/100k being negatively impacted. Even
my lowliest of instances serves 3000 in 2 hours. Before dying.

I said 15s spinup because I think that was your number.  I shoot for spin
ups that are under 4s.

If my spinup is more than 4s I create a new app, or dynamic back end to
handle the requests with specialized version of the microapp that is faster
to load.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Wed, Jul 18, 2012 at 3:55 PM, hyperflame  wrote:
>
> "Massively" being the key word there. Every Google service is massive.
> Even abandoned or low use Google services (i.e. Wave, Buzz) are going
> to use the equivalent of a minimum 10,000 F4 instances. The GAE
> scheduler does inefficient scheduling at less than 50 instances. It
> works wonderfully once you hit scale at 100+ machines(correct me if
> I'm wrong, but most of the people seeing problems here are probably
> less than 50 instances).

This is an interesting discussion.  How many apps do you think are on
GAE that run 100+ instances?  1000+?  While these are certainly going
to be the "important" apps for Google, I suspect they are fairly few
in number.  With a couple exceptions, the owners are very quiet on
this list.

I wonder also how many of them are on multithreaded runtimes, and of
those, how many of them are on Java runtimes.

>> b) By waiting for instances to warm up first, I don't think you would
>> really increase the maximum depth of the pending queue by a whole lot.
>
> I would have to disagree with you on this. My experience with big
> websites tells me that requests can very quickly pile up if you're not
> handling them expeditiously.

I think we're looking at this wrong.  The question is not "should
there be more than a single queue"; I certainly agree that we
certainly don't want one loadbalancer falling over and taking down a
thousand pending requests.

The important question is "should requests go to a queue that is
locked to a single (unstarted) instance", which is a different
question.  It's certainly possible to have multiple queues with the
ability to route requests to active instances.  At Google scale there
must certainly already be multiple queues; obviously one computer
cannot route the zillions of requests per second that www.google.com
gets.

Continuing the MegaWalmart example, you don't need to have one big
line routed around the store; you can have a number of lines each of
which route among banks of cashiers.  Presumably GAE has some sort of
equivalent construct.

"Less than 1% of your customers were inconveniced. 99% of people still
had a decent time checking out." is not very comforting.  If 1% of
Google searches took 8+ seconds to respond, the Googleplex alarms
would deafen people as far away as Oakland.

> Just to be clear, I am agreeing with you in that the GAE scheduler
> needs work; it is currently optimizing for high-scale apps, and not
> apps that are using double-digit or lower instances.

I fear that GAE is optimized for single-threaded Python2.5 apps which
take 2s to spin up, an environment that is becoming less relevant as
time goes on.  Given an app that takes 20+ seconds to start, would a
high-traffic app work any better than a low-traffic app?

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Wed, Jul 18, 2012 at 11:27 AM, Drake  wrote:
> Use warm up requests.

...except that thanks to a recent change, warmup requests don't
prevent users from seeing cold starts anymore.  In a burst of traffic,
GAE will happily route requests to cold instances even if an active
instance would have become available 1s later.

> Also optimize your code to load faster.   I have an app that is HUGE it
> loads in about 8s. On an F4. It doesn't fit in an F2.  Check that you really
> need all your libraries. Check that any of the large blocks of data you have
> in code files are in database/memcache. If you need use a Cron to put them
> back in memcache every 10 minutes.

Believe me, I've gone down this route.

8s is still unacceptable.  That's well beyond "user clicks reload
because they think site is broken".  And you aren't accounting for
sick periods when latency goes 2-3X that; how do your users feel about
a 24s wait?

I have optimized my codebase about as much as I consider reasonable.
I don't eager load data, I've turned off any kind of classpath
scanning.  I've deliberately used GAE-friendly libraries.  Hell, I
even wrote one myself.  When GAE is behaving, my startup time is
20-30s, and could go down to 10-15s if I wanted to quadruple my bill
with F4s (I don't).   The next step is more or less tantamount to
"give up on Java", or at least programming in something that looks
like modern Java:

 * In order to persist POJOs, the classes must be introspected at
startup.  The alternative is to use only the low-level api; that does
not scale to a complicated project.

 * Nearly every modern Java webapp framework uses annotations to
define a sitemap.  That means loading and introspecting all those
annotated classes before serving the first request.  The alternative
is to find some early-2000s-era relic based on XML files like Struts1
or WebWork (or this blast from the past: http://mav.sf.net).  I'm not
even sure it would help; the first thing most of these do is read the
XML and load the related classes.

 * I could probably squeeze a few more seconds out of startup if I
dropped Guice and AOP.  This would involve rewriting my app more or
less from scratch.  Aside from my fixed investment, I refuse to go
back to programming without tools like these.  It's the difference
between:

@Transact(TxnType.REQUIRED)
void doSomeWork() {
   ...do work
}

and littering my code with crap like this:

Transaction txn = datastore.beginTransaction();
try {
...do work
txn.commit();
} finally {
if (txn.isActive()) {
txn.rollback();
}
}

Actually, these are not comparable, because the AOP-based example is
smart about inheriting a pre-existing transaction context, or starting
one if necessary.  A truly equivalent second example would include a
ton more boilerplate.

Without techniques like AOP and DI, Java programming becomes miserable
busywork.  We picked GAE as a platform because without having to do
the work of an ops staff, we can write code and develop features much
faster.  If I am stuck working at half-speed with crappy tools, then
GAE has a negative value proposition - we'd be faster even with the
ops load, and paying a fraction of GAE's price to boot.

So yeah, I don't expect to see my startup times get much better than
what they are today... and if I double my codebase next year, it could
get worse.

Realistically, I do not expect Google will ever be able to squeeze
Java application starts into reasonable timeframes (ie,
http://www.useit.com/alertbox/response-times.html).  It's just not
realistic given the nature of classloading and the security
precautions that must be built around it.  That kind of JVM
engineering is rocket science, and in the last three years it hasn't
materialized (although, to Google's credit, incremental progress has
been made).

This is why I'm thinking about lateral solutions.  Honestly I don't
really care how long it takes my app to start as long as 1) users
never see cold starts and 2) I can expand capacity to meet increased
load.  From the outside, it really feels like all the pieces have been
implemented, they just are misconfigured right now.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Michael Hermus]

We ARE using warmup requests, that's is the while point of this thread. I dont 
believe that you (or anyone) has sufficiently explained how sending user 
requests to cold instances is ever better than warming it up first.. Said 
request can ALWAYS be pulled right off the pending queue and sent to the 
instance as soon as it is ready.

On Wednesday, July 18, 2012 2:27:45 PM UTC-4, Brandon Wirtz wrote:
> Use warm up requests.
> 
> Also optimize your code to load faster.   I have an app that is HUGE it
> loads in about 8s. On an F4. It doesn't fit in an F2.  Check that you 
> really
> need all your libraries. Check that any of the large blocks of data you have
> in code files are in database/memcache. If you need use a Cron to put them
> back in memcache every 10 minutes.
> 
> If you have multiple people using the same code base, make your app
> multi-tenant so that you have fewer spin ups and more instances to serve
> requests.
> 
> If you are storing init variables in datastore make sure you put them in mem
> cache so you can get them faster. Marshal or Pickle init variables.
> 
> Use inline imports for functions that are rarely called.
> 
> Strip unused class/functions from Libraries and frameworks.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/-AzInxgwLCAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Use warm up requests.

Also optimize your code to load faster.   I have an app that is HUGE it
loads in about 8s. On an F4. It doesn't fit in an F2.  Check that you really
need all your libraries. Check that any of the large blocks of data you have
in code files are in database/memcache. If you need use a Cron to put them
back in memcache every 10 minutes.

If you have multiple people using the same code base, make your app
multi-tenant so that you have fewer spin ups and more instances to serve
requests.

If you are storing init variables in datastore make sure you put them in mem
cache so you can get them faster. Marshal or Pickle init variables.

Use inline imports for functions that are rarely called.

Strip unused class/functions from Libraries and frameworks.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

I don't buy this.  Even if the checkout queue is 15s deep and it takes
15s to bring a cashier online, there's still no value in sending Bob
to wait by the register.  For one thing, it doesn't actually reduce
Bob's wait time by a material amount (he can move to the register in a
few milliseconds).  For another thing, we're only *speculating* that
the register will be open in 15s - sometimes it's 30s or 45s because
the cashier is hungover and keeps dropping the keys.

Sure, it's a lot easier to balance a 50-instance problem because a
load that size will have a lot more inertia and be less spiky.  But I
still want to know how what percentage of your user-facing requests
are hitting cold starts, and how many idle instances you need to keep
it from happening.

I have an e-commerce site.  It doesn't get a lot of traffic but each
request is very important - people are digging out their credit cards
and buying things.  It's *never* ok for a 20s pause to interrupt this
process; people don't like giving money to sites that seem broken.

Jeff


On Tue, Jul 17, 2012 at 9:55 PM, Drake  wrote:
> Jeff,
>
> Check the archive there are several check out lane analogies that I have
> posted.
>
> I agree that the Queue is sub optimal, but it is more sub optimal the
> smaller you are.  When you get to 50 instances it is amazing how well the
> load balancing works.  On the climb up to peak new instances spin up on
> requests rather than causing cascading failures or dramatic spin ups. And on
> the way down instances de-utilized and end of life gracefully.
>
> Using your grocery store analogy, imagine that you are optimizing for a
> guarantee that you will be checked out with in 30 seconds of entering the
> queue. The ideal scenario is that when you get to a spot where you know you
> are 15 seconds from being checked out, and it takes 15 seconds to "open a
> new lane" you want to send users to go stand in line while the register
> opens.
>
> Your goal is to never have to pay on that guarantee, not to serve the
> highest percentage in the least time.  When this is your ideal QoS the
> current load balancing does really well. It does better if it has 10
> registers and can open 2 at a time, rather than when it has 1 register and
> needs to decide if it is going to double capacity.
>
> -Brandon
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Richard Watson]

Ok, but what of Jeff's suggestion would impact performance negatively at 
your scale?  I assume the GAE team would ensure they don't harm your case 
while trying to improve the few-instances case.

On Wednesday, July 18, 2012 6:55:36 AM UTC+2, Brandon Wirtz wrote:
>
> Jeff, 
>
> Check the archive there are several check out lane analogies that I have 
> posted. 
>
> I agree that the Queue is sub optimal, but it is more sub optimal the 
> smaller you are.  When you get to 50 instances it is amazing how well the 
> load balancing works.  On the climb up to peak new instances spin up on 
> requests rather than causing cascading failures or dramatic spin ups. And 
> on 
> the way down instances de-utilized and end of life gracefully. 
>
> Using your grocery store analogy, imagine that you are optimizing for a 
> guarantee that you will be checked out with in 30 seconds of entering the 
> queue. The ideal scenario is that when you get to a spot where you know 
> you 
> are 15 seconds from being checked out, and it takes 15 seconds to "open a 
> new lane" you want to send users to go stand in line while the register 
> opens. 
>
> Your goal is to never have to pay on that guarantee, not to serve the 
> highest percentage in the least time.  When this is your ideal QoS the 
> current load balancing does really well. It does better if it has 10 
> registers and can open 2 at a time, rather than when it has 1 register and 
> needs to decide if it is going to double capacity. 
>
> -Brandon 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/USujDDZsQ94J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Jeff,

Check the archive there are several check out lane analogies that I have
posted.

I agree that the Queue is sub optimal, but it is more sub optimal the
smaller you are.  When you get to 50 instances it is amazing how well the
load balancing works.  On the climb up to peak new instances spin up on
requests rather than causing cascading failures or dramatic spin ups. And on
the way down instances de-utilized and end of life gracefully.

Using your grocery store analogy, imagine that you are optimizing for a
guarantee that you will be checked out with in 30 seconds of entering the
queue. The ideal scenario is that when you get to a spot where you know you
are 15 seconds from being checked out, and it takes 15 seconds to "open a
new lane" you want to send users to go stand in line while the register
opens.

Your goal is to never have to pay on that guarantee, not to serve the
highest percentage in the least time.  When this is your ideal QoS the
current load balancing does really well. It does better if it has 10
registers and can open 2 at a time, rather than when it has 1 register and
needs to decide if it is going to double capacity.

-Brandon


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Jeff Schnitzer]

On Tue, Jul 17, 2012 at 5:21 AM, Takashi Matsuo  wrote:
>
> On Tue, Jul 17, 2012 at 7:10 AM, Jeff Schnitzer  wrote:
>>
>> Hi Takashi.  I've read the performancesettings documentation a dozen
>> times and yet the scheduler behavior still seems flawed to me.
>
> I would rather not use a word 'flawed' here, but probably there is still
> room for improvement. First of all, is there any reason why you can not use
> min idle instances settings? Is it just because of the cost?

My goal is to have every request serviced with minimum latency possible.

Leaving aside implementation complexity, there doesn't seem to be any
circumstance when it is efficient to remove a request from the pending
queue and lock it into a cold start.  There are really two cases:

 1) The request is part of a sudden burst.  The request will be
processed by an active instance before any new instances come online.
It therefore should stay in the queue.

 2) The request is part of new, sustained traffic.  Whether the
request waits in the pending queue for new instances to warm up, or
waits at a specific cold instance, the request is still going to wait.
 At least if it's still in the pending queue there's a chance it will
get routed to the first available instance... which overall is likely
going to be better than any particular (mis-)behaving start.

Imagine you're at the supermarket checkout.  The ideal situation is to
have everyone waiting in one line and then route them off to cashiers
as they become available.  If the pending queue gets too long, you
open more cashiers (possibly multiple at once) until the line shrinks.
 If every cashier has a separate queue, it's really hard to optimize
the # of cashiers since you have some with a line 5 deep and some
sitting idle.

I'm fully prepared to believe there are implementation complexities
that make the single-pending-queue difficult, but what I'm hearing is
that Google deliberately *wants* to send requests to cold starts...
which seems "flawed" to me.  Am I missing something?

> If so, can 'introducing somewhat lower price for resident instance' be a
> workable feature request?
>
> Vaguely I have a feeling that, what you're trying to accomplish here is to
> save money while acquire good performance. If so, it is one of the most
> difficult thing to implement. However, in my opinion, it's worth trying to
> implement, so let's continue the discussion.

Let's forget price for a moment and just try to work towards the goal
of having an efficient system.  Presumably, a more efficient system
will be more cost-effective than a less efficient system that has lots
of idle instances sitting around hogging RAM.  Good for Google, good
for us.

> If you have an app with average +50s loading time, I totally understand that
> you strongly want to avoid sending requests to cold instances. On the other
> hand, there are also many well-behaved apps with <5 secs loading/warming
> time. Please understand that for those apps, it is still acceptable if we
> send requests to cold instances, so it's likely we can not prioritize the
> feature bellow over other things, however...

Someone at Google must have chart which has "Time" on the X axis and
"% of Startup Requests" on the Y axis - basically a chart of what
percentage of startup requests in the Real World are satisfied at
various time boundaries.  I have a pretty good idea, I think, of what
this chart looks like.  I'm also fairly certain that the Python chart
looks nothing like the Java chart.

For one thing, the Java chart _starts_ at 5s.  The bare minimum Hello,
World that creates a PersistenceManagerFactory with one class (the
Employee in the docs) takes 5-6s to start up.  And this is when GAE is
healthy; that time can easily double on a bad day.

So if you optimize GAE for apps with a <5s startup time, you're
optimizing for apps that don't exist - at least on the JVM.  I'd be
*very* surprised if the average real-world Java instance startup time
was less than 20s.  You just don't build apps that way in Java.  Given
a sophisticated application, I'm not even sure it's possible unless
the only datatypes you allow yourself are ArrayList and HashMap.

>> The min latency setting is actually working against us here.  What I
>> really want is a high (possibly infinite) minimum latency for moving
>> items from pending queue to a cold instance, but a low minimum latency
>> for warming up new instances.  I don't want requests waiting in the
>> pending queue, but it does me no good to have them sent to cold
>> instances.  I'd rather they wait in the queue until fresh instances
>> come online.
>
> For me, it look like a great idea. Can you file a feature request for this,
> so that we can get a rough idea of how many people want it, and start an
> internal discussion.

http://code.google.com/p/googleappengine/issues/detail?id=7865

I generalized it to "User-facing requests should never be locked to
cold instance starts".

Thanks,
Jeff

-- 
You received this message becaus

Re: [google-appengine] Re: Startup time exceeded...on F4?!

[Takashi Matsuo]

Hi Jeff,

On Tue, Jul 17, 2012 at 7:10 AM, Jeff Schnitzer  wrote:

> Hi Takashi.  I've read the performancesettings documentation a dozen
> times and yet the scheduler behavior still seems flawed to me.
>

I would rather not use a word 'flawed' here, but probably there is still
room for improvement. First of all, is there any reason why you can not use
min idle instances settings? Is it just because of the cost?

If so, can 'introducing somewhat lower price for resident instance' be a
workable feature request?

Vaguely I have a feeling that, what you're trying to accomplish here is to
save money while acquire good performance. If so, it is one of the most
difficult thing to implement. However, in my opinion, it's worth trying to
implement, so let's continue the discussion.

Once a request is taken from the pending queue and sent to an instance
> (cold or otherwise), it's dedicated to execution on that instance.  In
> the queue, it can still be routed to any instance that becomes
> available.  Why would we *ever* want to send a request to a cold
> instance, which has an unknown and unpredictable response time?  If I

were that request, I'd want to sit in the queue until a known-good
> instance becomes available.  Depending on the queue fill rate I might

still end up waiting for an instance to come online... but there's
> also a good chance I'll get handled by an existing instance,
> especially if traffic is bursty.
>

> "the scheduler starts a new dynamic instance because it is really
> needed at that moment."  -- this is not an accurate characterization,
> because new instances don't provide immediate value.  They only
> provide value 5+ (sometimes 50+) seconds after they start.  In the
> mean time, they have captured and locked up user-facing requests which
> might have been processed by running instances much faster.
>

If you have an app with average +50s loading time, I totally understand
that you strongly want to avoid sending requests to cold instances. On the
other hand, there are also many well-behaved apps with <5 secs
loading/warming time. Please understand that for those apps, it is still
acceptable if we send requests to cold instances, so it's likely we can not
prioritize the feature bellow over other things, however...


> The min latency setting is actually working against us here.  What I
> really want is a high (possibly infinite) minimum latency for moving
> items from pending queue to a cold instance, but a low minimum latency
> for warming up new instances.  I don't want requests waiting in the
> pending queue, but it does me no good to have them sent to cold
> instances.  I'd rather they wait in the queue until fresh instances
> come online.


For me, it look like a great idea. Can you file a feature request for this,
so that we can get a rough idea of how many people want it, and start an
internal discussion.

Thanks as always,

-- Takashi


>
> Jeff
>
> On Mon, Jul 16, 2012 at 1:15 PM, Takashi Matsuo 
> wrote:
> >
> > Richard,
> >
> >> But Tom seems to think that "1" is an appropriate number for his app.
> Why
> >> offer that option if it's automatically wrong?
> >
> > If his purpose is reduce the number of user-facing loading requests, and
> he
> > still sees many user-facing loading requests, the current settings is not
> > enough.
> >
> > Jeff,
> >
> >> I vaguely expect something like this:
> >>
> >>  * All incoming requests go into a pending queue.
> >>  * Requests in this queue are handed off to warm instances only.
> >>  * Requests in the pending queue are only sent to warmed up instances.
> >>  * New instances can be started up based on (adjustable) depth of the
> >> pending queue.
> >>  * If there aren't enough instances to serve load, the pending queue
> >> will back up until more instances come online.
> >>
> >> Isn't this fairly close to the way appengine works?  What puzzles me
> >> is why requests would ever be removed from the pending queue and sent
> >> to a cold instance.  Even in Pythonland, 5-10s startup times are
> >> common.  Seems like the request is almost certainly better off waiting
> >> in the queue.
> >
> > Probably reading the following section woud help understanding the
> > scheduler:
> >
> https://developers.google.com/appengine/docs/adminconsole/performancesettings#scheduler
> >
> > A request comes in, if there's available dynamic instance, he'll be
> handled
> > by that dynamic instance. Then, if there's available resident instance,
> > he'll be handled by that resident instance. Then he goes to the pending
> > queue. He can be sent any available instances at any time(it's fortunate
> for
> > him). Then according to the pending latency settings, he will be sent to
> a
> > new cold instance.
> >
> > So, if you prefer pending queue rather than a cold instance, you can set
> > high minimum latency, however, it might not be what you really want
> because
> > it will cause a bad performance on subsequent requests.
> >
> > Generally speaking, just looking a