RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Jeff,

Check the archive there are several check out lane analogies that I have
posted.

I agree that the Queue is sub optimal, but it is more sub optimal the
smaller you are.  When you get to 50 instances it is amazing how well the
load balancing works.  On the climb up to peak new instances spin up on
requests rather than causing cascading failures or dramatic spin ups. And on
the way down instances de-utilized and end of life gracefully.

Using your grocery store analogy, imagine that you are optimizing for a
guarantee that you will be checked out with in 30 seconds of entering the
queue. The ideal scenario is that when you get to a spot where you know you
are 15 seconds from being checked out, and it takes 15 seconds to "open a
new lane" you want to send users to go stand in line while the register
opens.

Your goal is to never have to pay on that guarantee, not to serve the
highest percentage in the least time.  When this is your ideal QoS the
current load balancing does really well. It does better if it has 10
registers and can open 2 at a time, rather than when it has 1 register and
needs to decide if it is going to double capacity.

-Brandon


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Use warm up requests.

Also optimize your code to load faster.   I have an app that is HUGE it
loads in about 8s. On an F4. It doesn't fit in an F2.  Check that you really
need all your libraries. Check that any of the large blocks of data you have
in code files are in database/memcache. If you need use a Cron to put them
back in memcache every 10 minutes.

If you have multiple people using the same code base, make your app
multi-tenant so that you have fewer spin ups and more instances to serve
requests.

If you are storing init variables in datastore make sure you put them in mem
cache so you can get them faster. Marshal or Pickle init variables.

Use inline imports for functions that are rarely called.

Strip unused class/functions from Libraries and frameworks.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I can make searches that are slow. You just have to pick things no one has
searched before.

That said, the 1% of bad check outs would only apply if you only serve 1
request from the slow server. If each instance serves 100k requests in its
lifetime, then you are talking about 1/100k being negatively impacted. Even
my lowliest of instances serves 3000 in 2 hours. Before dying.

I said 15s spinup because I think that was your number.  I shoot for spin
ups that are under 4s.

If my spinup is more than 4s I create a new app, or dynamic back end to
handle the requests with specialized version of the microapp that is faster
to load.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Dump PMF use the low level API
https://developers.google.com/appengine/docs/java/javadoc/com/google/appengi
ne/api/datastore/package-summary

Either drink the Kool Aid, or quit complaining :-)

5s will be 1.5




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Uploading a json file from local computers. URLfetch?

[Drake]

Use the DataStore. 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of soujiro0725
Sent: Thursday, July 19, 2012 10:48 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Uploading a json file from local computers.
URLfetch?

 

GAE for Python!

 

Hi. I am trying to develop a web service which users can upload json files
and modify the data.

 

Users create json files at local computers, and in the web service page,
browse the path to the files and press 'upload'.

Then, a python handler parses the file contents and make necessary changes.

 

My question is the process of uploading the files from local computers.

 

Am I supposed to use 'URL fetch' API? by indicating the URL to the local
file storage?

 

Or is there any other possible way to achieve this?

 

 

soichi

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/MBQ6_bOeEtYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

 

>If you follow the group longer you should know - Brandom lives in GAE
unicorn land and all you ever need are proper Edge cache settings. ;)




No, I just don't write code using unnecessary frameworks and I do a ton of
testing and architecture planning. 

Most everyoneelse never uses defer or threads or task queues. They don't
manage steady state tasks, or profile their imports.

It is easy to blame the platform, but if you write crap code it will run
like crap.

I don't think GAE is perfect. But most the people who complain about its
failings don't have the coding skills to be an intern at my shop. 

I blame Google for not documenting everything well. I blame google for not
telling you "don't use frameworks they suck".

But the number of people who load spring and the gripe that it is slow and
are only using 3 things out of it make me want to punch them in the head.

The people who don't know how to build APIs so that apps are task specific.
Piss me off. Build modular. Dump frame works. Defer often. Be your own
scheduler by shaping internal ops. Cache incoming. Cache reads cache writes.
Manage threads. Use warmups. This is not rocket science.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

All your pages should have the ability to have an optional ?debug=true

Then if you have set that in the url the first line in every one of your
functions, and before your imports,  should be "Timer start" and the last
thing should be "Log Timer"


> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Friday, July 20, 2012 11:00 AM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> On Fri, Jul 20, 2012 at 10:34 AM, hyperflame 
> wrote:
> > Discussing theoretical startup times is great, but I'd like to see
> > some real-world startups. Does anyone with high startup times ( say,
> > 30+ seconds) want to share the results of a code profiler/appstats?
>
> What code profiler works on GAE server-side?  It's not very helpful to
profile
> in the dev environment since the problem does not manifest there.
>
> Appstats is not helpful; my startup is long even with zero service layer
calls.
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Ah, yes, and we come back to Brandon must only run one kind of app, cause I
mean who runs more than one App? He can't know Java AND Python.

I'm apparently not the only person living in Unicorn land, Dave Chandler
also lives there
http://turbomanage.wordpress.com/2010/03/26/appengine-cold-starts-considered
/ So does Meetu Maltiar
http://thoughts.inphina.com/2010/10/09/managing-cold-start-on-google-app-eng
ine/

Just because you are too lazy to optimize your code, and want to blame the
platform don't take your anger out on me.

Looking at the archives you have been singing the same song for 2 years.
Not once have a I seen you say, "Yeah my imports are now being lazy loaded"
or " I serialized my configuration initialization variables" and "It saved
me 1s but I need to find 5 more where should I look"

All I see from you and Andre is Wha-Wha, Wha-Wha, Wha-Wha-Wha.

If Python is magic move to it. I run about 70% py 30% java because unlike
you I profiled my apps and used which ever was better. And in the cases
where Go Is Better. I'm Learning and  building code in Go now too.

So grow up. Quit whining read the docs on the internet and when you can post
"I did all those things and my app still doesn't load fast enough" and when
you tell me you modularized your code so that you can leverage backend for
steady state and large code base latency tolerant apps, come back to me.
I'll help you with a code review just to get you to stop complaining.

It's ok, not every programmer and learn to think in the Google code
construction methodology, or think in "cloud". But there are clearly highly
successful projects that have gotten around your issues. So stop saying
those people live in a magical land not available to everyone else.  Google
doesn't look at my Email associated with the app and give me a performance
bonus. (I Know I checked)










> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Saturday, July 21, 2012 1:19 AM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> Brandon, your comments are irrelevant and not constructive.  The Python
> runtime has a completely different startup profile from the Java runtime.
> The long startup delays in Javaland occur prior to any service calls; no
amount
> of caching, deferring, queueing, or serialization is going to help.
>
> Jeff



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 



Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 12:27 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Brandon, your 1.5 seconds - I call this nonsense in Java world, thats all.
Create empty Java Project with 1 Servlet, disable JDO/Datanucleus stuff, no
Framework, nothing.


2012-07-21 12:14:33.275 / 200 3525ms 1kb Mozilla/5.0 (Windows NT 6.0;
rv:14.0) Gecko/20100101 Firefox/14.0.1 



May be there was some time where your mantioned link could reach 2.5 seconds
- but it's not regular.
Do I have problems with 5 seconds startup time? No - it's considered very
fast in the Java world. I don't use any Frameworks, have learned that early.
Just saying...1.5 seconds. And your regular posts that all is fine and the
people are just doing it wrong are at least as annoying as mine.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/tv73lRqnY1MJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Read a local resource? You mean you do FileReader? 

 

Don’t do that. That’s bad. Put that in Memcache and read it from memcache.
Keep it in memcache at all times.

 

Never use FileReader.  Ever. Unless it is to get an app booted for the very,
very first time. It should be in the datastore, and in memcache.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 1:00 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Hi,

I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a
local ressource and writes it to the output stream, thats all.
Hello World.
Even the empty container does need some class loading - thats the Java
world. And currently people have startup timing problems.

1.5 Seconds are possible if all your stuff is static and in Edge cache,
nothing really starts up there.

Best regards,
André


Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 

Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Shit, even putting it in your static and Fetching by URL is faster than File
reader. 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 1:00 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Hi,

I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a
local ressource and writes it to the output stream, thats all.
Hello World.
Even the empty container does need some class loading - thats the Java
world. And currently people have startup timing problems.

1.5 Seconds are possible if all your stuff is static and in Edge cache,
nothing really starts up there.

Best regards,
André


Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 

Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/jOg0BaLfu6YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

No the point is a major limitation of Appengine is that File IO is SLOW so
you never do “open file” 

You want to do “dynamic” pull that from memory or datastore.

 

NEVER EVER LOAD DATA FROM FILE OTHER THAN TO POPULATE THE DATASTORE FOR THE
FIRST TIME IF YOU CARE ABOUT WARMUP

Yes I have an exception where I do that with a 40 MB pickled file that it is
faster than doing 40 reads from the datastore and doing a join because of
the memory usage, but that is a weird case.

 

But yes. You are doing it wrong. And if you would profile your apps you
would know you are doing it wrong.  

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Saturday, July 21, 2012 1:19 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

I use getRessourceAsStream an Stream-copy - but you miss the point...it's a
Hello World, done in 5 minutes as test for your post, stripping away all
excuses.
The calls after startup are answered in <80 ms.
That mans that I'm still far above 3 seconds for initializing of an empty
project. I could simply write Hello World into the Stream.

I could use static...yes...but thats not the point. Some people have dynamic
and user dependand data on each page.


Am Samstag, 21. Juli 2012 22:09:11 UTC+2 schrieb Brandon Wirtz:

Shit, even putting it in your static and Fetching by URL is faster than File
reader. 

 

 

 

Hi,

I do nothing. I have 1 Servlet, no additional Libs. The Servlet reads a
local ressource and writes it to the output stream, thats all.
Hello World.
Even the empty container does need some class loading - thats the Java
world. And currently people have startup timing problems.

1.5 Seconds are possible if all your stuff is static and in Edge cache,
nothing really starts up there.

Best regards,
André


Am Samstag, 21. Juli 2012 21:41:02 UTC+2 schrieb Brandon Wirtz:

The intent of that line was that you would save 3.5s dropping factory for
the GAE API

 

But one of my main apps is 2.2s average spinup time, I am pretty sure I can
get Hello world in under 1.5.  

 

Are you lazy loading? Are your lazy loads lazy loading? Have you threaded
your Lazy Loads?

Class 1 has dependencies ABC,  Class 2 has depencies DEF, 

Warm up does basically nothing but defer Class1 and Class2

 

If Cold Start only do Called Class,  If first call to instance defer other
classes for warm up.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/cTL6IofY1ygJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

When I said 1.5 I meant PMF was adding 3.5 you don’t need. PMF takes 5s the
API takes under 1s.  I think 2.5s is probably the lowest you can get and do
something truly useful, but it seems that I can do a LOT in 2.5s  (assuming
the F4 this thread was about when we started) (well actually assuming F2,
F4’s seem to need 3s no matter what you do)

But also you don’t have to do anything useful on warmup. So you shouldn’t
serve a ColdStart very often.

 

I have mentioned before I work to keep everything to under 4s. If my App
Gets to 8s I start slicing it up 

 

I am also 100% confident that if all you do is say “I am up” your warm up
will take under 1.5s  for 95% of starts. And that should really do “I am up”
and defer lazyloads of everything you need to be ready to do real work.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of hyperflame
Sent: Saturday, July 21, 2012 1:47 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

I just looked over the logs of a corporate GAE application. It's a very
simple "heartbeat" application; essentially our production applications have
to send a message to it periodically. The message sent is about 500
characters; the application does some checksum-ing and sends back a JSON
string about 100 characters long. The only external libraries are jars of
twitter4j and the org.json parser. It's really not much more complex than a
Hello World app.

Looking over the logs, the fastest start time from cold-instance to
first-request-served is 1695 ms (although on average, it needs 2.2 to 2.5
seconds to cold-start) on a F1 instance. Once the instance is warm,
subsequent requests take, on average, ~150ms to process and send back a
reply. I could probably get that lower if I upgraded to F4.

On Saturday, July 21, 2012 3:19:21 PM UTC-5, André Pankraz wrote:

I use getRessourceAsStream an Stream-copy - but you miss the point...it's a
Hello World, done in 5 minutes as test for your post, stripping away all
excuses.
The calls after startup are answered in <80 ms.
That mans that I'm still far above 3 seconds for initializing of an empty
project. I could simply write Hello World into the Stream.

I could use static...yes...but thats not the point. Some people have dynamic
and user dependand data on each page.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/TDuwyumyZAIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

If it makes you feel better I have this fight once a month. AWS that is
easier to build for or GAE which doesn't require any sys admin and is
infinitely elastic.

 

Someone will put something on AWS show me how awesome it is, and how cheap.
I will throw the load simulator at it, and watch it burn.  

They say but you didn't give it more instances.  

 

I will say fine every 4th  8 minutes when CNN runs a commercial for the site
you need to handle  5000 uesrs, and the rest of the time you need to handle
8. 

Which is cheaper?

 

They always lose.

 

How do we manage keeping the versioning the same on all instances in AWS?
How long is the down time on deploy of new version?  (GAE you can deploy,
warm up enough instances and change the "Default" version and not even lose
sessions)

 

Every service has plusses and minuses if they didn't then someone would be a
monopoly.

 

Edge Cache is amazing for most apps it will negate the AWS "savings". Rapid
Elasticity is amazing it will negate even RackSpace savings.

 

Not taking off the shelf code. That will cost you a developer. Sorry. I
would also say most the time, you should be rewriting that code anyway to be
exactly what you need, but I get that sometimes what you want is to use
Facebook Authorization, and there is no reason to strip stuff out.  Except
that you have to remember everyone who is not you writes sucky code. Generic
code. Meant to run everywhere. Except GAE is not everywhere, it is in
unicorn land where you traded having nice friendly donkey's who will put up
with what ever shit you feed them and still pull your wagon, for a flying
unicorn that shoots rainbows out of its ass but requires that you don't just
give it a sugar cube every now and then, but that it is raw organic sugar
fed to it by beautiful naked virgins.  In exchange you can go farther,
faster, and you don't have to clean up poo off the floor.  These are the
trades you make for living in Unicorn land.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Tomas
Sent: Saturday, July 21, 2012 2:50 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: Startup time exceeded...on F4?!

 

Hi guys,

I actually didn't want to reply to this thread originally even I was the one
who opened the small thread 3 months ago regarding the very slow startups on
gae with spring + velocity + objectify but after reading the email telling
me I should "write modular apps and don't use frameworks" I have to write up
something.

I'm 10+ years java developer and I'm big fan of Spring as it saves a hundred
hours of time - I originally developed the www.librarist.com 2 years ago as
a pet project on app engine as a just a prototype using simple JSPs +
Servlets and it works working fine (startups 3-5 secs).

But when I wanted to add another features like cart, OAuth login, use
templating, use proper caching - and I decided to go with Spring as this the
reason why we have frameworks. Plugging in Velocity is just matter of 1 line
of code in XML (plus some tweaks if you need them), the cache is again
couple of lines in XML and you automatically get in memory + memcache with
ehcache, authentication is 2 lines of code. And I can continue.

I use spring in work for other projects which runs on full java stack (but
thats just Tomcat/Jetty) and when I run the "new" librarist equivalent code
on Micro instance on AWS (which is like 600MB ram and 1.8Ghz - I might be
wrong here a little bit), the startup is 2-3 secs and I can serve all my
traffic with it with load at max 0.5.

When I run the similar code on GAE, I get 45+ secs startup (when I'm lucky,
sometime it just go over 1 nib and got killed) and then GAE keep spinning
another instances like crazy but very often use them for three request and
then dying to get started again. Worth to mention - I do not use the full
Spring stac, basically its just Spring core + Spring MVC, then objectify +
Velocity - and I really tried to do all the possible optimizations (packing
all the stuff into one big jar, used annotations + java confing, used XML,
disable scanning of annotations) but these changes just save me like 4-5
secs from 45 secs.

When I tried just Spring code + Spring MVC without any other library (and
here we talk about ~2MB of jars), use one JSP, my best startup time is 15
secs.

Okay thats cool, I can understand the  GAE is different and basically it
could be used just for small projects where you write everything by yourself
and don't use any normal stuff like Dependency Injection / Proxying and
similar. But my biggest question for GAE guys would be this - why the hell
the startup takes so long simply I don't get what is actually the issue
here:

- is it loading of the jars from the network (virtual drive) when the gae
loads each class as separate file
- is it the scanning of the classes (annotations etc)
- is it java reflections
- is it creating of proxies (which Spring does for almost everything)

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I didn't need task queues/defer because I was self managing queues and in
fact do that more than using task Queue still because it lets me do more
process shaping. So now short term tasks are defer, and long term tasks go
in to a self managed bucket that fires based on current load as determined
by the number of instances active.

>  * How do you persist data?  (low-level, jdo, objectify, etc)
Actually I use a multi-approach based on the work being done.  Python has
NDB, CachePy, and a number of things that Java seems to missing good analogs
for, but using the lowlevel API gets things up and running and does the
majority of the heavy lifting. When we don't want to have to manage a bunch
of code to handle stuff that is more complex or do things that are less time
sensitive like deep queries for single users, we use JPA. (I think JDO is
mostly dead.)

>  * How many entity kinds/classes do you have?
A shit ton. For the analytics app we built we found out that you are limited
to 65535 and that Dynamic is not as dynamic as you might think.

>  * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
How quaint.  Did you miss that I said one type of task per app/backent?
Web.XML at the individual app, but path routing is handled by the
caching/loadbalancing that is done using the python version of CDNInabox for
most our apps, and Squid on another service for some of the other Apps.  No
"App" ever is exposed directly to a user request, they are all proxied. (ps
this also allows us to do much of our own QoS manipulation because we can
send a request that is taking too long to an F8 that has multiple apps
installed on it and can be anything it needs at a moments notice.

>  * How many URL endpoints do you have?
Per app? Or In total? In many cases we have endpoints per domain in
multi-tenant apps, and we are running 10k plus domains.

>  * How many total java classes in your application?  What is total size?
>  * How many jars in your WEB-INF/lib?  What is total size?
Per app? Maybe 20. In an application deployment couple thousand.   Last time
I looked our code base was nearly 800 megs, with 80% of apps at less than 10
megs


Jeff, my "toys" are bigger than your "projects".   My unique indexed pages
in Google is over 1 Billion. I likely spend more in Datastore file costs
than you spend hosting your entire client base.



> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Saturday, July 21, 2012 9:27 PM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> Brandon, you talk a lot of shit for a guy who only discovered the task
queue 6
> months ago.
>
> Everyone who thinks they have the secret to starting up a Java app in <5s,
> answer these questions and prove that you're running more than a
> toy:
>
>  * How do you persist data?  (low-level, jdo, objectify, etc)
>  * How many entity kinds/classes do you have?
>  * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
>  * How many URL endpoints do you have?
>  * How many total java classes in your application?  What is total size?
>  * How many jars in your WEB-INF/lib?  What is total size?
>
> Having done a fair bit of experimentation around this issue, I know that
> you're talking out of your asses.  Clever suggestions like "rewrite app in
> Python" aren't nearly as clever as you think they are.
>  Suggestions that you can optimize startup time by some combination of
> caching, task queues, or threading merely indicate total lack of
understanding
> of the problem in the first place.
>
> To the people paying attention to this thread who are actually trying to
build
> real world apps on Appengine:  Brandon does not speak for the GAE team.
> Neither do I, but I occasionally talk to people that do.  I know that
Google
> really does want Java apps to run well on GAE.  They know about this issue
> and care about it - maybe even enough to do something about it.  It *is*
> constructive to "whine" politely about startup time because there are many
> issues competing for the team's time, and our feedback helps them
prioritize
> their efforts.
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: numpy and scipy question

[Drake]

SciPy is mostly in Fortran. So I don't give you good odds.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 12:05 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: numpy and scipy question

 

Hi Anand,

 

Are there plans to also support SciPy in the future?

 

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het
volgende:

NumPy is available on the Python 2.7 runtime, but SciPy is not.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

And this would be why we don't use contractors.  Optimizations aren't for
"Google" they are good practice.  When you run on 600 mhz with 128 Megs of
ram, you see the optimizations when you have 3.2 Ghz * 8 and 64 gigs of ram,
you can't hardly profile an app. That doesn't mean you can't make it twice
as efficient.

 

The tools you know and love suck. You just don't know it because you are
spoiled by big computers.  I love my '50 Merc and my '66 Stang, but my
MiniCooper has a lower cost of operation and has a faster 0-60 and top speed
than either of the old Fords do.  Because it is optimized. 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Mauricio Aristizabal
Sent: Sunday, July 22, 2012 12:28 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

I really don't care who has the shiniest toys in this playground, bottom
line is that I consider every minute spent getting my app to load faster a
complete waste of my time.  Adding business logic, minimizing user request
times and costs, that's what I care about, not jumping through hoops to get
around some design flaw that Google could easily fix if they chose to.

 

If you have a zillion apps and a huge budget and/or team, and it is cost
effective to do a crapload of optimizations, good for you! but I suspect
most of us just want to get our apps to work reasonably efficiently, in as
little time, and with as many of the tools we already know and love as
possible, so we can move on to our next project, client, idea, etc.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/gmXX3iD35ccJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

> I will be generous and assume for the moment that this makes sense for
> your particular application.  At best you are arguing that you have a
wacky
> application.  You won't find too many people building business apps that
way,
> especially when you have elaborate transactional logic.
>  There are sometimes good reasons to compose an app out of more
> fundamental services, but this should be driven by application design
> - not a dumb limitation of how many classes you can pack into a JAR before
> instance startup times get out of control.
>
> > Web.XML at the individual app, but path routing is handled by the
>
> If I went the web.xml route, it would map 326 servlets.  Not really fun.
> There's a reason people started inventing "web frameworks" in the late
> 90s... because that kind of programming sucks when your app is even mildly
> complicated.
>

And this is where you prove your stupidity.

This is how every large "app" on the planet works.
A load balancer serves requests to the right "app" and each app is optimized
for what it is supposed to do. Data is shared between apps using API's,
Memcache, and Datastore.

I know what an "end point" is. Apparently you don't.  Each of what you call
and End point should be a micro app. A single purpose App that handles one
type of request.  This lets you build my Unicorns. Fast little apps that do
one thing, and do it fast. Also because they do only one thing they get to
maximize their instance memory.

Java gets REALLY slow and has more warm ups when you have 200 megs of code
and 128 megs of ram. (as in it doesn't work, and if by some miracle it does
make it past warm up typically you hit soft memory limit and it dies)

You clearly are building Monolithic code in a world of Micro-instances. STOP
IT. You will only have pain doing so. Build single purpose instances that do
what they need to do. I'm sorry your app is too small for there to always be
one instance doing each task.  That's not my fault. Get more popular or
something. In the mean time use lazy loads so that you can fake having
instances that do one and only one thing by only loading enough stuff to do
one thing until you need to do that other thing.

I'm clearly not going to convince you to do anything, because you don't want
to.  You want somebody else to fix your problem so you won't even look at
solutions.
At which point I have to point out you are always going to be sad and
miserable.

PS
Survivor Finale. And Comments aren't static. Nor are "post to facebook". Nor
are Oauth calls.






-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

200 1625ms 98kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4)
AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6
70.162.197.131 - - [22/Jul/2012:01:38:53 -0700] "GET /www.xyhd.tv HTTP/1.1"
200 98752 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4)
AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6"
"second.cdninabox-java.appspot.com" ms=1626 cpu_ms=844 api_cpu_ms=0
cpm_usd=0.042899 loading_request=1
instance=00c61b117c382cd6d2d719025a767570ad95c0
I 2012-07-22 01:38:53.885
This request caused a new process to be started for your application, and
thus caused your application code to be loaded for the first time. This
request may thus take longer and use more CPU than a typical request for
your application.

I read that as 1.626 MS.

What I did in that time on an F1.
Initialized the App from nothing.
Read the configuration out of Datastore (not even from Memcache)
Did a URL Fetch to read the contents of XYHD.tv (like you doing your read
from file)
Regexed it to change all the paths from relative to absolute.
Deferred a write to the datastore so I wouldn't have to do that for the next
request that comes along in the next 30 seconds.
Output the data.

I still have 370 Ms to do something else.  And that is on an F1.

I think that counts as Hello world.

> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Sunday, July 22, 2012 1:18 AM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!
>
> On Sat, Jul 21, 2012 at 11:32 PM, Drake  wrote:
> >>  * How do you persist data?  (low-level, jdo, objectify, etc)
> > Actually I use a multi-approach based on the work being done.  Python
> > has NDB, CachePy, and a number of things that Java seems to missing
> > good analogs for, but using the lowlevel API gets things up and
> > running and does the majority of the heavy lifting. When we don't want
> > to have to manage a bunch of code to handle stuff that is more complex
> > or do things that are less time sensitive like deep queries for single
> > users, we use JPA. (I think JDO is mostly dead.)
>
> This thread is about Java instance startup time.  Enough about Python
> already.  It's irrelevant.
>
> You've completely - and lamely - dodged ALL of my questions.  You
insinuate
> that you have sophisticated Java applications that start up quickly.  I'm
calling
> bullshit.  Give me ACTUAL numbers for one ACTUAL Java application.
>
> >>  * How many entity kinds/classes do you have?
> > A shit ton. For the analytics app we built we found out that you are
> > limited to 65535 and that Dynamic is not as dynamic as you might think.
>
> Generated kinds or classes don't count.  I mean this literally, how many
user-
> defined entity types does your Java app load?  You're going to tell me you
> have Java apps that load 64k JPA classes?  Not a chance.
>
> See, this is getting to the ACTUAL problem with the Java environment -
> classloading is slow.  Entity classes must be loaded and introspected
before
> datastore operations can start.  Near as I can tell, the only way around
this
> particular problem is to stay with the low-level api - which, like
assembly
> programming, severely limits how sophisticated an application you can
> reasonably build.
>
> >>  * How do you manage URL routing?  (ie, jsp files?  web.xml?  JAX-RS)
> > How quaint.  Did you miss that I said one type of task per app/backent?
>
> I will be generous and assume for the moment that this makes sense for
> your particular application.  At best you are arguing that you have a
wacky
> application.  You won't find too many people building business apps that
way,
> especially when you have elaborate transactional logic.
>  There are sometimes good reasons to compose an app out of more
> fundamental services, but this should be driven by application design
> - not a dumb limitation of how many classes you can pack into a JAR before
> instance startup times get out of control.
>
> > Web.XML at the individual app, but path routing is handled by the
>
> If I went the web.xml route, it would map 326 servlets.  Not really fun.
> There's a reason people started inventing "web frameworks" in the late
> 90s... because that kind of programming sucks when your app is even mildly
> complicated.
>
> >>  * How many URL endpoints do you have?
> > Per app? Or In total? In many cases we have endpoints per domain in
> > multi-tenant apps, and we are running 10k plus domains.
>
> I was really asking about your "unicorn" java app that loads in 2 seco

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

> All I have to say is "wow".  I'm really glad you're just a troll here and
not
> actually responsible for anything I depend on.

Let's see... You spend your life complaining how the platform sucks. I
release tutorials on how to make it suck less.  I'm the troll?

Anyone who thinks the low level api is analogous to assembly probably
doesn't want to depend on me because the laughter when they asked a question
might be deafening.

Read EVERY optimization guide on Java for GAE and that is the first thing
they tell you to do.  If that is too hard, then this is not the platform for
you.  I have vouchers for Azure, hit me up off list I'll send them along,
you will be happier in .Net where you never have to do anything low level.
J# is supported for 3 more years... So you should have plenty of time to
learn basic principles of cloud development in that time.


All animosity aside.  I'm sure you are a good Java dev, but this isn't Java.
The Python I write isn't Python either.  I mean it is, but not how you would
write it if you were on a "real" server. You have to get over that. You are
going to suck at GAE until you do. NO FRAMEWORKS. NONE. Deal with it. Every
email you beat me over the head with why you need PMF. Well Frak PMF it is
holding you back. It is making you stupid. You don't need it. If you need it
for one really hard thing once in a great while that is what a lazy load is
for.

Yes classes suck we went over that. Which is why you build micro apps that
are purpose built and only use limited classes.

Yes I have an F8 that can be anyone of my Micro Apps, it has 63k Entities.
It lazy loads. So it doesn't break, It is used for tests and for emergency
off loading of traffic.
You don't have that kind of traffic so you don't need that, I get that.

But you need to get over yourself and look at profiling your app, and
determining where you can cut the fat, and how you can break a single
monolithic app in to 4 smaller apps.  I don't know what that looks like,
maybe it is Customer authentication and session handling, Credit card
processing and Order insertion, and Product Look up and shipping
calculation.

In so doing you can limit your classes, you can make better use of your
instance memory, and you can likely keep from ever hitting warm ups because
each app will run more efficiently than the monolith.

Yeah that is a lot of work, and sure it sucks that this guy who called you
stupid told you to do it, but it is the right thing to do.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I don't use frameworks in Python either. Ask my Dev team. They all but riot 
when they find a library they want to use and I say, figure out what it does 
and make your own, we don't have room for that thing.

Java aint Ruby on Rails. (or Oh please don't let it be) I get that the appeal 
of Java is that it is older than dirt and there is a framework for everything. 
You can use 1 frame work if it is a small one :-).

You are trading rapid elasticity for ease of development. You have to remember 
that when you are small Servers are cheap and humans are expensive. When you 
get to Google Scale that reverses.  GAE is optimized for near Google scale. 
(not all of Google, but Small Google product) if you are 1/1000th that the dev 
resources for optimization will kick your ass. Just like the guys who say they 
wont' do it.  Those are guys who don't spend $50k a month on hosting. When the 
amount you spend is double the cost of a dev, it is time to look at ways to 
cut that hosting in half using a dev, because you will break even, and give 
higher QoS.




> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Aleksei Rovenski
> Sent: Sunday, July 22, 2012 2:05 AM
> To: Google App Engine
> Subject: [google-appengine] Re: Startup time exceeded...on F4?!
>
> I can understand that GAE is more optimized for python than for java.
> Maybe it is a highly specialized tool for really tiny apps that use no
> frameworks. But I don't get one thing. How Google plans to compete for java
> apps by selling platform that forces you to not use frameworks? I simply
> refuse to go back into the 90's (in software development aspect, no problem
> otherwise :)). I simply do not want to drop dependency injection and other
> patterns I have been doing for many years already. If I must, then I think 
> it
> just isn't worth it.
> Seriously Java is about frameworks. Saying not to use frameworks in java is
> same as use python instead. No thanks.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: numpy and scipy question

[Drake]

I don't know then. I thought NumPy was all C. 

Any how I use Numpy but not SciPy. Apparently Google caters to my every whim
and so they gave me what I needed.  J

Depending what you need Matrix.py, and Euclid.py get you many of the most
common things you would use Numpy and Scipy for.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 1:34 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

NumPy is also Fortran. 

 

But my confusion is that if SciPy will not be supported, then why support
NumPy? NumPy and SciPy are hardly ever used separately.

 

Hans

Op zondag 22 juli 2012 09:11:00 UTC+2 schreef Brandon Wirtz het volgende:

SciPy is mostly in Fortran. So I don't give you good odds.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 12:05 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: numpy and scipy question

 

Hi Anand,

 

Are there plans to also support SciPy in the future?

 

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het
volgende:

NumPy is available on the Python 2.7 runtime, but SciPy is not.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iM54xnkAilIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Frameworks on GAE

[Drake]

In an argument we often polarize beyond what is entirely true.

 

Libraries. More than Frameworks are "OK" 
Some Frameworks are closer to libraries, and some Libraries are closer to
frameworks.

You should always look at what your frame work gets you and what it costs
you.  Putting a timer before your load of the framework is a good way to
make that judgment.

 

Sometimes it means stripping out everything you aren't using from a frame
work.

 

Other times it means using the framework as a stop gap until you can replace
it with your own code.

 

And sometimes it means doing the optimizations yourself.  



I am working on a project that uses the Python Natural Language Tool Kit.
That doesn't even Fit in an F4 instance.  Just loading the NLTK times out
and never does anything. So I stripped it down to just the parts I needed,
and then optimized how it uses memory.  That is a Sucky way to spend 2
weeks. BUT I didn't have to write my own Sentence and parts of speech parser
which would have been a sucky way to spend 9 months. 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of glimmung
Sent: Sunday, July 22, 2012 2:33 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Frameworks on GAE

 

Hi All,

 

I've been reading, initially with amusement but more recently with concern,
the "dialogue" (for want of a better word) between Brandom Wirtz and Jeff
Schnitzer re. startup time/optimisation. Brandom has now made the following
very strong statement: "NO FRAMEWORKS. NONE. Deal with it."

 

This leads me to ask the Google team for their position on this: Is it your
position that GAE is an unsuitable platform for framework-driven apps?

 

I'm using a framework, and trust the framework's authors to optimise their
part of the piece as much as possible, but I'm paid to solve business
problems, and am not about to dive into that timesink of esoterica. It's
outside my skill-set, and properly so in my view. I've only really tinkered
with GAE so far, and this is putting me off investing more time in what is
starting to look like a risky platform for me.

 

So Google peeps, if I want to write B2B apps using a framework, am I your
market or not? 

 

Or is it your position that the low-level optimisation to balance start-up
time and hosting cost will always be required?

 

 

--

 

Cheers,

 

PhilK

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/sRtfq4gGlwYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

  And your
> absurd proposal that all 326 of my url endpoints should be separate
> applications...  Just.  Wow.

If you had actually read what I posted I said that you should group by task
type, and used class so that you had many smaller apps, that were optimized
for like tasks and minimizing classes. Which is just good design because
then you can run fewer instances likely move to f1s or f2s and better
utilize in instance memory, hitting the soft memory limits less.

And no it was not a constructive conversation it was a Gripe fest.

If you want it to be constructive, post your code and I'll whack 30% off
your start time just to get you to stop griping about how it is not possible
to speed up start times because Google sucks.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: numpy and scipy question

[Drake]

I am building an API that does much of what NLTK does.  Which parts are you
looking to use? Likely making API calls to my optimized version would cost
less than actually running the full version, (which includes a lot of things
that won't work on GAE  (like the cool drawing of the sentence diagram)

If the parts you need don't take too much effort to add, I'll put them on my
teams task list for the API release in 2 weeks.

 

-Brandon

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 7:04 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

The application I develop uses libraries such as gensim and NLTK, which
depend on SciPy. Currently my plan is to run it on Heroku, but I would much
prefer to use appengine.

 

Hans

On Sun, Jul 22, 2012 at 11:24 AM, Drake  wrote:

I don't know then. I thought NumPy was all C. 

Any how I use Numpy but not SciPy. Apparently Google caters to my every whim
and so they gave me what I needed.  J

Depending what you need Matrix.py, and Euclid.py get you many of the most
common things you would use Numpy and Scipy for.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 1:34 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

NumPy is also Fortran. 

 

But my confusion is that if SciPy will not be supported, then why support
NumPy? NumPy and SciPy are hardly ever used separately.

 

Hans

Op zondag 22 juli 2012 09:11:00 UTC+2 schreef Brandon Wirtz het volgende:

SciPy is mostly in Fortran. So I don't give you good odds.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 12:05 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: numpy and scipy question

 

Hi Anand,

 

Are there plans to also support SciPy in the future?

 

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het
volgende:

NumPy is available on the Python 2.7 runtime, but SciPy is not.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iM54xnkAilIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine%2bunsubscr...@googlegroups.com> .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Naked https (SSL) domain?

[Drake]

Talk to the Apps For Domains guys.  If you get a good answer share it with
the list. Currently I think this is the biggest limitation of GAE/GAFD.  You
pretty much have to have a redirect server out in the world that 301s all
your naked requests to the www domain. 

 

That is a sucky solution. (it is also sub optimal if you want short domain
names) 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Thomas Wiradikusuma
Sent: Sunday, July 22, 2012 10:13 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Naked https (SSL) domain?

 


Hi guys,

 

I'm using GAE's official SSL SNI support in my app. Opening
https://www.temanmudik.com is fine. But opening https://temanmudik.com
(naked domain) gives warning (in Chrome) that essentially says, "Hey, the
cert is not for temanmudik.com, it's Google's". Non-SSL
(http://temanmudik.com) properly redirects to www. My appid is "neytap".

 

So I got creative, I logged in to enom (from Google Apps) then removed all A
mappings (@ -> google IPs) and replaced them with one URL Redirect (@ ->
http://www.temanmudik.com). Non-SSL naked domain is still working, but now
Chrome gives me "This webpage is not available. Google Chrome's connection
attempt to temanmudik.com was rejected. The website may be down, or your
network may not be properly configured." when I try to access
https://temanmudik.com.

 

Anyone know how to properly redirect naked https domain in GAE? I notice
there is similar question in this forum
(https://groups.google.com/forum/?fromgroups#!newtopic/google-appengine/goog
le-appengine/A-8I1X9vouk), but it's 2009 and uses unofficial SSL support.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/BtUQJeGPVCcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I have a similar theory.

We have 9 Million instances, Let's look busy.  :-)

I think that they assign instances to somebody at all times, and if that
somebody suddenly needs one it raises the QoS.  Doesn't cost anything to
idle instances that weren't doing anything. And you make the customer happy
if you avoid a warm up.



> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of hyperflame
> Sent: Sunday, July 22, 2012 10:28 AM
> To: Google App Engine
> Subject: [google-appengine] Re: Startup time exceeded...on F4?!
>
> On Jul 22, 5:12 am, Aleksei Rovenski 
> wrote:
> > Regarding min idle instances, I must admit there is something strange
> > going on in my Instance tab in Dashboard.
> > I have settings like this: idle instances min=auto, max=1, pending
> > queue min and max=15s. I have some working instances that I get
> > charged for and then few more that are anyway created by google just
> > waiting out there for hours and even days and I do not seem to pay for
> > them. If I kill them, they are coming back anyway. This looks nice
> > indeed, but could someone explain me what is behind that. I'm worried
> > because it may be a drug that google will suddenly stop providing me
> > :)
>
> I get that as well. My theory is that at non-peak times, the Google
scheduler
> opens up extra instances to be ready for spikes.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: numpy and scipy question

[Drake]

I think we have everything you need.  We are working on some speed
improvements because unfortunately the 60s limit doesn't let you work with
very big documents.

Do you know how many words at a time you need to put through?

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 3:51 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

I am mainly using the classify and the cluster packages.

 

Thanks for considering this to be included.

 

Hans 

Op zondag 22 juli 2012 19:18:25 UTC+2 schreef Brandon Wirtz het volgende:

I am building an API that does much of what NLTK does.  Which parts are you
looking to use? Likely making API calls to my optimized version would cost
less than actually running the full version, (which includes a lot of things
that won't work on GAE  (like the cool drawing of the sentence diagram)

If the parts you need don't take too much effort to add, I'll put them on my
teams task list for the API release in 2 weeks.

 

-Brandon

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 7:04 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

The application I develop uses libraries such as gensim and NLTK, which
depend on SciPy. Currently my plan is to run it on Heroku, but I would much
prefer to use appengine.

 

Hans

On Sun, Jul 22, 2012 at 11:24 AM, Drake  wrote:

I don't know then. I thought NumPy was all C. 

Any how I use Numpy but not SciPy. Apparently Google caters to my every whim
and so they gave me what I needed.  J

Depending what you need Matrix.py, and Euclid.py get you many of the most
common things you would use Numpy and Scipy for.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 1:34 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: numpy and scipy question

 

NumPy is also Fortran. 

 

But my confusion is that if SciPy will not be supported, then why support
NumPy? NumPy and SciPy are hardly ever used separately.

 

Hans

Op zondag 22 juli 2012 09:11:00 UTC+2 schreef Brandon Wirtz het volgende:

SciPy is mostly in Fortran. So I don't give you good odds.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Hans Then
Sent: Sunday, July 22, 2012 12:05 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: numpy and scipy question

 

Hi Anand,

 

Are there plans to also support SciPy in the future?

 

Hans Then

Op woensdag 25 januari 2012 00:34:53 UTC+1 schreef Anand Mistry het
volgende:

NumPy is available on the Python 2.7 runtime, but SciPy is not.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/-WRRRE6BhD8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iM54xnkAilIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine%2bunsubscr...@googlegroups.com> .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/ESn4rOHx3a8J.
To post to this group, send email to google-appengine@goo

RE: [google-appengine] Re: Frameworks on GAE

[Drake]

Thomas,

 

How do you access the datastore?

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

It's not a who's a better guru.  I'm sure Jeff writes better Java than I do.
But "This aint your grandmother's java".   I participate in a HipHop (php)
discussion group too, and people will say "Well XY and Z aren't possible
with out ZEND"  well, no they are just hard without it.

If you are running Wordpress for your personal blog you can do PHP on
DreamHost. If you are Facebook you use HipHop. Writing a small App with no
performance requirements you can use PHP and Zend. But at scale you need
that "Simpler" version of the language.

 

Even Python on AppEngine isn't "Python" you don't get all the C stuff. The
Fortran stuff. 

 

You are Accessing Google API's via a language. You can pick your language,
Python, Java, Go, but you are writing in "AppEngine" it's there world we
just play in it.

The good news, if you write to those rules and wrap the API's in
Class/functions of your own (which you need to make things work in the
sandbox), your code will be scalable portable, and happy on anyone else's
platform.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Paul v
Sent: Monday, July 23, 2012 8:38 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

Well I love the who's-a-better-guru argument.  I've learned a ton reading
this thread, I hope they keep going at it. 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/bXo-VqpW7LQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

On GAE it's not just the startup.. Well it is, but for more reasons. You
have less memory too. So someone's really optimized framework might be
AMAZING for CPU optimizations, but the cost of adding it from a memory
standpoint causes you to hit the soft limit instance death more often. Which
causes you to cold start A LOT more because the scheduler didn't spin the
instance, the instance ceased to exist.

 

And frankly, this discussion hinges entirely on instance cold start issues.
If you have enough money to keep spare idle instances sitting around, or
your traffic is steady and constant (as opposed to spiky and unpredictable)
you really won't hit this problem very much.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

August 25th I have an app launch. It is no small thing. My CDN is a Toy in
the view you have of it.  But the internal version is designed to make all
the other stuff work. It is a load balanacing url handling, code
modularizing tool.  But you keep slamming my CDN because you like those
other "CDN guys" but it is just an enabling technology for everything else I
do.

Because of that tool I can do www.myappp.com/user/12354 as an app in python,
and www.myapp.com/products/12345?view=json in Java, and
www.myapp.com/search/frogs as Go.  And if I really want, /checkout/ can go
to PHP on Dreamhost.  That gives me infinite more control.  It also lets me
share databases because java.myapp.appspot.com can share data with
python.myapp.appspot.com  . So that comment that you can't convert your code
to Python... You don't have to convert all of it, or all of it all at once.

Maybe my bias is that because without such tech you have to do web.xml stuff
or that it is hard to modularize your app that I'm so jaded by it's
awesomeness that I can't see past, but...

Typically when a company asks me to refactor code the first thing I do, is
break their code in to groups that share classes/imports/frameworks and
create 3-4 apps from those, and then use a "load balancer" or a "front end
proxy" or a "Templated API wrapper" to make that app faster and more
scalable.

Step two is convert as much as possible to lowlevel api.

Step 3 is to strip everything out of every framework that they aren't using.

Step 4 is to create flatter versions of their data to make queries simpler,
faster, and more friendly to the lowlevel API

Step 5 is to prioritize writes and reads so that things can be delayed if
they aren't for realtime data

Step 6 is to optimize start ups using serialization of configuration
strings, and other data. Pushing any file streams to Memcache

Step 7 is to create a task database to create precaclulated versions of
common requests, and to handle cron and steady state tasks.

Step 8 is to make determinations about lazy loading. (I try hard to not lazy
load because it makes your app harder to profile, because if you lazy load a
large frame work you may over run the soft memory limit. If you do
everything up front then you know what your steady state memory usage is)

Step 9 is to self manage parts of the memory. Rather than relying on garbage
collection start throwing away memory that you aren't using.

Step 10 is to profile decision trees, to optimize order of operations.

Step 11 is to manage threading (this is often late in life because it makes
debugging hard, and the gains are often not amazing in apps with lots of
simultaneous users)

Step 12 is to strip all the debugging, logging and non-esential stuff from
the code.

Rinse and repeat.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. I
don't even promote the damn thing it helps with my old SEO clients. Our
analytics and Ad management platforms are full on business apps. We do real
database stuff. We did 1million write ops in an hour the other day. ( I can
post pictures) we do queries and reports, and we use the lowlevel API for
all that.

(we can do a Join in Lowlevel we don't even use GQL )



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Skipping the start time issue since there doesn’t appear to be a single lazy
load in the whole project….

 

And not having time to sort through the duplicate imports which are
definitely killing load time...

 

And ignoring the fact that you are doing Dotted imports which add up to most
if not the entire import which is slowing things down even more…….  (Stars
aren’t always bad because in GAE the IO is more of an issue than the compile
time)

 

And that those 3 things would likely speed load time by a lot, if not all of
my 30% goal….

 

The PolyLines String builder isn’t threaded, and profiling the app that
seems to be really frakking slow.  

 

Create 4 threads and have each take ¼ of the array.

I haven’t implemented the code but that should be about a 45% increase in
speed.

 

Am I required to actually check the changes in? or can you guys handle
threading the slowest part of the code? ( I assume I’m optimizing for low
volume and reduced latency)

 

Can I replace your Sha256 with something else? That is also eating a lot of
CPU time and I can’t really tell from the 5 minutes looking at the code why
you are doing that. Sha1 should suffice and would buy you 50% increase in
speed on those requests that use it. And as far as I can tell that looks
like all authenticated requests?  It also doesn’t look like you are caching
your HashCalculations? Based on privacy and such you may not want to
memcache those… but at least store some number of them in instance memory.

 

 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Jon Stevens
Sent: Monday, July 23, 2012 7:35 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

On Monday, July 23, 2012 6:37:15 PM UTC-7, Brandon Wirtz wrote:

CDN isn't my primary. I mostly do other things. You seem stuck on my CDN. I 
don't even promote the damn thing it helps with my old SEO clients. Our 
analytics and Ad management platforms are full on business apps. We do real 
database stuff. We did 1million write ops in an hour the other day. ( I can 
post pictures) we do queries and reports, and we use the lowlevel API for 
all that. 

(we can do a Join in Lowlevel we don't even use GQL ) 

 

Unfortunately, both Jeff and I have prior experience with working with
people just like Brandon.

 

All talk (mostly gibberish), but no listen and no do.

 

Brandon, your next response on this impossibly long thread should be a link
to a pull request showing how you've made motomapia start up 30% faster
while playing by the rules Jeff sent out. Nothing more, nothing less. Any
other response will be proof that you are all bogus talk and no action.

 

It's time to put up or shut up.

 

jon

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/AwjWPmk8szMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

PS, sorry I was driving home between when you through down the gauntlet, and
10 minutes before I posted this. And I had to pee, which slowed me down.

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Really?

You don't think that doing lazy class loads would speed up your startup?

Did you try?

Did you benchmark Dotted Imports vs full import, or "less dotted"?  I
knocked 1s of the startup just by * importing objectify, that you took all
of but did so in 6 places.

No, you never try anything.

I think I made it clear that threading wouldn't change the startup, it would
speed up the app. And since your issue was startup plus response time and it
appears that can take 6-8 seconds and tests at 2-5 after threading that
would appear that would make your QoS much better, and based on the load you
were describing that would seemingly reduce your number of starts, and need
for more instances.

You aren't doing hash caching. That's a huge, easy speed up. Even if you
don't "downgrade" to sha1, that will make a huge difference in your need to
spin up more instances.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Also None of your references are weak? Do you just hate garbage collectors?
That won't help much with startup, but again it would reduce your need to
startup as often.

You do a lot of String + String..  use stringbuffer instead

You do a lot of .Trim when you should use Vector

And yes in a few spots that would change your warm up. (not as much as it
would drop your memory foot print)


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I blew up or created some circular reference because it's late and I had a
lousy week... I'm not familiar enough with the code, but you have several
objects whose creation should be moved to the top of your code. And a few
places you should swap to mutable returns.

The mutable returns won't speed up startup, but will drop your memory usage
significantly.

Though it looks like a crap ton of memory is being allocated, more than
should be needed, for what I can tell is going on.

Do your logs have a lot of Soft Memory limit errors?   There is what I would
call a bug in GAE where if your only instance is killed by the Soft memory
limit that the replacement instance loads really slow. I think because the
pending request queue is transferred to the new instance.

I think your biggest win would be in lazy creating the objects and classes
that you don't use much.

Next is
Threading your large loops. Caching your Hashes. Changing your trims to
vectors, and your Concats with + to use stringbuffer. Weak the stuff that is
temporary variables so they don't get clung to.





-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Odd. I get different results than you do.

Quite different actually.

.* in the sandbox is slower (by a lot)

.* on f4 is faster

Also, I think you are very wrong about dynamically/lazily loading classes.
Since my version warms up in 2.5s on an F2. (though I have an initialization
missing somewhere cause when I do places I get an error about an object)







-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I like that you use +'s to concat strings, it shows a real lack of
experience doing optimizations since that is the very first thing on every
list.

> System.out.println(new Date() + ":" + new
Random().nextInt());

At least most of your code uses objects correctly I didn't find any
instances of places you did a type conversion functionally. That is usually
one of the lowest hanging fruits, because it creates additional objects, and
is a memory hog.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

> That makes absolutely no sense.  The app doesn't cache data in instances;
> there's utterly no point to using weak references.

Your app runs for multiple seconds, you get garbage collection in that time
and since you also feel no need to destroy objects you aren't using... this
would be a lazy way to deal with it.

> "this" + "that" is compiler syntactic sugar for using a StringBuilder to
assemble
> the string.  There is no difference between the two.  It's probable that I
could
> do the same trick I did with imports and create with two classes that
> generate identical bytecodes, but figuring out the exact pattern is more
> effort than I care to spend at the moment.

No, the intermediary objects are different and you have a much larger memory
foot print using loops with + than stringbuilder
For Print "this" + "that" you are essentially correct, but for 1000+ loops
the difference is huge.  Typically for small stuff it isn't worth messing
with but you have some long loops of this.


>
> Also, never use StringBuffer; it's a legacy class from JDK 1.0 which
> unnecessarily synchronizes method calls.  StringBuilder is the
replacement.
>

Yes I misspoke. I have been up for 48 hours and would be sleeping except
someone said I couldn't do hello world in 1.5 seconds and I missed it by
100ms on an F1, so I didn't want to be slow getting response to this.


> > You do a lot of .Trim when you should use Vector
>
> I'm genuinely afraid to ask, but... huh?  Vector? (!!!)

Vector resizes an object without creating an intermediary or placeholder
object, so there is nothing for garbage collection to deal with, and you
don't have to create a new object which gets renamed to the old object.
(which is what trim does)

Again a Pain in the ass for doing a trim on something once, but when you
have it in a for loop it adds up.





-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Oh, and you are right the biggest slow down is grabbing data via HTTP.
(usually is)
But that doesn't seem to be hurting concurrency.

The obvious question.
Why aren't you mirroring the site so that you have your own fast copy?



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I can't get over how much memory this thing uses before it does something.
Like just warming up I'm at 248M of memory.

On an F2 I can do like 5 places and it soft memories and restarts.

On an F4 I do about 350

How many requests are you seeing.

Why do you hate garbage collection so much?

I don't live in objectify, is it putting everything in an instance cache?
Each request seems to add about 1.5Megs to the used memory.

I can't see any difference (300ms) between load speed on an f2 and an F4
That usually means you are object creation bound, but you aren't front
loading the Object creation, so that seems unlikely. (though you aren't
doing lazy classes so maybe you are creating every object in your entire
code at startup?)

Tell me again why you are against Lazy Classes. Like I am the 3 year old you
seem to think I am?



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

What does your usage profile look like?  It looks like if I kill off all the
places code I can get this to 86 megs after warm up.

If you put places on a backend, it looks like you could serve this off of an
always one F1 backend, and alwasys on F1 frontend. And your warmups would be
about 2.5 seconds, and you would have a very minor code tweak. Like you
would access places like it was an API.  Places should scale well since it
isn't CPU dense.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

That's on your stock code. No changes.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

Ok so it works its way down to the same levels you are seeing it takes 3-5
minutes.

That's a first for me as well.

In your production can you hit shutdown an app, hit it and see what the
memory is 30 seconds after start?



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

I am crashing now.

Splitting out the app I can make it load much faster. That seems to work.
You would have two "apps" one living only on a backend, and accessed via an
"api" (places)  This gets your warm up way down, and shouldn't cost much
more money.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

On 5 hours now of sleep so I can form sentences:

 

>Your micro-optimizing suggestions for Java ...I think all Java guys are
like ROFL now.

>Star visa Full Imports, String Concatenations in this example visa
StringBuilder



Y'all'd be right if this was 100 lines of code in Java 101.

 

What you aren't looking at is that what is slow in this code set is object
creation and object recycling.

 

You are right that

 

Import Io.*

 

And 

 

Import Io.Blah

Import Io.Blah2

 

Results in the same code,

 

What doesn't result in the same code is doing import Io.Blah in classa and
import io.blah2 in classb and io.blah3 in classC because it changes the
order and timing of the Object and imported Class Creation.

 

+ doesn't matter if you do it once, but at one point this code is doing up
to 15k +'s in a loop, and + doesn't do efficient recycling of
temporary/intermediary objects.

 

So laugh all you want, but there are serious gains in profiling the code and
seeing where object and memory manipulation are sub optimal, especially when
you have long loops.

 

The problem I see with you guys is that you don't test, "REAL" application
you test micro bits of code and don't look at the impact or history of how
the code got to where it is.  You say "it doesn't matter in 4 lines of
codes" but you don't have "THOSE" 4 lines of codes in your app, you have 4
lines intermingled in with 80 other lines that change what is going on.

 

There are times when just changing the order of your Imports can add or
remove 200ms to your app.

 

As to architecture changes. No one liked those. Don't calculate hashes if
you are already in a session, use memcache and instance cache. Thread really
long loops.

 

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

With a shower so the sentences might be a little better.

Lazy Loading classes is a variation on the same Object creation stuff.

 

On warm up all you are doing is initializing Objects, and creating classes.
If you lazy load those, whole branches of your code won’t do that until you
need to use those things.

 

As a result an “Empty” warm up can go from 15s to 2. your first request of
type X will be 4s, and of type y will be 4s and Z will be 4s until you have
paid 20% more than what the non-lazy version would be, but the advantage is
that a “cold” request will be 2+4s not 15s

 

This is especially useful when you have something like the “places” in this
sample code that needs a whole bunch of objects that aren’t used anywhere
else. And uses libraries no other part of the code shares.

 

You aren’t making the code more efficient, and it sucks because you have to
know what dependencies each bit of your code has or things explode (much
like when you do threading) but when your goal is to reduce the maximum
suckage for the end user and instead spread the suckage around. (like the
walmart check out analogy) Lazy Object and Class creation can be a big win. 

 

Lastly, there are some really hierarchical objects in this code, which is
great for Object oriented code writing and is easier on the dev, but I think
that functionalizing some of this would speed up the code base pretty
significantly as Objects with like 4 tiers in their hierarchy don’t store
efficiently. 

 

If you do need those, create them with the hierarchy early rather than
attaching to them later, because you get an intermediary copy of the old
object in creating the new object.

 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Tuesday, July 24, 2012 5:25 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

You should really focus an general architecture of the app for
optimizing...havn't looked into the provided example.

Your micro-optimizing suggestions for Java ...I think all Java guys are like
ROFL now.

Star visa Full Imports, String Concatenations in this example visa
StringBuilder

Really, if I carefully craft line numbers or embed no debug info in classes
I can generate byte-identical class files with either import and string
style.
Import statements are not stored at all in Java class code and this
string+string in one expression automatically use the StringBuilder pattern.


you can lazy load classes - I just fear you mean something different ;)
and it's the same problem like separating end points into micro apps -
complex business code doesn't work that way - you have to load all that
stuff anyway.




Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz:

I like that you use +'s to concat strings, it shows a real lack of 
experience doing optimizations since that is the very first thing on every 
list. 

> System.out.println(new Date() + ":" + new 
Random().nextInt()); 

At least most of your code uses objects correctly I didn't find any 
instances of places you did a type conversion functionally. That is usually 
one of the lowest hanging fruits, because it creates additional objects, and

is a memory hog. 




-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

>the same problem like separating end points into micro apps - complex
business code doesn't work that way - you have to load all that stuff
anyway.

First, but not all at once.  Which I addressed in Lazy Load, but since you
are still attacking the micro-app / API model

 

This sample app has a “Places” that is the slowest part of the code, and
loads things nothing else uses. Spinning that off to an API handled by a
backend would make a HUGE difference. Any functionalized code can be
converted to standalone apps, and the overhead is that you have a wrapper,
but the advantage is that you have huge memory and caching advantages.  So
any very slow memory intensive function can be converted to an API
especially if it doesn’t need to modify globals. If it does need to modify
globals you can do that on return.

 

Say for example you have an imaginary app that does business stuff.

Selling stuff and searching stuff.

 

Searching works better the more in instance caching you can do.  You might
want an F4 just for the memory.  And because of the shared memory 128
+128+128+128 doesn’t have the cache hit ratio of 1 512.  Because of that an
F4 with 100% of its available memory dedicated to caching search info your
single purpose F4 could serve as many search requests as 8 F1s. but even
when that isn’t the case if your “Idle” is 78 megs, an F2 will still idle at
78, but instead of having 50 megs for cache will have 178.   Even better if
that is the only place you need a wrapper for the complex queries to the
datastore this is the only place you need Factory or Objectify.

 

The selling stuff app is likely the only place you need Crypto libraries and
because you want to cache sessions again having more memory in instance
dedicated to the task of session management increases your cache hits by
orders of magnitude.

 

In both cases reducing the number of imports and the frame works required
for both of these “microapps” is going to reduce Cold start time.

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Tuesday, July 24, 2012 5:25 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Startup time exceeded...on F4?!

 

You should really focus an general architecture of the app for
optimizing...havn't looked into the provided example.

Your micro-optimizing suggestions for Java ...I think all Java guys are like
ROFL now.

Star visa Full Imports, String Concatenations in this example visa
StringBuilder

Really, if I carefully craft line numbers or embed no debug info in classes
I can generate byte-identical class files with either import and string
style.
Import statements are not stored at all in Java class code and this
string+string in one expression automatically use the StringBuilder pattern.


you can lazy load classes - I just fear you mean something different ;)
and it's the same problem like separating end points into micro apps -
complex business code doesn't work that way - you have to load all that
stuff anyway.




Am Dienstag, 24. Juli 2012 09:00:15 UTC+2 schrieb Brandon Wirtz:

I like that you use +'s to concat strings, it shows a real lack of 
experience doing optimizations since that is the very first thing on every 
list. 

> System.out.println(new Date() + ":" + new 
Random().nextInt()); 

At least most of your code uses objects correctly I didn't find any 
instances of places you did a type conversion functionally. That is usually 
one of the lowest hanging fruits, because it creates additional objects, and

is a memory hog. 




-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iYnYrW5SL2EJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Startup time exceeded...on F4?!

[Drake]

And you get a new string builder each loop. If you use string builder and
recycle it you don't have to re-create the object. Which doesn't create so
much garbage.
You have the answer and you don't understand it.



> Try compiling these two classes.  They produce *identical* class files:
>
>  File m1/Stringy.java 
> public class Stringy {
> public static void main(String[] args) {
> String foo = "def";
> System.out.println("abc" + foo);
> }
> }
>  File m2/Stringy.java 
> public class Stringy {
> public static void main(String[] args) {
> String foo = "def";
> System.out.println(new
> StringBuilder().append("abc").append(foo).toString());
> }
> }


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Always Warm Pick your QoS mix your code base auto-fix your HTML

[Drake]

Fine I'm tired of people who want to optimize the scheduler for low traffic
users, so I fixed it. (And made it better for High Volume users with Mixed
task latencies)

 

The CDN in a box platform is now available as a Java or a Python frontend,
and has support for managing backend instances.

 

The Front end spins up in 750ms on python and 1.8s on Java, and will track
the request queue which it passes to backend instances which it creates
based on load and job type.

 

Each endpoint has an expected time to completion, and can be prioritized to
maximize the performance of your Queue by putting small latency request
ahead of high latency requests.

 

I don't want to support you if you aren't serious about needing this, so the
pricing is $3500 set up and $500 a month, and you must be in the Google
Premiere support tier.

 

You also get the caching, HTML fixing, HTML Speed Optimizations that are
built in to our premier CDN product.  

 

Lastly because you can do Queue reordering/prioritization and backend
spin-up and sizing by task we expect that you can save about 30% on your
operational costs, while improving your average QoS.  These savings from our
ability to reorder tasks right up until they have started to run, so in
testing we have fewer idle CPU cycles. And because we have more ability to
route tasks to servers handling similar tasks we get more efficient in
instance memory usage.

 

Downsides. You lose some elasticity.  We are an intermediary, and if you get
a spike of 100k users we have to spin up front end proxies and then spin up
backends so if you need 10 new frontends and 50 new backends you will have
to wait for a little of both to get everything in to the queue, and in
testing we have seen that can be problematic, but we can't really tell how
much more so than the existing schedule which suffers from lots of cold
startups in this scenario. Also because of the increased efficiencies that
come from this architecture you may be able to handle more requests with
fewer resources so elasticity may not be as needed.

 

 

I strongly suggest using the Python version of this app, because the speed
is the best in most cases, and since your app can run whatever you like
there is little downside to this approach.  The Java version of this app has
less over all overhead and uses fewer frontend hours, and maybe more
performant for users averaging less than 1000 requests per minute.

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Differentiated service with Google App Engine

[Drake]

If you are on a mult-threaded platform. you can achieve this by just adding
a Wait to the people you want to run slower. That will free CPU for the
other request.

 

When you go over 8 requests per instance this will have less effect, but
generally how it works is 

User 1 Process start:

User 1 runs until you hit an API

User 2 Starts:

User 2 runs until you hit an API

User 1 Resumes:

User 1 runs until an API

User 2 resumes:

User 2 runs until an API

User 1 finishes

User 2 finishes

 

If you insert Pause/wait/DoNothing   Spacers in your code that are condition
on the QoS of the user then the non-delayed user will get priority. 
I do a similar trick to make certain that low priority requests don't cause
high priority requests that are near the timelimit from finishing.

 

Better to know that you can serve 100 small tasks and only hit 75% CPU than
to have a big task drop because small tasks ate all the CPU.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of David Hardwick
Sent: Thursday, July 26, 2012 8:17 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: Differentiated service with Google App
Engine

 

If you put in a feature request to have more than one app access the same
datastore natively, then I will surely start that issue!

On Thursday, July 26, 2012 3:11:48 PM UTC-4, rerngvit yanggratoke wrote:

Hello,

   Anyone have an idea on how to achieve a differentiated service with
Google App Engine? I have two types of users. One is premium while another
one is free user. I would like to set different minimum latencies for firing
up new GAE instances between the two. In other words, I am willing to pay
extra for the traffic generated from the premium users. Nonetheless, for the
free users, I would prefer to pay at minimum or within the daily free quota
provided by Google. In my case, I expect that the free users are able to
tolerate higher latency than the premium ones. I believe that this case
should be quite general and apply to many people.

   Assuming that I can identify types of users by the URLs they use or
via other mechanisms. The only way I could think of is creating two
separated applications and letting the premium users access the one that
enables billing, while free users access the one that doesn't. However, my
free and premium users need to share the datastore because they are also
have to communicate within the application. Unfortunately, it seems that
sharing a datastore across applications is not possible at the moment (feel
free to correct me if I am wrong). 

 

Any ideas or suggestions?

 

Cheers,

Rerngvit

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/cs1aYijZARwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Google App Engine, rogue crawlers, and PageSpeed Insights

[Drake]

And then when Google Spam team bot shows up you would be delisted... That
would Rock...

Don't ever serve anything to a bot other than the content, or a permission
denied. (maybe a busy signal)


If you have an htaccess try:  You can add Appengine, or anyother bot pretty
easy using this.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craft...@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* - [F,L]

> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jeff Schnitzer
> Sent: Thursday, July 26, 2012 6:41 PM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Google App Engine, rogue crawlers, and
> PageSpeed Insights
>
> It would have to be by something at "Layer 7" that understands HTTP.
> What web server/technology are you using?  With apache you can do it with
> mod_rewrite.
>
> Blocking IP addresses is really a clumsy way to do it anyways since GAE
> urlfetch changes IP ranges periodically.
>
> If you really don't like the scraper, I suggest an alternative to simply
blocking
> them.  That'll probably just put a bunch of errors in their logs and alert
them
> to the problem.  More fun is to silently replace the content with
something
> nefarious.  The best option would probably content that Googlebot will
> detect as being spammy/low quality so it kills their search ranking.
>
> Jeff
>
> On 

RE: [google-appengine] Re: Differentiated service with Google App Engine

[Drake]

This shouldn't increase your number of instances in most cases. The
threading will mean that you prioritize the other threads by putting wait
states in for the less important ones. There is some overhead, but not much.

 

And if you only insert small waits through out the app, you just give the
other threads a chance to run 10ms becomes a minimum time to wait, and if
you put it each 3rd of the way through your code, you add 30ms minimum to a
low priority user, and you guarantee that your high priority user gets
opportunities to barrel through.

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Differentiated service with Google App Engine

[Drake]

It works, it is painfully slow.  And it makes Race States really dangerous.
It is really meant for local sandboxing with the real datastore.
>
> Correct me if I'm on the wrong track, but I thought two applications could
> share the same datastore via the Remote API:
> https://developers.google.com/appengine/docs/java/tools/remoteapi



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Boat Load of Questions (which might turn in to feature requests)

[Drake]

Even with threading I don't seem to be able to get an F8 backend to do more
than 1.9 CPU Seconds per second. Which seems to be the same number I get on
an F4 Backend.

Am I doing something wrong? Or is the Python Gil not capable of using all
the CPU on this system? Or is an F8 CPU Second twice as much as an F4 and
this is as expected.  If the latter why is the app only 15% faster on an F8
(I know you can't answer that with out the code)


If the gil is limited case why not allow backends to have more than one
request at a time?  If the Gil is not limited still why can I only have 1
request per backend?

 

Why can't I configure how many requests  Frontends handle? F1's serving 8
requests is silly. F4's handling 8 requests is often equally silly in the
opposite direction.

Why can't I configure how many requests a backend can handle? Or have a back
end act like a front end and handle 8?

 

Why can't I specify a specific version of an App in Apps For domains?  It
would be very useful to have  User.Somedomian.com be a version of an app
and Affiliates.somedomian.com be an app and they would share a database and
nothing else.

 

Public backends can run longer than 60s but only handle one request.  But
they 503 if the request before them in line takes more than 10s to complete.
Why is this not configurable?  Especially since dynamic Public frontends
seem to only spin up a new instances if there are 8 requests in the queue.
This means you get a lot of 503's 

 

Defer task buckets don't have a mechanism for manual token addition.  I'd
like this. It would let me read the queue and decide how many backends to
spin up, and when tasks completed in 1 to 500s I would put a token in the
bucket. (that's not a question so much as a Does this make sense?)

Memcache gets slow sometimes. Really slow. Slower than datastore. Is there a
"memcache health status" that we can hit to identify when we should stop
using it for a while? MasterSlave had a Readonly we could detect and change
behavior. I can't find any such thing for HR.

 

Requests through edge cache get edge cached. Why do outgoing requests not do
the same?  There seems to be a lot of people "scraping" content via
Appengine. Why are those requests not "edge" cached so that a. it would be
harder to build appengine DOS attacks. B. fetches would take less time.

 

Because I am threaded and use 200% more CPU seconds than Realtime seconds I
suspect I'm a bad neighbor to share an instance with.  Is this true? Am I
clobbering the neighbor?

 

Do we always have to go through Apps For Domains? How much money do I have
to send you to never have to talk to their tech support again? Similar
question Naked Domians. I want one. Building a URL Shortener that has to
have www. On the front is breaking my heart.

 


Brandon Wirtz 
Stremor.com Chief Technology Officer
BlackWaterOps: President / Lead Mercenary 

Description: http://www.linkedin.com/img/signature/bg_slate_385x42.jpg



Work: 510-992-6548 
Toll Free: 866-400-4536 

IM: drak...@gmail.com (Google Talk) 
Skype: drakegreene 
YouTube:   BlackWaterOpsDotCom 



  BlackWater Ops 

  Cloud On A String Mastermind Group





 

 

 

 

 

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

<>

RE: [google-appengine] Newbie

[Drake]

Convince a VC it is a good idea. Raise $500k. Pay someone.

 

B.  Using driver's license bar code for DB lookups is illegal in all states
without written consent of the licensee.

 

C.  While I'm usually all for telling people to build on appengine. Newbie
is someone who has installed the SDK and Written "Hello World"  your comment
is "I want to stand on my soap box". Come back when you have $500k to build
something, or have the SDK installed and have gone through the online
tutorials.

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of jonathan drake
Sent: Saturday, July 28, 2012 6:33 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Newbie

 

I have no idea what I'm doing, but a new app must be created for the safety
of the community!  No one else is taking preventive actions against
dangerous individuals and registered sex offenders, so I feel that I must.
I would like to create an app for smart phones where you can either
scan/photograph the bar code on the back of a Driver's liscense or type in
their full name and search both the Maryland court cases (for major crimes)
and the federal registered sex offender site.  Both of these databases are
open to the public, but neither are accessible through phone.  After a
recent incident in my community, I feel like I need to make that happen.
Any suggestions are much appreciated. 

 

Thank you,

Jonathan

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/OruiXyz3o_MJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] How can I block curl requests

[Drake]

You can check the user agent of the request. Is the Use agent the same?

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Kate
Sent: Wednesday, August 01, 2012 2:15 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] How can I block curl requests

 


Yes I can but there's a limit to the number - 100 I think, and there are
tens of thousands of them. I guess I'll just have to move from GAE as there
doesn't seem any way of doing this and I can't be paying for these requests.

 


On Wednesday, August 1, 2012 4:51:04 PM UTC-4, rerngvit yanggratoke wrote:

I think you can blacklist those IPs. Have a look in
https://developers.google.com/appengine/docs/java/config/dos#About_dos_xml
or https://developers.google.com/appengine/docs/python/config/dos.

 

 

On Wed, Aug 1, 2012 at 10:43 PM, Kate  wrote:

I am getting tens of thousands of curl requests - many thousands time
browser requests and I want to block them. I'm using python. They coming
from many different IPs most in Europe.  If I can't stop them I will have to
close my site or go to a new provider.

 

Thank in advance,

 

Kate

 

 

 

 

 

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/L7wDtd5rrUcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
 .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 




 

-- 
Best Regards,
Rerngvit Yanggratoke 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/X1_iYMXzpKEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

CloudFlare biting you in the ass.

It happens.  Bonus...

Cloudflare doesn't form its proxy request headers correctly. Check the
archive this isn't "undocumented" I've mentioned a lot when helping other
people with the same error.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

Not if they form their request headers properly.

CF sends IPs that are malformed, nonsensical, and non-responsive. (and often
blank)

My CDN solution says "Request on behalf of X" the right way.

Oh, and CF has already tried doing that to me. More than once.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

That's what y'all say every 9 months...

https://groups.google.com/forum/?fromgroups#!topic/google-appengine/om1DF_61
Xl4



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

Instead of whitelisting your IPs Fix your Proxy.  Google is not the only
place with these issues. Don't lie about the IP the request is on behalf of,
don't create IPs that don't exist or are in unassigned ranges. Don't try to
piggy back session headers that aren't from the same session. Google smacks
you for a reason.  Band-aiding is what causes your users to get de-listed,
de-ranked, and have very bad days.

I don't have to read minds. I can see the headers, and I have helped LOTS of
customers after CF ruined their lives.

It's still Brandon, but I put Drake on because I get so much spam from bots
scraping lists.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] How can I block curl requests

[Drake]

Yes you'd still have to check user agent, but that could be very early in
the processing, and returning a denied message will typically cause the
offender to go away.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Kyle Finley
Sent: Wednesday, August 01, 2012 4:12 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] How can I block curl requests

 

Even if she did the check in a middleware, she would still have to
handle/reject the requests by spinning up instances, though. Right?  

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/UV1fA4QJYRcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

This is a mirror of the conversation from 9 months ago and the one from 9
months before that, and the conversation on the Amazon forum, and the
Rackspace forum and the Google Webmasters forum...

https://groups.google.com/forum/#!msg/google-appengine/om1DF_61Xl4/cRtKFojKc
BMJ
http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CFgQFjAB
&url=http%3A%2F%2Fproductforums.google.com%2Fd%2Ftopic%2Fwebmasters%2FsILDRV
zPo6U&ei=38cZUMO_L4mMqwGk6oCICg&usg=AFQjCNHk6YVJTj6-hj_2NYP1OBBIuSFQXw&sig2=
dRQ9_wSJ8bDC4nzzS6IQVQ


CF does headers wrong. CF doesn't play well with Google bot. Not news to
anyone.

CF "Double wraps" its proxy some times. Don't know why.  When this happens
Google Freaks.   CF doesn't report what IP the request is made on behalf of
sometimes. Google Freaks out.  CF reports that it is proxying on behalf of a
Google Internal IP sometimes GOOGLE REALLY FREAKS OUT.

You guys can be assholes, but I'm telling you where your problem is. I told
you 2 years ago, I told you 9 months ago. I am telling you now. You can
stick your fingers in your ears, but this is not Google doing the wrong
thing, it is CF. You won't have this problem if you use Akamai's DSA/PLC
solution. Which is what I would recommend to anyone who needs international
access on GAE, or who wants to leverage enterprise class caching.  It's what
we are using for our top tier deployments.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

Here from when I was working with the guys at Google about proxying requests
when I was consulting for Akamai.

Your requests will be blocked if you cross a thresh hold of requests that
meet the following criteria.


PROXY reports request on behalf of Google Internal IP address. (or any IP
assigned to Google)  (CF does this a lot)
You can check that.

PROXY Reports Request on behalf of a session/token which has expired. (CF
does this a lot)
You can check that.

PROXY reports request on behalf of invalid IP. (octet missing)
You can check that.

PROXY reports request on behalf of invalid IP. (127 or 169  address range)
(CF does this a lot)
You can check that.

PROXY Requests SLURPbot on behalf of non-Yahoo IP Address. (CF does this a
lot)
You can check that.

PROXY made request on behalf of User Agent which is unavailable on platform
indicated in the request. (CF does this a lot)
You can check that.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

> Are you so certain of this that you're willing to grant me free license to
DOS your CDN at a time of my choosing?  I have some reasonable suspicions as
to how to >tickle Google's defenses, and shutting down your entire business
in the middle of prime-time would be a really spectacular way to prove
someone wrong on the >internet.

One of the CF guys does this everytime I bash them, Why would it be any
different that I would expect you don't play too?

The resume transfer by bytes of a file on the next to last byte of the file
was a neat trick. I was impressed.

I only have one CDN client at the moment. As I have stated before I don't
have any desire to be in the business, I got in to it because so many
clients who adopted CF had their livelihoods ruined.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

> If I read this right, you're telling me that anyone can DOS your CDN by
making
> a lot of requests with bogus User-Agent strings (which, incidentally, was
one
> of my guesses).  Forget CloudFlare for a moment.
>  You aren't the least bit concerned about that?

If your CDN is worth their salt they are blocking the people doing this "to
you".

That is a feature they advertise.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

I never wanted to be in the CDN business.  As a load balancer and traffic
router I have a number of clients, but I can make a LOT more money on the
affiliate programs and markup from other peoples products, and I hate
customer support. Because people don't seem to bother to RTFM.

> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Jon Stevens
> Sent: Wednesday, August 01, 2012 6:16 PM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] APP DOWN due to some sort of
> undocumented Google security system
>
> On Wed, Aug 1, 2012 at 6:03 PM, Drake  wrote:
> > I only have one CDN client at the moment.
>
> I think this says it all.
>
> jon
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

>whitelist / headers

Playing with the headers sucks.
CF plays with theirs a lot, so if you are in the business already, be in the
business.

If you Cache you kind of have to, because Referrer often isn't going to be
what is expected.

CF runs on other people's IPs so they have less control over being white
listed. (if this is outdated I apologize)

NGinX isn't designed to run quite the way they run their setup, and there
are some weird issues that arise from that.  This is one of the hardest
things about being a CDN, you are trying to do the best fetching possible
and minimize load, but you have to muck with stuff because if Shiela in
Australia requests a page, and Bobby in the UK requests a page you want to
serve the latter out of cache, but that means that when Bobby request the
next page his sessions, and referrer is going to look strange to the target
server.

At this point... There is almost no need for a CDN on GAE. Page Speed does
this MUCH better than CF. Almost as well as CDN In A Box does, but with
fewer "gotcha's" than either (CIAB never liked Authenticated users as much
as I wanted it to... and it has had trouble with Japanese and Korean
characters from time to time, (so does CF))

For mom and pop, use Page Speed by Google it is a one click bolt on to GAE.
For Enterprise Go with Akamai.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

[Drake]

Jeff,

Your CDN should have the conversation, it should never hit the list.  Google
is not the only place that CF bumps heads on the network edge. There have
been similar discussions on other cloud providers.

Google is doing some things with security through obscurity. Which sucks.
They are also good about telling the people they have peering arrangements
the rules.

Akamai won't bump in to this because they are whitelisted.

Limelight, same deal.

Also since a CDN should be handling DDOS detection, they should be keeping
you from encountering this.

The hoops I had to go through to get on the Azure white list... And I was a
former MSFT employee, working on making that work for a Tier 1 MSFT partner
at the time. (3 letters start with an A ends with a T)

As to DDoS-ing via the CDN... that worries me less on a daily basis than the
things I know about ways to crush a GAE instance that don't require a DDoS
because a single machine (or another appengine instance) can generate 10k
times more load than it puts on the attackers machine... That bothers me.
(and the same attack works even better on CF because the same flaw exists in
them, it is not in LimeLight, Akamai you can fix it if you set things
"wrong")




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] How can I block curl requests

[Drake]

I think you change 401 in this code to 429

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Kyle Finley
Sent: Friday, August 03, 2012 8:02 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] How can I block curl requests

 

Yes, thank you.  Do you have any thoughts on how to return error code 429?

 

On Aug 3, 2012, at 9:51 AM, Joshua Smith wrote:





There are couple problems with your snippet.

 

First, she's getting HEAD not GET requests, so you need to use different
handler.

 

Also, you aren't returning, so if you were in a GET request, it would
proceed to handle the request regardless.

 

Something more like this (untested):

 

class MainHandler(webapp.RequestHandler):

  def head(self):

self.error(401)

 

  def get(self):

if (self.request.headers['User-Agent'].startswith('curl'))

  self.error(401)

  return

# rest of the get handler

 

On Aug 3, 2012, at 10:34 AM, Kyle Finley  wrote:





Hi Joshua, 

Thank you, that's a good thought. 

 

Kate sent me some files offline, and I believe we've figured out the
problem. For the middleware to work you must be using WSGI not CGI. Someone
please correct me if I'm wrong, but I believe she would have to upgrade here
App to python27 to use it. The alternative is to do the check in the webapp
request handler:

 

def check_for_curl(self):

if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):

return self.error(401)

 

class MainHandler(webapp.RequestHandler):

def get(self):

check_for_curl(self)

# handle request

 

The problem is that webapp doesn't recognize error code 429 so we have to
use something else. Unless there's a simple way to make it write 429?

 

- Kyle

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] How can I block curl requests

[Drake]

Ah, I hadn't checked. I usually return a permission denied Error, or a Busy
Error, 503 I think (sorry not at my desk)

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Kyle Finley
Sent: Friday, August 03, 2012 8:43 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] How can I block curl requests

 





I think you change 401 in this code to 429

 

I wish it was that easy. Webapp2 uses dictionary to return the status code /
message and 429 didn't make the list.

 

Traceback (most recent call last):

  File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleApp
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/e
xt/webapp/_webapp25.py", line 701, in __call__

handler.get(*groups)

  File "/Users/finley/dev/scotch/operation_curl_block/main.py",
line 7, in get

return self.error(429)

  File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleApp
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/e
xt/webapp/_webapp25.py", line 435, in error

self.response.set_status(code)

  File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleApp
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/e
xt/webapp/_webapp25.py", line 279, in set_status

message = Response.http_status_message(code)

  File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleApp
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/e
xt/webapp/_webapp25.py", line 341, in http_status_message

raise Error('Invalid HTTP status code: %d' % code)

Error: Invalid HTTP status code: 429

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] What are the pros and cons of using Google App engine for my startup?

[Drake]

I have a video on the topic coming shortly.  Boils down to GAE is pretty
much the only PAAS, and IAAS still requires an I Manager. So you save a head
count out the get go.

 

Plus GAE is the most elastic of the Rapidly Elastic, so when
CrunchMashSlashScobleForbesModo writes about you all on the same day you
won't be down.  And when you have crickets. You aren't going broke.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Levi Campbell
Sent: Friday, August 03, 2012 2:38 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] What are the pros and cons of using Google App
engine for my startup?

 

I'm building a startup, and I'm considering GAE as the platform, however
I've been having a hard time finding information on why a startup might
consider GAE instead of the many cloud providers out there. Let me explain
what I'm working on.

 

I'm a big fan of David Allen's Getting Things Done: The Art of Stress-Free
Productivity

, and after trying several tools and online services that (claim to)
implement the GTD methodology, I couldn't find anything that I loved, so I
decided to build my own and make it available as a SaaS offering. This app
will allow users to pull in their info_crap from email, facebook, twitter
(and yes, I do have plans to add support for more social networks.), and RSS
feeds and organize it by relationship to the sender (i.e. family, work
colleague, vendor, and the like), project (i.e. planning a family
vacation.), and context (Either the when or where something should
happen.).\

 

Would GAE be a good fit for the application I'm developing? Why?

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/mncmD6uYBooJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Is "Google Apps for your Domain going away and what's the impact for GAE users?

[Drake]

Apps for your Team is a thing no one used, and is different from apps for
domains.

This likely won't affect you unless you got bamboozled a long while ago by
one of the people who was a Apps Affiliate.

And you'd remember because the thing would have ruined your life so long ago
that you'd never have moved to GAE.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Emanuele Ziglioli
Sent: Friday, August 03, 2012 5:01 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Is "Google Apps for your Domain going away and
what's the impact for GAE users?

 

I'm puzzled: http://support.google.com/a/bin/answer.py?hl=en

&answer=2669622

 

1. What's "apps for your team"? Google always used "apps for your domain",
where did that name come from

2. GAE has forced us to create a "Google Apps for your Domain" account in
order to map the app to a custom domain and for using SSL

3. Are being forced to upgrade our free "Google Apps for your Domain" to a
pay-for "Apps for Business"? Enough of paying, enough of multiple bills,
can't you just have one Google bill that covers everything?

4. Three week notice? you must be joking!

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/9Bz56OyDXPIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: What are the pros and cons of using Google App engine for my startup?

[Drake]

If you have lockin you are doing it wrong.

Wrap your Google Specific API calls so that they function like the Heavy
versions you can't/shouldn't use.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Rerngvit Yanggratoke
Sent: Saturday, August 04, 2012 1:49 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: What are the pros and cons of using
Google App engine for my startup?

 

As many people have stated about advantages of GAE for startup already, let
me provide some disadvantages. The purpose of this is not to discourage
people to use GAE but inform them about both sides of the story.

 

The first and most important issue is a vendor lock-in. GAE normally
requires you to spend significant efforts to develop and optimize
specifically for the platform to run your applications nicely and cost
effectively. For example, it is recommended to use a platform-specific
database layer like Objectify than the portable but heavy-weight one like
JPA or JDO. Another example is that it is recommended to minimize the size
of your applications to reduce the cold-start time. This also means that
very useful but heavy-weight frameworks like Spring are not going to be
comfortable with GAE unless you are willing to pay a steep price of
"reserved instances". As a result, if one day you have to move your
applications away from GAE to somewhere else, porting your applications is
not going to be an easy task. You might have to spend significant resources
for that. The possible reason for the movement might be that Google
increases the price significantly such that your business model is not
sustainable anymore. This happened before to many people. Have a look in the
highly popular forum topic
(https://groups.google.com/forum/?fromgroups#!topic/google-appengine/obfGjbI
kOTI).  Now, some people might argue by bringing up Opensource alternative
of GAE like Typhoonae (http://code.google.com/p/typhoonae/) or
AppScale(http://code.google.com/p/appscale/). Despite the fact those are
great and honorable attempts, neither of them are fully compatible with GAE.
In other words, if your applications run smoothly on GAE, they might simply
break apart on their platforms. 

 

The second issue is the development efforts and learning curve required. GAE
is designed for highly scalable applications and require you as a programmer
to think the same. It certainly requires a new mindset compared to a
traditional web applications development. Generally speaking, everything you
do should be able to implement in a fully distributed way. For instance, you
should not use a relational database like Mysql because it is a major
bottleneck in traditional web applications. (You actually can use it now
with Google Cloud SQL. However, it deteriorates the purpose of running an
application for GAE). 

 

The third issue is a sharing of fate. This is actually not just for GAE but
for any other cloud solutions. You are going to run your applications on a
shared infrastructure. You do not know what applications are run besides you
in the same IP. They might be spam or malicious software developed by
hackers. Then, the IP might be blocked for a security reason. An example of
this is the recent CloudFlare blocking issue
(https://groups.google.com/forum/#!msg/google-appengine/Q6yQ4d9ov-o/hAyrgn0x
ZQwJ).

 

All in all, the major thing to concern is picking the right tool for the
right job. GAE is suitable if your applications require a very high
scalability and you don't have or want to operate an operation team (System
administrators). Other than that, I would recommend you to find other
options. 

 

On Sat, Aug 4, 2012 at 1:03 AM, hyperflame  wrote:

It would be a good fit for most of those subjects, but keep in mind
that GAE cannot access email through IMAP or POP3 because it doesn't
support sockets, only standard url fetch. If you want to include email
in your service, you'll have to base a proxy somewhere else, such as
Rackspace, Heroku/AWS, etc.


On Aug 3, 4:37 am, Levi Campbell  wrote:
> I'm building a startup, and I'm considering GAE as the platform, however
> I've been having a hard time finding information on why a startup might
> consider GAE instead of the many cloud providers out there. Let me explain
> what I'm working on.
>
> I'm a big fan of David Allen's Getting Things Done: The Art of Stress-Free

>
Productivity,

> and after trying several tools and online services that (claim to)
> implement the GTD methodology, I couldn't find anything that I loved, so I
> decided to build my own and make it available as a SaaS offering. This app
> will allow users to pull in their info_crap from email, facebook, twitter
> (and yes, I do have plans to add support for more social networks.), and
> RSS feeds and organize it by relationship to the sender (i.e. family,
> work colleague, vendor, and the like), projec

RE: [google-appengine] Re: What are the pros and cons of using Google App engine for my startup?

[Drake]

You likely don't need "ALL" of JDO JPA's functionality.

 

I have nothing against JDO/JPA if you don't need the whole thing strip out
what you don't need. That is hard with the frame work.

 

But. If you have a

 

deletePersistent

 

equivalent in your code, name it that.

If you have a JDO/JPA function that is (Object,thing=array,target=None) for
its use, and you have a very similar function in your make your code take
the same arguments with the same name.

 

This is just good practice for making sure your code is readable.

 

 

Let's say you want to do a store.

 

public void apply()
{
Product newProduct = new Product();
newProduct.setName(pname);
newProduct.setPrice(Double.parseDouble(pprice));
newProduct.setStock(Integer.parseInt(pstock));
 
PersistenceManager manager = factory.getPersistenceManager();
 
Transaction tx = null;
tx = manager.currentTransaction();
tx.begin();
 
manager.makePersistent(newProduct);
 
tx.commit();
 
manager.close();
}

 

 

You can easily make your own class that includes makePersistent and Commit

makePersistent

entity.setProperty(newProduct);

 

commit

DatastoreService datastore = DatastoreServiceFactory

.getDatastoreService();

datastore.put(entity);

 

 

Bonus we could add to commit code so that it does a Datastore Put, AND
writes the data to memcache

 

Even if you don't replace the most complicated functions/classes you reduce
your lockin.

 

Conversely, when you are doing rapid development and proof of concept coding


 

You can start with classes that are your own that wrap the JDO/JPA, and once
you have the app working you can start stripping JDO out of those functions.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Rerngvit Yanggratoke
Sent: Saturday, August 04, 2012 2:25 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: What are the pros and cons of using
Google App engine for my startup?

 

Hello Drake,

  That is possible but would increase your application size. Then, why
not use JDO or JPA instead? It is the same lines of reason.

On Sat, Aug 4, 2012 at 11:13 AM, Drake  wrote:

If you have lockin you are doing it wrong.

Wrap your Google Specific API calls so that they function like the Heavy
versions you can't/shouldn't use.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Rerngvit Yanggratoke
Sent: Saturday, August 04, 2012 1:49 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: What are the pros and cons of using
Google App engine for my startup?

 

As many people have stated about advantages of GAE for startup already, let
me provide some disadvantages. The purpose of this is not to discourage
people to use GAE but inform them about both sides of the story.

 

The first and most important issue is a vendor lock-in. GAE normally
requires you to spend significant efforts to develop and optimize
specifically for the platform to run your applications nicely and cost
effectively. For example, it is recommended to use a platform-specific
database layer like Objectify than the portable but heavy-weight one like
JPA or JDO. Another example is that it is recommended to minimize the size
of your applications to reduce the cold-start time. This also means that
very useful but heavy-weight frameworks like Spring are not going to be
comfortable with GAE unless you are willing to pay a steep price of
"reserved instances". As a result, if one day you have to move your
applications away from GAE to somewhere else, porting your applications is
not going to be an easy task. You might have to spend significant resources
for that. The possible reason for the movement might be that Google
increases the price significantly such that your business model is not
sustainable anymore. This happened before to many people. Have a look in the
highly popular forum topic
(https://groups.google.com/forum/?fromgroups#!topic/google-appengine/obfGjbI
kOTI).  Now, some people might argue by bringing up Opensource alternative
of GAE like Typhoonae (http://code.google.com/p/typhoonae/) or
AppScale(http://code.google.com/p/appscale/). Despite the fact those are
great and honorable attempts, neither of them are fully compatible with GAE.
In other words, if your applications run smoothly on GAE, they might simply
break apart on their platforms. 

 

The second issue is the development efforts and learning curve required. GAE
is designed for highly scalable applications and require you as a programmer
to think the same. It certainly requires a new mindset compared to a
traditional web applications development. Generally speaking, everything you
do should be able to implement in a fully distributed way. For instance, you
should not use a relational database like Mysql beca

RE: [google-appengine] Re: Instances/Java go crazy

[Drake]

Well I live in unicorn land. But this was my score from pingdom for
www.xyhd.tv .

 

 



Pingdom Monthly Report


2012-07-01 - 2012-07-31


 



Overview: Average of all checks 


Uptime

Outages

Response time


99.99%

3

376 ms


Checks with downtime 


Check name 

Uptime

Downtime

Outages

Response time 


XYHD

99.99%

0h 05m 00s

3

376 ms


Checks without downtime 


Check name 

Uptime

Downtime

Outages

Response time 

This is a scheduled report from Pingdom. If you wish to no longer receive
this report you can unsubscribe by logging in to  
Pingdom Panel and updating your email report settings. 


Copyright C 2012 Pingdom AB

 

 

> -Original Message-

> From: google-appengine@googlegroups.com [mailto:google-

> appeng...@googlegroups.com] On Behalf Of Mos

> Sent: Saturday, August 04, 2012 12:00 PM

> To: google-appengine@googlegroups.com

> Subject: [google-appengine] Re: Instances/Java go crazy

> 

> Again a not very reliable GAE month:

> 

> Pingdom July

> 

> Uptime Downtime  Outages  Response time

> 99.90%  0h 43m 35s  27   466 ms

> 

> 

> On Wed, Aug 1, 2012 at 11:50 PM, Mos < 
mosa...@googlemail.com> wrote:

> > And again I have to pay for GAE issues:

> >

> > On Jul 30 the Frontend Instance Hours goes beyond the free limit.

> > First time this week in my application history. Remember:  The

> > application was unusable on this day because GAE starts instances like
crazy

> (on low traffic).

> > Frontent Instance Hours were consumed of the buggy instance scheduler.

> >

> > But to rescue there is another bug in GAE. The billing does not work.

> > It's stuck for months and I still read the following message on my

> > billing history:

> > "We were unable to process your last payment. If the account balance

> > ($2.32) is not paid in full by 06/01/2012, this application's quotas

> > may be reset to the free levels."

> >

> > That's ingenious, Google.  ;)

> >

> >

> > On Mon, Jul 30, 2012 at 5:47 PM, Mos < 
mosa...@googlemail.com> wrote:

> >> Anyone else seeing issue with GAE instance management?

> >> In one minute - 10 requests - around 5 instances are started. Old

> >> ones do not response. New instances are created again and again...

> >>

> >>  
http://code.google.com/p/googleappengine/issues/detail?id=7910

> 

> --

> You received this message because you are subscribed to the Google Groups

> "Google App Engine" group.

> To post to this group, send email to

google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to
 google-

  >
appengine+unsubscr...@googlegroups.com.

> For more options, visit this group at

>  
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Is "Google Apps for your Domain going away and what's the impact for GAE users?

[Drake]

Yes this is a complaint I have, It is hard to set up Apps for Business/
Domains with a free account.  Even when you do you get sales calls telling
you why you should upgrade.

 

When I moved 100 domains we were ready to kill the people in that sales
group, we'd talk to the same idiot telemarketer 5 times in a row. In a 15
minute span.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Emanuele Ziglioli
Sent: Sunday, August 05, 2012 3:44 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Is "Google Apps for your Domain going away
and what's the impact for GAE users?

 

Thanks Brandon,

 

I haven't hears or seen anyone else commenting on "Apps for your Team".

So from my understanding, "Apps for your Team" was a way to use a non-gmail
account for accessing some Google Services.

Those non-google accounts @mydomain.com were considered "conflict accounts"
the minute one would claim "mydomain.com" on Google Apps.

Now Google is saying they're not allowing any more the ability of using non
Gmail domains   for Google Apps *unless* "mydomain.com" is setup for Google
Apps for your Business, Education, etc.

 

What's not clearly stated by Google is whether Google Apps for your Business
is still free for less than 25 users. In fact when you setup Google Apps,
you need to look for that option.

 

On Saturday, 4 August 2012 12:39:53 UTC+12, Brandon Wirtz wrote:

Apps for your Team is a thing no one used, and is different from apps for
domains.

This likely won't affect you unless you got bamboozled a long while ago by
one of the people who was a Apps Affiliate.

And you'd remember because the thing would have ruined your life so long ago
that you'd never have moved to GAE.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Emanuele Ziglioli
Sent: Friday, August 03, 2012 5:01 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Is "Google Apps for your Domain going away and
what's the impact for GAE users?

 

I'm puzzled: http://support.google.com/a/bin/answer.py?hl=en

&answer=2669622

 

1. What's "apps for your team"? Google always used "apps for your domain",
where did that name come from

2. GAE has forced us to create a "Google Apps for your Domain" account in
order to map the app to a custom domain and for using SSL

3. Are being forced to upgrade our free "Google Apps for your Domain" to a
pay-for "Apps for Business"? Enough of paying, enough of multiple bills,
can't you just have one Google bill that covers everything?

4. Three week notice? you must be joking!

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/9Bz56OyDXPIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/g3RcT9QFIvcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] HR datastore batch put

[Drake]

If Write latency is not an issue do a double defer to a function that then
does each write as a defer.

 

This will allow you retries per op.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Alfred Fuller
Sent: Sunday, August 05, 2012 10:30 PM
To: zorro
Cc: google-appengine@googlegroups.com; Robert Kluin
Subject: Re: [google-appengine] HR datastore batch put

 

Correct, you cannot rely on them being atomic outside a transaction.

On Sun, Aug 5, 2012 at 7:23 PM, zorro  wrote:

So to summarize the above mess, If I get it right: Batch put for entities
from the same entity group is not atomic. Right? Why can not somebody say
it?


On Monday, March 14, 2011 6:52:55 PM UTC+1, Alfred Fuller wrote:

Batch put is not atomic. When using a batch put across entity groups it is
very important to make sure the operation is idempotent (for example by not
using auto id allocation) as retrying a failed batch put may produce
duplicate/orphaned entities (if some succeed or some fail).

On Tue, Mar 8, 2011 at 10:29 PM, Robert Kluin 
wrote:

Hi Matija,
 The batch-put gets 'parallelized,' that is part of the reason I do
not think the put (across multiple entity groups) will be able to be
atomic;  I don't think you'll be able to figure out which entities did
or did not get written if you get an exception.  Unless, of course,
the app engine datastore guys are working on cross-entity group nested
transactions.  Of course, then you would probably lose some of the
performance benefits of the batch put being parallelized.




Robert







On Tue, Mar 8, 2011 at 06:07, Matija  wrote:
> Hi Robert,
> if you think of batch put as simple code helper for multiple entity put
> statement everything is pretty clear.
> But because latency for single and batch put is almost same batch put is
not
> simple code helper. I don't ask for some kind of multiple entity group
> serialization isolation level. I ask if HR datastore batch put has
atomicity
> guarantee or not.
> After every successful put statement I am able to get entity key for every
> stored entities but I don't need this information very often. I think that
> it would be nice to have in any kind of async/no response way that they
can
> provide atomic batch put statement. I think that this paxos/transactional
> log replication/'jada jada' arhitecture provides means for this very
> interesting database concept. Or maybe I am wrong. So wrong :D
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
 .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
 .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: What are the pros and cons of using Google App engine for my startup?

[Drake]

If your Instance cost is too high it means you aren't using API's enough.  A
well designed app is going to be close to 50% API, a superbly designed App
will be 85% API when you look at the logs and see ms=42 cpu_ms=2
api_cpu_ms=14  you know you are doing it right.

Again this means not using heavy frame works, but instead creating proper
datastore queries the right way, and optimizing your code for the platform.

Doing lots of optimizations does create lockin, but that is the trade off of
optimizing for "Instance cost" the same is true of AWS or OpenCloud
technologies. The more you Optimize the more you have lockin.

Reliability is not spotty unless you are building apps that take more than
35s per request. If you are doing that, your users already hate you and
won't be surprised when you are down.

Backend can save you a ton of money use them. Use them often.

One trans per entity per second is only an issue if you don't know how to
use shards and build a write buffer, (which you should have anyway for
managing contention). We are now handling 1 billion writes a day in an app.
In some cases logical entities are being updated 15 times per second. We are
having no trouble. We did end up building our own memcache-like server to
help with that, so to get to those numbers you need a backend.

It is not "All sunshine and unicorns" lack of naked domains, and Apps For
Domains weird rules are a pain in the ass.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: What are the pros and cons of using Google App engine for my startup?

[Drake]

To make AppEngine work for you. You have to say "I WANT TO BE GOOGLE". You
have to say "I WANT TO BUILD LEGO BLOCKS" you have to say "200MS is TOO
FRAKKING SLOW"

> Once more you are generalizing from an extremely limited problem domain.

You mean my natural language processor that has to compute the parts of
speech for every word in a document?
Or maybe my Image tool that auto-crops images so that they fit in a given
aspect ratio?
Or my content classifier that puts content in to classifications based on
the context and word usage?
Or my GEO tool that converts your IP Address to a metro, and generates best
version of ads for local targeting?
Or my SEO tool that scores a body of content against rules Google uses to
rank content?
Maybe you were talking about my widget tool that does Content relatedness
for suggesting other articles on the same topic?


> There are myriad applications that consume CPU resources - and if you're
on
> Python, the raw cost of serialization is a significant CPU hog.

Serialize in the background.
API your most CPU intensive stuff to backends.  Use light weight front ends
to handle the bulk of your requests.  Use F4's and F8s when you need bursts
of CPU power.

F1's suck at "dense" code that has lots of CPU constrained tasks.  But F4's
suck "Sparse" code that doesn't have enough CPU.
To get the best of both worlds  you need to have a mixture.

This takes getting used to. Most people haven't written for large
multi-server environments, and that is what you are really building with
Appengine.
You are building lots of microapps designed to be super efficient at the
thing they do, so that that thing fits nicely in 128M of Ram, and is
performant on a 600Mhz CPU

You can do this in 1 of two ways, you can shard your code so that frontend
instances spin up one way and explode when they get their 85th request and
your behemoth app crashes

Or you can design for "endpoints" , "APIs" or what ever you want to call it,
where specialized instances do their thing.  In some cases this means using
the "Versions" so that you can have ThingIDoRarely.Myapp.appsot.com and
ThingMyAccountantDoes.MyApp.Appspot.com

To make AppEngine work for you. You have to say "I WANT TO BE GOOGLE". You
have to say "I WANT TO BUILD LEGO BLOCKS" you have to say "200MS is TOO
FRAKKING SLOW"

Want me to solve 50k Transactions in 1 second?
If transaction rate is < X use default
if Transaction rate is > X & < Y Use Shard
if Transaction Rate is > Y use backend

And your backend uses in instance memory, and defers to populate a
persistent record. When the transaction rate drops below a threshold you do
commits and re-merge with master.







-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Computationally complex done fast: Was What are the Pros

[Drake]

So here are more generalizations for APP Engine:

Cache Everything:
Your CPU is slow, when profiling weigh the CPU to the memory cost and speed
for things which you calculate often. It is surprising that even simple
stuff denormalized can be so much faster than the complex operation.
GetByID("Sales-and-Shipping-AZ-1599-1lb") takes less than 3 MS out of
Instance memory cache served from a dedicated backend, 10ms from memcache
120ms from data store.  You can't do the 4 lookups and the calculation that
fast. So Don't.  Remember all your "Complex business rules" are cacheable.
Anything worth doing is worth caching :-)

API Everything:
Concurrency is your friend. Concurrency comes from using the API.  Anytime
your frontend is doing API stuff in request 1, Request 2 can do CPU stuff.
To get your maximum Concurrency you want to make requests to the API, or to
backends that function like API's  In the above example you would have a
Ship and Tax Calculator "API" that would calc and store data which could be
"Fetched" by ID. The tiering would only calculate it if the answer wasn't
already stored somewhere. During the calculation simpler requests are not
blocked or held in queue.

The above methodology also allows you to reduce your code foot print and
leverage "specialized memory". If an instance can use all 128M of ram for
saved calculations you can serve those calculations very, very quickly. And
you don't have to manage memory or worry about soft limit instance
termination. If you dump the memory to something that can be init'ed on
startup every so often, you can create a setup where you always on infinite
scale non-expiring memory.

Cascade Defers:
Building defers that do defers themselves can sound scary, but this allows
you to do more throttling of tasks in a very well threaded environment. When
you have task of different sizes and use the max concurrent, you can have
big tasks spawn small tasks and have fewer errors in your tasks. This is
also better for leveling out your peaks and valleys so that your app doesn't
get "sluggy" when there is a spin up. This is because if you have natural
traffic your "sluggy" tasks won't spawn as many "fast" tasks. So you won't
use as many instance hours.

Use Specialized DataTypes:
The biggest thing you give up with frameworks is the ability do the really
awesome GQL stuff with special data types.
Did you know you can do queries based on RGB values of your images?  I can
find "Images like this" using just appengine API's (sorry that bit of code
is secret)
But let's say tomorrow you wanted to build an Eharmony clone:

SELECT * FROM Person WHERE age >= 18 AND age <= 35 AND "white" IN
potential_races AND distance(location, geopoint(-33.857, 151.215)) < 15)

The Above Means you don't have much to write.  You can even use calculated
values which can off load a LOT of CPU cycles.
When you use frameworks however lots happens in the instance rather than in
the API, AND you pay for more queries in terms of both datastore calls and
CPU time.

To me the above list is the reason to do GAE.









-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Dynamic Backend Instance Scaling

[Drake]

Backends only handle serial requests. This is one of the biggest limitation
of Backends.  

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Sean O'Connor
Sent: Monday, August 06, 2012 2:55 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Dynamic Backend Instance Scaling

 

Hello,

 

I am working on a project where we are using a dynamic backend as a pull
queue worker.  Whenever we add a batch of tasks to our pull queue, we also
add a task to a push queue targeted to the backend to wake up the backend
and kick off the worker code.

 

Overall this has been working well for us as it allows us to offload work
from our front-end instances without paying too much for idle backend
instances.  The biggest problem we've been having with this approach has
been understanding and predicting how the dynamic instances are scaled.  To
that end, I have two questions:

1.  Is the "threadsafe" option enabling concurrent request handling in
the python27 runtime used by dynamic backends?
2.  If so, does the dynamic backend instance scaling work similarly to
front-end instance scaling or is there different rules at play here?

We've tried to read up on this in the docs and watching Google I/O talk
videos but all we've been finding is somewhat vague or
contradicting/outdated information.

 

Thanks!

 

-Sean O'Connor

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/6dLL0xNbjPUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Dynamic Backend Instance Scaling

[Drake]

H. I'll check again we have had a lot of issues with 503 errors because
instances wouldn't spin up, and if a request was in the Queue for more than
10s we'd get an error.

 

I'll have to check what we are doing wrong.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Kyle Finley
Sent: Monday, August 06, 2012 9:35 PM
To: Amy Unruh
Cc: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Dynamic Backend Instance Scaling

 

Yes-- the limits are the same as frontend instances.  Actually, at the
moment the limit is 8 for python, and 10 for java.

 

OK, Thanks. Is this documented anywhere officially?

 

If anyone is interested in making this number configurable please star this
future request:

http://code.google.com/p/googleappengine/issues/detail?id=7927

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Computationally complex done fast: Was What are the Pros

[Drake]

https://developers.google.com/appengine/docs/python/datastore/queries

 

Shows examples of <> which can be stacked, and "In List".  You can also do
calculated/weighted values where you say +4 if red and -2 if blue.

 

Get by ID should be around 150MS if you use the ID. 100ms is minimum
typically it trends up to 350ms. GAE has to be having a bad day for GetbyID
/key to be more than 500ms.

 

We are seeing 8MS average on fetches from backends In instance memory. And
for Static large data sets we have code that list literally as simple as 

this = 'that'

That = 'this'

Shipping400lbsto90201 = '345.97'

 

Which is very close to a no-op.

 

Our spin up time for a 100 meg static data set is 4s. Best of 100,000
requests was 3ms worst of was 114ms (again with an average of 8ms)

 

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Backend performance, compared

[Drake]

Jeff your Low volume is effecting your numbers. The more you do the more
consistent the numbers become.  I don't know what black magic Google does,
but the more people I have the more my instances get faster, the faster my
datastore reads (and writes) become, and the better the hit ratio on my
memcache.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] The Data Collection for Game Question

[Drake]

I don't have time to build a solution write now, because we are too close to
our own product launch, but in thinking about this I wondered about Channel
API between servers..

 

This is a solution I have been thinking about and if someone knows why it
won't work that might save me some headache or might reduce theirs if it
works.

 

Could you not have frontends use the channel API to send the data to the
backend.  The data would be streamed to the backend and the buffer in the
channel would allow the backend to read the stream of data in to its memory.
The calculations could be done, and then written to datastore and memcache
so that when the client re-polled the scores would be correct and shared?

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Backend performance, compared

[Drake]

Number of Instances -

Details

Average QPS

Average Latency

Average Memory


3 total (1 Resident)

1.017

568.7 ms

61.5 MBytes

 


Hit count:

237801


Miss count:

1940100


Hit ratio:

10%


Item count:

43785 item(s)


Total cache size:

78798944 byte(s)


Oldest item age:

8 hour(s) 36 min(s) 47 second(s)

 

 

 

 

Same App serving same content, but in a different Market

 


Number of Instances -
 Details

Average QPS

Average Latency

Average Memory


5 total

41.062

98.7 ms

51.6 MBytes

 

 

Mecache


Hit count:

2610910


Miss count:

308020


Hit ratio:

89%


Item count:

54803 item(s)


Total cache size:

92765915 byte(s)


Oldest item age:

8 hour(s) 27 min(s) 31 second(s)

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Computationally complex done fast: Was What are the Pros

[Drake]

I'll try and dig up the distance one... It was a client doing it, not my
code I don't do much with GEO that way, we do a hex of the entire US in 5
mile chunks and put in you a list.

But the client was doing distance.

(if you are anywhere in hex 2345 that is your "Tier 1" hex, then we know
that 23445 borders hex's 91234, 92345,93456,etc, and we use geographic
features to remove neighbors that you can't navigate to directly... (like
you can't cross Lake Eerie by bridge, and if you are doing a 15 mile radius
and are 9 miles south of the San Mateo Bridge a guy 5 miles as the crow
flies across the SF bay is not a neighboring hex)

You can do < and > for sure, but it two "ANDs"



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Computationally complex done fast: Was What are the Pros

[Drake]

And yes it is poorly documented.




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Anybody interested in a Datastore Manager for GAE?

[Drake]

Possibly. The Admin Console does suck, I would be happy to look at some
screen shots, see what you bring to the table.  We'd rather pay once.

When we looked in to this, the challenge was really that the datastore calls
add up fast.


> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Backpack
> Sent: Wednesday, August 08, 2012 2:42 PM
> To: Google App Engine
> Subject: [google-appengine] Anybody interested in a Datastore Manager for
> GAE?
>
> Something like phpMyAdmin for MySQL but adapted to the datastore.
> If so, answer these questions:
>
> - What datastore: db, ndb or SQL?
> - How much would you pay for that?
> - Pay once use forever with yearly versions?  ~$100/ver
> - or pay monthly with free updates? ~$10/mo
> - web based or desktop app?
>
> I am thinking on doing one with the latest HTML,CSS,JS stuff (sorry
> IE) and just wanted to see if there was any interest in a tool like that.
The
> data viewer and data admin stuff offered by gae are limited in
functionality.
>
> Thanks for your time in answering the questions.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Can you tweak the memacache timeout

[Drake]

I have asked for this in the past. When MemCache starts acting up we log it
and switch to 100% our own "memcache like server" for 20 minutes but that is
less than ideal.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Emanuele Ziglioli
Sent: Thursday, August 09, 2012 4:07 PM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Can you tweak the memacache timeout

 

Hi,

 

this morning we're experiencing repetitive memcache errors. I see a
"LogAndContinueErrorHandler" so requests don't fail but they take an awful
amount of time to be fullfilled.

Is there any way to force a maximum response time for memcache (something
like 50ms?).

I will profile it using appstat

 

 

com.google.appengine.api.memcache.LogAndContinueErrorHandler
handleServiceError: Service error in memcache
com.google.appengine.api.memcache.MemcacheServiceException: Memcache
getIdentifiables: exception getting multiple keys
   at
com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandle
r.handleApiProxyException(MemcacheServiceApiHelper.java:76)
   at
com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.absorbParentExc
eption(MemcacheServiceApiHelper.java:120)
   at
com.google.appengine.api.utils.FutureWrapper.handleParentException(FutureWra
pper.java:53)
   at
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:92)
   at
com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServi
ceImpl.java:28)
   at
com.google.appengine.api.memcache.MemcacheServiceImpl.getIdentifiables(Memca
cheServiceImpl.java:61)
   at
com.googlecode.objectify.cache.EntityMemcache.getAll(EntityMemcache.java:215
)
   at
com.googlecode.objectify.cache.CachingAsyncDatastoreService.get(CachingAsync
DatastoreService.java:253)
   at
com.googlecode.objectify.cache.CachingAsyncDatastoreService.get(CachingAsync
DatastoreService.java:216)
   at
com.googlecode.objectify.cache.CachingDatastoreService.get(CachingDatastoreS
ervice.java:137)
   at
siena.gae.GaePersistenceManager.get(GaePersistenceManager.java:2146)
   at siena.core.batch.BaseBatch.get(BaseBatch.java:60)

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/a6s1gdXVHUMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Can you tweak the memacache timeout

[Drake]

OOOooohhh.. Ahhh..

In a scenario where you were optimizing for performance regardless of cost,
you could do something similar to this and get from Memcache and from
datastore at the same time and check every 2ms to see which answers you
have.   That could save me 10ms average. Muahhahahahahaa.

 

Thanks for the idea. If I get to implementing it I will post the code back
here.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Johan Euphrosine
Sent: Friday, August 10, 2012 6:55 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Can you tweak the memacache timeout

 

I answered your question on S/O:

http://stackoverflow.com/a/11902993/656408

 

tltr; It's less straightforward in Java than it is in Python, you should
feel a feature request on the public issue tracker to improve the API.

http://code.google.com/p/googleappengine/issues/entry?template=Feature%20req
uest

 

Hope that helps.

 

On Fri, Aug 10, 2012 at 5:18 AM, Emanuele Ziglioli
 wrote:

I'm doing some experiments on
com.google.appengine.api.memcache.MemcacheServiceImpl

 

  private static  T quietGet(Future future) {

try {

//  return future.get();

  return future.get( 100, TimeUnit.MILLISECONDS);

} catch (TimeoutException ex) {

//  throw new ExecutionException(createDeadlineException());

  logger.log(Level.WARNING, ex.getMessage() + " " + future );

  return null;

}

catch (InterruptedException e) {

  Thread.currentThread().interrupt();

  throw new MemcacheServiceException("Unexpected failure", e);

} catch (ExecutionException e) {

  Throwable cause = e.getCause();

  if (cause instanceof RuntimeException) {

throw (RuntimeException) cause;

  } else if (cause instanceof Error) {

throw (Error) cause;

  } else {

throw new UndeclaredThrowableException(cause);

  }

}

  }

 



I'm not sure if it's working 100% yet, the web app seems to open faster, but
there are still errors.

I'm sure something can be improved around there

I'm off next week, have fun

 

 

 

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.

To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/aKrs3-BElJwJ.


To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
 .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.





 

-- 
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] How can I block curl requests

[Drake]

If they didn't follow your robots.txt and you know who they are, send a
legal order, sue for damages and wipe them off the map.

 

.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Next Time you think the scheduler is Broken...

[Drake]

So the next time you think the Check out line analogy is flawed.. I think it
is accurate but I'm not sure the scheduler does what we all expect..

 

This is about 100 Gigs worth of xfer in 70k Requests a day. 

 

 


0.000

0.0 ms

57

0

1:30:37

32.4 MBytes

 
 View Logs

Description: Dynamic IconDynamic


1.683

21.2 ms

21023

0

7:50:49

51.0 MBytes

 
 View Logs

Description: Dynamic IconDynamic


0.000

0.0 ms

266

0

8:04:53

31.6 MBytes

 
 View Logs

Description: Dynamic IconDynamic


0.000

0.0 ms

25

0

8:11:32

31.0 MBytes

 
 View Logs

Description: Dynamic IconDynamic



 

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

<>

RE: [google-appengine] Next Time you think the scheduler is Broken...

[Drake]

I'm pretty sure the most efficient check out system is not to have one
cashier handle 21k customers in 8 hours and the register that was open 15
minutes sooner handle only 25.

 

Or to have a 4th lane open for 57 requests.

 

No combination of this was optimal for the 15 minute billing increment.
Even if those 25 requests are 14 min 59 sec apart, 25 /4 is only 6 hours and
change.  

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer
Sent: Friday, August 17, 2012 11:58 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Next Time you think the scheduler is
Broken...

 

What are you trying to say here?

 

Jeff

On Sat, Aug 18, 2012 at 2:08 AM, Drake  wrote:

So the next time you think the Check out line analogy is flawed.. I think it
is accurate but I'm not sure the scheduler does what we all expect..

 

This is about 100 Gigs worth of xfer in 70k Requests a day. 

 

 


0.000

0.0 ms

57

0

1:30:37

32.4 MBytes

 
<https://appengine.google.com/logs?app_id=s~stremorcdn&version_id=28.3611098
27147846613&severity_level_override=1&severity_level=3&filter=instance%3D00c
61b117c8ba9543aab3ecfc850baacd6dd4dcf&filter_type=regex&date_type=now&limit=
20&view=Search> View Logs

Description: Dynamic IconDynamic



1.683

21.2 ms

21023

0

7:50:49

51.0 MBytes

 
<https://appengine.google.com/logs?app_id=s~stremorcdn&version_id=28.3611098
27147846613&severity_level_override=1&severity_level=3&filter=instance%3D00c
61b117cceee87a5c8b3c26bfdef9393196486&filter_type=regex&date_type=now&limit=
20&view=Search> View Logs

Description: Dynamic IconDynamic



0.000

0.0 ms

266

0

8:04:53

31.6 MBytes

 
<https://appengine.google.com/logs?app_id=s~stremorcdn&version_id=28.3611098
27147846613&severity_level_override=1&severity_level=3&filter=instance%3D00c
61b117c5937b082ee32869e0155dd6dd48ffb&filter_type=regex&date_type=now&limit=
20&view=Search> View Logs

Description: Dynamic IconDynamic



0.000

0.0 ms

25

0

8:11:32

31.0 MBytes

 
<https://appengine.google.com/logs?app_id=s~stremorcdn&version_id=28.3611098
27147846613&severity_level_override=1&severity_level=3&filter=instance%3D00c
61b117c320e688017ba4d38f95222d9430596&filter_type=regex&date_type=now&limit=
20&view=Search> View Logs

Description: Dynamic IconDynamic



 

 

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine%2bunsubscr...@googlegroups.com> .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

<>

[google-appengine] Google Support Request: Can I get more Quota and Apps

[Drake]

Can I get more apps for appsad...@stremor.com I am almost out and will need
5 on Monday which will put me over my limit. And 10 more in 2 weeks. 

 

Also can I get the URL fetch limit taken off of stremor-basher which is
meant to be a load tester so I can DDoS myself.

 

Thanks!

 

-Brandon

 

PS

www.unpartial.com is coming Monday.

 

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Google Support Request: Can I get more Quota and Apps

[Drake]

Maybe, but I think it takes time for them to become available after enabling
billing.

Because of memory limitation mult-tenant is not an option.

> -Original Message-
> From: google-appengine@googlegroups.com [mailto:google-
> appeng...@googlegroups.com] On Behalf Of Johan Euphrosine
> Sent: Friday, August 24, 2012 4:00 PM
> To: google-appengine@googlegroups.com
> Subject: Re: [google-appengine] Google Support Request: Can I get more
> Quota and Apps
>
> On Fri, Aug 24, 2012 at 11:45 PM, Drake  wrote:
> > Can I get more apps for appsad...@stremor.com I am almost out and will
> > need
> > 5 on Monday which will put me over my limit. And 10 more in 2 weeks.
>
> Are they paid app? AFAIK this limit only applied to free app.
>
> >
> >
> >
> > Also can I get the URL fetch limit taken off of stremor-basher which
> > is meant to be a load tester so I can DDoS myself.
> >
> >
> >
> > Thanks!
> >
> >
> >
> > -Brandon
> >
> >
> >
> > PS
> >
> > www.unpartial.com is coming Monday.
> >
> >
> >
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/google-appengine?hl=en.
>
>
>
> --
> Johan Euphrosine (proppy)
> Developer Programs Engineer
> Google Developer Relations
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-
> appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Google Support Request: Can I get more Quota and Apps

[Drake]

Let's say you want to keep sessions cached in local instance. If you have
room to cache 1000 users in instance memory and 3 tenants each have 5000
simultaneous users, you can't fit everyone in memory.

 

So you need multiple apps to avoid massive cache misses.

 

Say you have 100k products per app, and you want to search them, you can't
fit 300k items in memory so you can't multi-tenant.

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Aza Tek
Sent: Saturday, August 25, 2012 2:19 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Google Support Request: Can I get more Quota
and Apps

 

Drake, could you please explain the memory limitation associated with
multi-tenant?

On Sat, Aug 25, 2012 at 2:34 AM, Drake  wrote:


Because of memory limitation mult-tenant is not an option.

 

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.