Re: [google-appengine] Re: What's going on with my CPU times

2011-05-15 Thread Robert Kluin 
Hey Steve,
  I see something similar since I've enabled always on.  But in my
case it seems to be much less extreme.  Appstats numbers are
more-or-less consistent between the two requests, clearly there is
something going on here.  The really funny thing is that the CPU ms
number is higher than would be caused by a loading request to my app.

This is exactly the same operation, ie fetching the same entity from
the datastore (no caching):
Warm request:
  Logs:  ms=24 cpu_ms=35 api_cpu_ms=8 cpm_usd=0.001236
  Appstats: real=11ms cpu=18ms api=8ms overhead=0ms

Cold request:
  Logs:  ms=110 cpu_ms=2225 api_cpu_ms=8 cpm_usd=0.062068
  Appstats:  real=93ms cpu=20ms api=8ms overhead=0ms



Robert






On Sun, May 15, 2011 at 15:51, stevep prosse...@gmail.com wrote:
 Thanks Brandon,

 I probably mis-communicated. I'm not doing any image manipulation. On-
 line handler receives the jpeg bytes posted by client and saves them
 in a db.BlobProperty entity property. When the call for the jpeg comes
 in, handler simply does a get_by_id, and writes out the
 db.BlobProperty value.

 The image call is pretty much exactly like the example at Google's
 docs page, but without the query, just the get_by_id:
 http://code.google.com/appengine/articles/python/serving_dynamic_images.html

 Did a little testing today, and am pretty sure the cpu variations are
 due to my limited volumes. Below are three consecutive image gets.
 Notice how only the first one (which came after a period of
 inactivity) has the high cpu total.

 I had thought paying to have the on-line handler resident avoided that
 startup cpu costs for the first activity after a period of non-
 acitivity. Just not groking that I think. So for now a non-issue I
 think. My minute volumes are never going to approach using my CPU
 daily quote even if every image get averages 13,000ms. If volumes
 start to rise, clearly the number of startup cpu instances falls.

 Thanks again,
 Steve

 *Log Extract underscores preceding cpu ms*
  subsequent calls 
 2011-05-15 12:36:10.357 /cardimage/80002.jpg 200 201ms_66cpu_ms
 8api_cpu_ms
 2011-05-15 12:36:04.040 /cardimage/73005.jpg 200 312ms_63cpu_ms
 8api_cpu_ms
  startup call 
 2011-05-15 12:35:56.980 /cardimage/74003.jpg 200 204ms_2427cpu_ms
 8api_cpu_ms

 On May 14, 3:20 pm, Brandon Wirtz drak...@digerat.com wrote:
 Well Duh, You are generating a JPG, you said send the bytes but you are
 modifying, rendering, and compressing a JPG.  That takes some CPU.

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: What's going on with my CPU times

2011-05-14 Thread Brandon Wirtz 
Well Duh, You are generating a JPG, you said send the bytes but you are
modifying, rendering, and compressing a JPG.  That takes some CPU.

My app sends jpgs and uses no CPU as it were because it really is do I have
this? Oh I do? Great here it is 

Your app is Hey Draw me a pretty picture Um... Ok let me do that for
youYou probably want that as a JPG not a BMP let me compress that
Let me save it to a blob incase you need it later...  here you go

It's the difference between having an intern running a photo copy and asking
Me to paint you a picture of you with my cat.   And now I'm certain you are
using APIs which have a minimum CPU per action cost.

Likely if these are one time use images, you should just write them, no
blob.
These are small, so likely if you need to keep them around you should write
them to data store
If you only kind of need them sometimes, consider memcache so you don't have
to clean up in a week when you have lots of these.



-Original Message-
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of stevep
Sent: Saturday, May 14, 2011 3:06 PM
To: Google App Engine
Subject: [google-appengine] Re: What's going on with my CPU times

Thanks Brandon. Every image is unique, and is generated by users from the
client. So statics limitations make it infeasible.

Right now I think this has to do with the on/off again hits. I went into my
Dashboard after a few individual calls, and saw I had gone from 3 resident
instances: 3 resident and 3 dynamic. Although this was serving a query rate
of 0.018 per second. My average instance latency was 65ms I think.

Have no idea what the resident instances are doing. But guess will wait and
see.

Cheers,
stevep

On May 14, 12:27 pm, Brandon Wirtz drak...@digerat.com wrote:
 Put it not in a blog.

 Consider using Static (depending what else you are doing)

 With out code it is hard to tell what you are doing but you may be 
 using API cpu and not CPU Cpu

 Most of my CPU bill is API.



 -Original Message-
 From: google-appengine@googlegroups.com

 [mailto:google-appengine@googlegroups.com] On Behalf Of stevep
 Sent: Saturday, May 14, 2011 8:04 AM
 To: Google App Engine
 Subject: [google-appengine] What's going on with my CPU times

 Can anyone help me understand these CPU times.

 I have an on-line handler function that basically does a get_by_id(), 
 and sends out the bytes for a 150x100 pixel jpeg BLOB record value. A 
 pretty straightforward image retrieval from the datastore.

 My CPU loads are all over the map. I am $9 paying for 3 instances to 
 stay resident. An instance takes just less than 15 meg of RAM. Very 
 low volumes right now, so maybe that affects this?

 Looking at AppStats, the function itself is consistently taking 200ms 
 or less for every call. So how am I getting 13,000ms for this simple
function?

 I've copied/pasted a series of calls for this image below. Here is an 
 extract of the last 4 url calls:
 242ms...80cpu
 351ms...13,740cpu
 466ms4,903cpu
 182ms2,922cpu

 Looking at the full list below, I only occasionally get fast cpu 
 times. This is the simplest function, can't imagine where the 
 inefficiencies are. Is the HR datastore that slow at pushing a small 
 payload of bytes out the door? Man oh man, averaging these cpu times 
 for a 150x100 pixel jpg, I better switch my image saves/retrieves to
 S3 mucho soon.

 Thanks in advance,
 stevep

 2011-05-14 07:20:10.949 /cardimage/59005_t.jpg 200 242ms 80cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-14 07:03:04.967 /cardimage/59005_t.jpg 200 351ms 13740cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-13 14:50:48.065 /cardimage/59005_t.jpg 200 466ms 4903cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-13 13:57:09.990 /cardimage/59005_t.jpg 200 182ms 2922cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-13 10:39:27.922 /cardimage/59005_t.jpg 200 196ms 1474cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-13 09:01:08.347 /cardimage/59005_t.jpg 200 494ms 777cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-13 08:13:33.672 /cardimage/59005_t.jpg 200 501ms 435cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-13 07:23:52.577 /cardimage/59005_t.jpg 200 371ms 117cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip(gfe),gzip(gfe),gzip(gfe)
 2011-05-13 07:06:09.403 /cardimage/59005_t.jpg 200 880ms 1585cpu_ms 
 8api_cpu_ms 35kb facebookplatform/1.0 (+http://
 developers.facebook.com),gzip