[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-10-08 Thread iceanfire

I just wanted to follow up on this discussion to (shamefacedly) point
out what was causing this problem: imports.

I had copied and pasted some code earlier, which I had later deleted
to slim down my code. What I had forgotten to do was: remove the
imports for that piece of code. Once I had removed them, I noticed a
great improvement in performance.

Ever since I removed that piece of code + changed debug to false and
removed the profiler code, I haven't had any cpu warnings for that
particular piece of code.

So the lesson here is: If you're having trouble reducing your CPU
warnings, try looking at imports and make sure that you only call the
essential ones.



On Oct 1, 2:24 am, iceanfire [EMAIL PROTECTED] wrote:
 I will try removing the debug stuff, maybe that is what's causing this
 problem. I'll let you know if this helps.

 Thanks.

 On Sep 29, 6:44 am, Sylvain [EMAIL PROTECTED] wrote:

  Hi,

  memcache can decrease the average, that's all.
  memcache is really good, but it can't be the answer for everything.

  Else, I think with this CPU/Warning, there is no good solution. Now,
  I'm just waiting for higher quota (free or not).

  Did you remove all DEBUG flag (webapp, django render option,...)
  because it uses a lot of CPU.
  The profiler code uses a lot of CPU too.

  Regards

  On 29 sep, 12:57, Arun Shanker Prasad [EMAIL PROTECTED]
  wrote:

   Hi Barry,

   I know that the memcache is free, but it will push out stuff if the
   memory usage is high, I already have much bigger query result cached
   in the memcache, I want to keep them there as long as possible. :)

   Thanks,
   Arun Shanker Prasad.

   On Sep 29, 3:44 pm, Barry Hunter [EMAIL PROTECTED]
   wrote:

On Mon, Sep 29, 2008 at 11:18 AM, Arun Shanker Prasad

[EMAIL PROTECTED] wrote:

 My app is also causing the same problems, I have used etags to set the
 response to 302 if cached, I've tried everything short of memcache, I
 have many images, I don't think that is a viable solution for me.

Why not? memcache is 'free', doesn't seem to be any reason NOT to use 
it?

memcache is already designed to keep 'hot' items in the cache - so it
will automatically discard little used images (or what ever you
store).

You should I would of thought be looking to cache everything possible.

 Any suggestions are welcome.

 On Sep 29, 2:10 am, iceanfire [EMAIL PROTECTED] wrote:
 Thanks for the suggestions. I see the need for Cacheing to reduce the
 load, but I don't understand why the current request is causing high-
 cpu warnings (2 times the average cpu request). At this rate, if a 
 few
 first time users use my application  try to open images that haven't
 been memcached (etc..).. then my application will crash.

 My main question is: why is this request causing a high cpu warning,
 and no one has been able to answer that. For a request that only
 takes  0.020 CPU seconds (according to profiler), appengine sends
 out a warning stating that 2463mcycles have been used.

 So my question is: how do I completely stop high-cpu warnings for a
 request that shouldn't be causing them in the first place?

 I understand that there are ways I can mitigate the problem..but i'd
 like to get to the root if possible.

 thanks!

 On Sep 25, 9:26 am, Bryan A. Pendleton [EMAIL PROTECTED] wrote:

      -So are textProperties more efficient than StringProperties
  because
      they're not indexed?

      You'd have to find the talk from Google IO to be sure. I 
  believe
  it
      was the one about scalability, in the QA section. But yes, 
  that is
  my
      understanding.

  As I understand it, every field that's not a TextProperty or a
  BlobProperty are implicitly indexed (this is how all = conditions 
  are
  dealt with in queries). So, whenever you write such an object, it 
  will
  take longer (because of the index updates).

  Another way of thinking about it, is that if you never need to 
  query
  on a single value, make it a TextProperty or BlobProperty, if
  possible.

      -Wouldn't adding etag--while increasing efficiency if I have 
  the
  same
      users loading the sameimageagain and again--actually decrease
      efficiency for users who are opening up an thumbnail for the 
  first
      time? In that situation,  I'd have another column for etags in 
  my
      datastore being requested w/ every query.

      Yes, you absolutely should generate the etag when you save the
      thumbnail, and save it in the model itself.Cachingit separately
  is
      however still desirable as you can then avoid pulling the rest 
  of
  the
      data into memory if it's not needed, or you can opt to not 
  cache
  the
      rest of the data at all, instead onlycachingthe etag, to be 
  

[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-10-08 Thread yejun

The default cache setting is no-cache, the browser will revalidate the
same image ever time the image being displayed.
You may try to set this explicitly to a week.
self.response.headers['Cache-Control'] = 'public, max-age=63'

On Sep 24, 10:59 pm, iceanfire [EMAIL PROTECTED] wrote:
 I'm still having trouble with high-cpu warnings for thumbnails. This
 time around I took some profiler data to see what's causing it. But
 before I get into that here is the model i'm using:

 class ImageThumb(db.Model):
   binId = db.IntegerProperty()
   thumb = db.BlobProperty(default=None)
   building = db.ReferenceProperty(Buildings)
   apartment = db.ReferenceProperty(Apartments)
   mime = db.StringProperty()
   type = db.StringProperty(choices=['Floor Plan','Picture'])
   created_by =  db.UserProperty()

 So here's the Profiler cpu data:
 2538 function calls (2472 primitive calls) in 0.028 CPU seconds (rest
 of the data:http://docs.google.com/Doc?id=dgfxff5_30hc9tjsgg)

 But here's the warning: This request used a high amount of CPU, and
 was roughly 1.3 times over the average request CPU limit. High CPU
 requests have a small quota, and if you exceed this quota, your app
 will be temporarily disabled.

 Here's the megacycle info from the new Admin console: 1339mcycles 7kb

 Here's the code that pulls up the data for thumbnails:
 class Apt_thumb (webapp.RequestHandler):
     def get(self,id):
                 image = ImageThumb.get_by_id(int(id))
                 if image:
                         self.response.headers['Content-Type'] = 
 str(image.mime)
                         self.response.out.write(image.thumb)
                 else:
                         self.error(404)

 The images stored are 4kb each in the datastore. As I said, I had this
 problem earlier, so I had taken out the full image and put that in its
 own Model. Clearly that didn't help.

 Any idea what's causing this high-cpu error  how I can fix it?

 Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---



[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-10-01 Thread iceanfire

I will try removing the debug stuff, maybe that is what's causing this
problem. I'll let you know if this helps.

Thanks.

On Sep 29, 6:44 am, Sylvain [EMAIL PROTECTED] wrote:
 Hi,

 memcache can decrease the average, that's all.
 memcache is really good, but it can't be the answer for everything.

 Else, I think with this CPU/Warning, there is no good solution. Now,
 I'm just waiting for higher quota (free or not).

 Did you remove all DEBUG flag (webapp, django render option,...)
 because it uses a lot of CPU.
 The profiler code uses a lot of CPU too.

 Regards

 On 29 sep, 12:57, Arun Shanker Prasad [EMAIL PROTECTED]
 wrote:

  Hi Barry,

  I know that the memcache is free, but it will push out stuff if the
  memory usage is high, I already have much bigger query result cached
  in the memcache, I want to keep them there as long as possible. :)

  Thanks,
  Arun Shanker Prasad.

  On Sep 29, 3:44 pm, Barry Hunter [EMAIL PROTECTED]
  wrote:

   On Mon, Sep 29, 2008 at 11:18 AM, Arun Shanker Prasad

   [EMAIL PROTECTED] wrote:

My app is also causing the same problems, I have used etags to set the
response to 302 if cached, I've tried everything short of memcache, I
have many images, I don't think that is a viable solution for me.

   Why not? memcache is 'free', doesn't seem to be any reason NOT to use it?

   memcache is already designed to keep 'hot' items in the cache - so it
   will automatically discard little used images (or what ever you
   store).

   You should I would of thought be looking to cache everything possible.

Any suggestions are welcome.

On Sep 29, 2:10 am, iceanfire [EMAIL PROTECTED] wrote:
Thanks for the suggestions. I see the need for Cacheing to reduce the
load, but I don't understand why the current request is causing high-
cpu warnings (2 times the average cpu request). At this rate, if a few
first time users use my application  try to open images that haven't
been memcached (etc..).. then my application will crash.

My main question is: why is this request causing a high cpu warning,
and no one has been able to answer that. For a request that only
takes  0.020 CPU seconds (according to profiler), appengine sends
out a warning stating that 2463mcycles have been used.

So my question is: how do I completely stop high-cpu warnings for a
request that shouldn't be causing them in the first place?

I understand that there are ways I can mitigate the problem..but i'd
like to get to the root if possible.

thanks!

On Sep 25, 9:26 am, Bryan A. Pendleton [EMAIL PROTECTED] wrote:

     -So are textProperties more efficient than StringProperties
 because
     they're not indexed?

     You'd have to find the talk from Google IO to be sure. I believe
 it
     was the one about scalability, in the QA section. But yes, that 
 is
 my
     understanding.

 As I understand it, every field that's not a TextProperty or a
 BlobProperty are implicitly indexed (this is how all = conditions are
 dealt with in queries). So, whenever you write such an object, it 
 will
 take longer (because of the index updates).

 Another way of thinking about it, is that if you never need to query
 on a single value, make it a TextProperty or BlobProperty, if
 possible.

     -Wouldn't adding etag--while increasing efficiency if I have the
 same
     users loading the sameimageagain and again--actually decrease
     efficiency for users who are opening up an thumbnail for the 
 first
     time? In that situation,  I'd have another column for etags in my
     datastore being requested w/ every query.

     Yes, you absolutely should generate the etag when you save the
     thumbnail, and save it in the model itself.Cachingit separately
 is
     however still desirable as you can then avoid pulling the rest of
 the
     data into memory if it's not needed, or you can opt to not cache
 the
     rest of the data at all, instead onlycachingthe etag, to be more
     cache friendly.

 A quick and easy hack for this is to generate the etag before 
 creating
 the Thumbnail model instance - and use that etag as the named key.
 Then, you can do lookup andcachingbased on the etag alone, where
 that makes sense. Unless you have some specific meaning in your ID
 already, this should simplify the how to deal with etags question
 quite a bit.

   --
   Barry

   -www.nearby.org.uk-www.geograph.org.uk-
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en

[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-09-29 Thread Arun Shanker Prasad

My app is also causing the same problems, I have used etags to set the
response to 302 if cached, I've tried everything short of memcache, I
have many images, I don't think that is a viable solution for me.

Any suggestions are welcome.

On Sep 29, 2:10 am, iceanfire [EMAIL PROTECTED] wrote:
 Thanks for the suggestions. I see the need for Cacheing to reduce the
 load, but I don't understand why the current request is causing high-
 cpu warnings (2 times the average cpu request). At this rate, if a few
 first time users use my application  try to open images that haven't
 been memcached (etc..).. then my application will crash.

 My main question is: why is this request causing a high cpu warning,
 and no one has been able to answer that. For a request that only
 takes  0.020 CPU seconds (according to profiler), appengine sends
 out a warning stating that 2463mcycles have been used.

 So my question is: how do I completely stop high-cpu warnings for a
 request that shouldn't be causing them in the first place?

 I understand that there are ways I can mitigate the problem..but i'd
 like to get to the root if possible.

 thanks!

 On Sep 25, 9:26 am, Bryan A. Pendleton [EMAIL PROTECTED] wrote:

      -So are textProperties more efficient than StringProperties
  because
      they're not indexed?

      You'd have to find the talk from Google IO to be sure. I believe
  it
      was the one about scalability, in the QA section. But yes, that is
  my
      understanding.

  As I understand it, every field that's not a TextProperty or a
  BlobProperty are implicitly indexed (this is how all = conditions are
  dealt with in queries). So, whenever you write such an object, it will
  take longer (because of the index updates).

  Another way of thinking about it, is that if you never need to query
  on a single value, make it a TextProperty or BlobProperty, if
  possible.

      -Wouldn't adding etag--while increasing efficiency if I have the
  same
      users loading the sameimageagain and again--actually decrease
      efficiency for users who are opening up an thumbnail for the first
      time? In that situation,  I'd have another column for etags in my
      datastore being requested w/ every query.

      Yes, you absolutely should generate the etag when you save the
      thumbnail, and save it in the model itself.Cachingit separately
  is
      however still desirable as you can then avoid pulling the rest of
  the
      data into memory if it's not needed, or you can opt to not cache
  the
      rest of the data at all, instead onlycachingthe etag, to be more
      cache friendly.

  A quick and easy hack for this is to generate the etag before creating
  the Thumbnail model instance - and use that etag as the named key.
  Then, you can do lookup andcachingbased on the etag alone, where
  that makes sense. Unless you have some specific meaning in your ID
  already, this should simplify the how to deal with etags question
  quite a bit.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---



[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-09-29 Thread Barry Hunter

On Mon, Sep 29, 2008 at 11:18 AM, Arun Shanker Prasad
[EMAIL PROTECTED] wrote:

 My app is also causing the same problems, I have used etags to set the
 response to 302 if cached, I've tried everything short of memcache, I
 have many images, I don't think that is a viable solution for me.

Why not? memcache is 'free', doesn't seem to be any reason NOT to use it?

memcache is already designed to keep 'hot' items in the cache - so it
will automatically discard little used images (or what ever you
store).

You should I would of thought be looking to cache everything possible.


 Any suggestions are welcome.

 On Sep 29, 2:10 am, iceanfire [EMAIL PROTECTED] wrote:
 Thanks for the suggestions. I see the need for Cacheing to reduce the
 load, but I don't understand why the current request is causing high-
 cpu warnings (2 times the average cpu request). At this rate, if a few
 first time users use my application  try to open images that haven't
 been memcached (etc..).. then my application will crash.

 My main question is: why is this request causing a high cpu warning,
 and no one has been able to answer that. For a request that only
 takes  0.020 CPU seconds (according to profiler), appengine sends
 out a warning stating that 2463mcycles have been used.

 So my question is: how do I completely stop high-cpu warnings for a
 request that shouldn't be causing them in the first place?

 I understand that there are ways I can mitigate the problem..but i'd
 like to get to the root if possible.

 thanks!

 On Sep 25, 9:26 am, Bryan A. Pendleton [EMAIL PROTECTED] wrote:

  -So are textProperties more efficient than StringProperties
  because
  they're not indexed?

  You'd have to find the talk from Google IO to be sure. I believe
  it
  was the one about scalability, in the QA section. But yes, that is
  my
  understanding.

  As I understand it, every field that's not a TextProperty or a
  BlobProperty are implicitly indexed (this is how all = conditions are
  dealt with in queries). So, whenever you write such an object, it will
  take longer (because of the index updates).

  Another way of thinking about it, is that if you never need to query
  on a single value, make it a TextProperty or BlobProperty, if
  possible.

  -Wouldn't adding etag--while increasing efficiency if I have the
  same
  users loading the sameimageagain and again--actually decrease
  efficiency for users who are opening up an thumbnail for the first
  time? In that situation,  I'd have another column for etags in my
  datastore being requested w/ every query.

  Yes, you absolutely should generate the etag when you save the
  thumbnail, and save it in the model itself.Cachingit separately
  is
  however still desirable as you can then avoid pulling the rest of
  the
  data into memory if it's not needed, or you can opt to not cache
  the
  rest of the data at all, instead onlycachingthe etag, to be more
  cache friendly.

  A quick and easy hack for this is to generate the etag before creating
  the Thumbnail model instance - and use that etag as the named key.
  Then, you can do lookup andcachingbased on the etag alone, where
  that makes sense. Unless you have some specific meaning in your ID
  already, this should simplify the how to deal with etags question
  quite a bit.
 




-- 
Barry

- www.nearby.org.uk - www.geograph.org.uk -

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---



[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-09-29 Thread Arun Shanker Prasad

Hi Barry,

I know that the memcache is free, but it will push out stuff if the
memory usage is high, I already have much bigger query result cached
in the memcache, I want to keep them there as long as possible. :)

Thanks,
Arun Shanker Prasad.

On Sep 29, 3:44 pm, Barry Hunter [EMAIL PROTECTED]
wrote:
 On Mon, Sep 29, 2008 at 11:18 AM, Arun Shanker Prasad

 [EMAIL PROTECTED] wrote:

  My app is also causing the same problems, I have used etags to set the
  response to 302 if cached, I've tried everything short of memcache, I
  have many images, I don't think that is a viable solution for me.

 Why not? memcache is 'free', doesn't seem to be any reason NOT to use it?

 memcache is already designed to keep 'hot' items in the cache - so it
 will automatically discard little used images (or what ever you
 store).

 You should I would of thought be looking to cache everything possible.





  Any suggestions are welcome.

  On Sep 29, 2:10 am, iceanfire [EMAIL PROTECTED] wrote:
  Thanks for the suggestions. I see the need for Cacheing to reduce the
  load, but I don't understand why the current request is causing high-
  cpu warnings (2 times the average cpu request). At this rate, if a few
  first time users use my application  try to open images that haven't
  been memcached (etc..).. then my application will crash.

  My main question is: why is this request causing a high cpu warning,
  and no one has been able to answer that. For a request that only
  takes  0.020 CPU seconds (according to profiler), appengine sends
  out a warning stating that 2463mcycles have been used.

  So my question is: how do I completely stop high-cpu warnings for a
  request that shouldn't be causing them in the first place?

  I understand that there are ways I can mitigate the problem..but i'd
  like to get to the root if possible.

  thanks!

  On Sep 25, 9:26 am, Bryan A. Pendleton [EMAIL PROTECTED] wrote:

       -So are textProperties more efficient than StringProperties
   because
       they're not indexed?

       You'd have to find the talk from Google IO to be sure. I believe
   it
       was the one about scalability, in the QA section. But yes, that is
   my
       understanding.

   As I understand it, every field that's not a TextProperty or a
   BlobProperty are implicitly indexed (this is how all = conditions are
   dealt with in queries). So, whenever you write such an object, it will
   take longer (because of the index updates).

   Another way of thinking about it, is that if you never need to query
   on a single value, make it a TextProperty or BlobProperty, if
   possible.

       -Wouldn't adding etag--while increasing efficiency if I have the
   same
       users loading the sameimageagain and again--actually decrease
       efficiency for users who are opening up an thumbnail for the first
       time? In that situation,  I'd have another column for etags in my
       datastore being requested w/ every query.

       Yes, you absolutely should generate the etag when you save the
       thumbnail, and save it in the model itself.Cachingit separately
   is
       however still desirable as you can then avoid pulling the rest of
   the
       data into memory if it's not needed, or you can opt to not cache
   the
       rest of the data at all, instead onlycachingthe etag, to be more
       cache friendly.

   A quick and easy hack for this is to generate the etag before creating
   the Thumbnail model instance - and use that etag as the named key.
   Then, you can do lookup andcachingbased on the etag alone, where
   that makes sense. Unless you have some specific meaning in your ID
   already, this should simplify the how to deal with etags question
   quite a bit.

 --
 Barry

 -www.nearby.org.uk-www.geograph.org.uk-
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---



[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-09-28 Thread iceanfire

Thanks for the suggestions. I see the need for Cacheing to reduce the
load, but I don't understand why the current request is causing high-
cpu warnings (2 times the average cpu request). At this rate, if a few
first time users use my application  try to open images that haven't
been memcached (etc..).. then my application will crash.

My main question is: why is this request causing a high cpu warning,
and no one has been able to answer that. For a request that only
takes  0.020 CPU seconds (according to profiler), appengine sends
out a warning stating that 2463mcycles have been used.

So my question is: how do I completely stop high-cpu warnings for a
request that shouldn't be causing them in the first place?

I understand that there are ways I can mitigate the problem..but i'd
like to get to the root if possible.

thanks!

On Sep 25, 9:26 am, Bryan A. Pendleton [EMAIL PROTECTED] wrote:
     -So are textProperties more efficient than StringProperties
 because
     they're not indexed?

     You'd have to find the talk from Google IO to be sure. I believe
 it
     was the one about scalability, in the QA section. But yes, that is
 my
     understanding.

 As I understand it, every field that's not a TextProperty or a
 BlobProperty are implicitly indexed (this is how all = conditions are
 dealt with in queries). So, whenever you write such an object, it will
 take longer (because of the index updates).

 Another way of thinking about it, is that if you never need to query
 on a single value, make it a TextProperty or BlobProperty, if
 possible.

     -Wouldn't adding etag--while increasing efficiency if I have the
 same
     users loading the same image again and again--actually decrease
     efficiency for users who are opening up an thumbnail for the first
     time? In that situation,  I'd have another column for etags in my
     datastore being requested w/ every query.

     Yes, you absolutely should generate the etag when you save the
     thumbnail, and save it in the model itself. Caching it separately
 is
     however still desirable as you can then avoid pulling the rest of
 the
     data into memory if it's not needed, or you can opt to not cache
 the
     rest of the data at all, instead only caching the etag, to be more
     cache friendly.

 A quick and easy hack for this is to generate the etag before creating
 the Thumbnail model instance - and use that etag as the named key.
 Then, you can do lookup and caching based on the etag alone, where
 that makes sense. Unless you have some specific meaning in your ID
 already, this should simplify the how to deal with etags question
 quite a bit.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---



[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-09-25 Thread Thomas Johansson

At Google IO I believe it was mentioned that TextProperty's are fine
for anything that you don't want indexed; They are no less efficient
than strings for short data. You might want to look into trying that
out with mime and type if you don't filter/order by them.

Other than that, I'd suggest trying to memcache the images, something
to the effect of:

id = int(id)
cache_key = 'ImageThumb:%i' % id
image = memcache.get(cache_key)
if not image:
image = ImageThumb.get_by_id(id)
if image:
memcache.set(cache_key, image, 3600)
if image:
self.response.headers['Content-Type'] = str(image.mime)
self.response.out.write(image.thumb)
else:
self.error(404)

In addition to that, I'd recommend you generate cache headers, in
particular etag and expiration. A good idea would be to generate the
etag when the ImageThumb is created, and then compare against that
from cache, before you write out the image. As an added bonus you
could store the etag on it's own in the cache, separate from the
image, and only pull out the actual image in case the etags don't
match.

On Sep 25, 4:59 am, iceanfire [EMAIL PROTECTED] wrote:
 I'm still having trouble with high-cpu warnings for thumbnails. This
 time around I took some profiler data to see what's causing it. But
 before I get into that here is the model i'm using:

 class ImageThumb(db.Model):
   binId = db.IntegerProperty()
   thumb = db.BlobProperty(default=None)
   building = db.ReferenceProperty(Buildings)
   apartment = db.ReferenceProperty(Apartments)
   mime = db.StringProperty()
   type = db.StringProperty(choices=['Floor Plan','Picture'])
   created_by =  db.UserProperty()

 So here's the Profiler cpu data:
 2538 function calls (2472 primitive calls) in 0.028 CPU seconds (rest
 of the data:http://docs.google.com/Doc?id=dgfxff5_30hc9tjsgg)

 But here's the warning: This request used a high amount of CPU, and
 was roughly 1.3 times over the average request CPU limit. High CPU
 requests have a small quota, and if you exceed this quota, your app
 will be temporarily disabled.

 Here's the megacycle info from the new Admin console: 1339mcycles 7kb

 Here's the code that pulls up the data for thumbnails:
 class Apt_thumb (webapp.RequestHandler):
     def get(self,id):
                 image = ImageThumb.get_by_id(int(id))
                 if image:
                         self.response.headers['Content-Type'] = 
 str(image.mime)
                         self.response.out.write(image.thumb)
                 else:
                         self.error(404)

 The images stored are 4kb each in the datastore. As I said, I had this
 problem earlier, so I had taken out the full image and put that in its
 own Model. Clearly that didn't help.

 Any idea what's causing this high-cpu error  how I can fix it?

 Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---



[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.

2008-09-25 Thread Bryan A. Pendleton


-So are textProperties more efficient than StringProperties
because
they're not indexed?

You'd have to find the talk from Google IO to be sure. I believe
it
was the one about scalability, in the QA section. But yes, that is
my
understanding.


As I understand it, every field that's not a TextProperty or a
BlobProperty are implicitly indexed (this is how all = conditions are
dealt with in queries). So, whenever you write such an object, it will
take longer (because of the index updates).

Another way of thinking about it, is that if you never need to query
on a single value, make it a TextProperty or BlobProperty, if
possible.


-Wouldn't adding etag--while increasing efficiency if I have the
same
users loading the same image again and again--actually decrease
efficiency for users who are opening up an thumbnail for the first
time? In that situation,  I'd have another column for etags in my
datastore being requested w/ every query.

Yes, you absolutely should generate the etag when you save the
thumbnail, and save it in the model itself. Caching it separately
is
however still desirable as you can then avoid pulling the rest of
the
data into memory if it's not needed, or you can opt to not cache
the
rest of the data at all, instead only caching the etag, to be more
cache friendly.


A quick and easy hack for this is to generate the etag before creating
the Thumbnail model instance - and use that etag as the named key.
Then, you can do lookup and caching based on the etag alone, where
that makes sense. Unless you have some specific meaning in your ID
already, this should simplify the how to deal with etags question
quite a bit.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---