[google-appengine] Re: Intercepting Model Creation

[Kugutsumen]

Here is another way to check if an instance is saved without
checking _from_entity.

from google.appengine.ext import db
import types

class A(db.Model):  # works with Expando too
name = db.StringProperty(required=True)

def __init__(self, _is_new=True, key_name=None, _app=None,
   _from_entity=False, parent=None, **kwds):
if not parent and type(_is_new) is not types.BooleanType:
parent = _is_new
print # before __init__: is new %s % bool(_is_new)
super(db.class_for_kind(self.kind()), self).__init__(parent,
key_name, _app, _from_entity, **kwds)
print # after __init__: is new %s % bool(_is_new)

a = A(key_name=ka,name=a)
# before __init__: is new True
# after __init__: is new True
a.put()

b = A(parent=a,key_name=kb, name=b)
# before __init__: is new True
# after __init__: is new True

c = A.all()[0]
# before __init__: is new False
# after __init__: is new False

c = A.get(a.key())
# before __init__: is new False
# after __init__: is new False

c = A.get_by_key_name(a.key().name())
# before __init__: is new False
# after __init__: is new False

When you try to get model instances using db.Model.get,
Model.get_by_key_name, etc. db.get(keys) will be called to get the
entities from the datastore and convert them to models instances using
the class method Model.from_entity(entity). Query fetches also use
from_entity to return model instances.

In from_entity, the model instance is called this way: cls(None,
_from_entity=True, **entity_values). It passes None to the parent
argument; when you retrieve an entity
parent and key_name are never passed to the model's __init__  so it is
safe
to grab it with _is_new and make the default True.

Now that I wasted more time with this ad-hockery, I am going to stick
with the _from_entity method which is much cleaner.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Intercepting Model Creation

[Kugutsumen]

On Mar 23, 11:27 am, Kugutsumen kugutsu...@gmail.com wrote:
 Same problem here; I starred 844.

 In the meantime if I need to check within __init__ if an entity is
 already
 in the datastore I use this ugly workaround:

 bool(self.is_saved() or kwds.has_key('_from_entity') and kwds
 ['_from_entity'])

There is a typo, to test if an entity is in the datastore in the
__init__ method, use:

if self.is_saved() or _from_entity:

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Intercepting Model Creation

[Andy Freeman]

Another idea is to add a keyword argument to your instance creation
code (not __init__, but the code that calls it) and use that keyword
argument in your __init__ (which is called after theirs) to decide
whether to call Load.  You'll also want to call Load in your from-
memcache code (unless you let memcache store referenced instances).

On Feb 16, 6:36 pm, RobertG dr.robert.gard...@gmail.com wrote:
 Short version:
 I want to execute code during creation of a Model subclass but I need
 to know whether or not the object is saved in the datastore and/or
 execute datastore queries. Since Model.is_saved() never returns True
 inside __init__(), overriding this method doesn't work. I also tried
 intercepting construction with a metaclass, and numerous other
 approaches, but nothing I've tried works. From examining the source
 code it appears that the Model objects aren't fully constructed until
 after get() returns, but there doesn't seem to be a convenient,
 documented location to put my intercept code.

 My specific situation is that I want to replace a ReferenceProperty
 with a value I have stored in a cache to avoid a trip to the
 datastore.

 Here's an example:
 class A(db.Model):
   str = db.StringProperty()

 class B(db.Model):
   a = db.ReferenceProperty(A)

 I want the following code to succeed (assuming B is stored in
 datastore):
 a = from_cache(a_key)
 b = B.get(key)
 assert a == b.a

 I can populate B.a with the proper value using something like the
 following in B:
 def Load(self):
   a_key = B.a.get_value_for_datastore(self)
   self.a = from_cache(a_key)

 But I can't figure out where to put the call to Load().

 Long version:
 The reason I'm trying to do this is for performance. I have some very
 long-lived objects that use ReferenceProperty to manage relationships
 between them, plus a large number of frequently-created objects that
 use ReferenceProperty to refer to the long-lived objects. After many
 DeadlineExceededErrors I started using memcache to store the long-
 lived objects. That helped but not enough. Profiling showed that most
 of my time was now being spent in memcache, apparently because
 memcache goes to the network on every get() call. So I introduced an
 in-memory cache in front of memcache and now first search in-memory,
 then in memcache, then go to the datastore.

 That helped a lot and I'm now avoiding the DeadlineExceededErrors but
 many of my pages still take many seconds to load. Further profiling
 pointed out that the ReferenceProperty attributes (primarily in my
 frequently-created objects that refer to my long-lived objects) are
 hitting the datastore and causing most of my performance problems.
 That's the problem I'm trying to solve -- I want to be able to load
 all those objects and have them use the cached long-lived objects
 instead of going to the datastore to get them.

 I thought this would be a common problem but I haven't been able to
 find anyone else addressing this. Either I'm not looking in the right
 place, people haven't addressed this yet, those who have are not
 sharing, or my data model is screwed up and doesn't work with the
 AppEngine datastore very well!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Intercepting Model Creation

[Kugutsumen]

Same problem here; I starred 844.

In the meantime if I need to check within __init__ if an entity is
already
in the datastore I use this ugly workaround:

bool(self.is_saved() or kwds.has_key('_from_entity') and kwds
['_from_entity'])

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Intercepting Model Creation

[Andy Freeman]

Please star http://code.google.com/p/googleappengine/issues/detail?id=844
.

The fix is for the db.get code to pass in __entity during model
creation time so db.Model.__init__ can set it.

On Feb 16, 5:36 pm, RobertG dr.robert.gard...@gmail.com wrote:
 Short version:
 I want to execute code during creation of a Model subclass but I need
 to know whether or not the object is saved in the datastore and/or
 execute datastore queries. Since Model.is_saved() never returns True
 inside __init__(), overriding this method doesn't work. I also tried
 intercepting construction with a metaclass, and numerous other
 approaches, but nothing I've tried works. From examining the source
 code it appears that the Model objects aren't fully constructed until
 after get() returns, but there doesn't seem to be a convenient,
 documented location to put my intercept code.

 My specific situation is that I want to replace a ReferenceProperty
 with a value I have stored in a cache to avoid a trip to the
 datastore.

 Here's an example:
 class A(db.Model):
   str = db.StringProperty()

 class B(db.Model):
   a = db.ReferenceProperty(A)

 I want the following code to succeed (assuming B is stored in
 datastore):
 a = from_cache(a_key)
 b = B.get(key)
 assert a == b.a

 I can populate B.a with the proper value using something like the
 following in B:
 def Load(self):
   a_key = B.a.get_value_for_datastore(self)
   self.a = from_cache(a_key)

 But I can't figure out where to put the call to Load().

 Long version:
 The reason I'm trying to do this is for performance. I have some very
 long-lived objects that use ReferenceProperty to manage relationships
 between them, plus a large number of frequently-created objects that
 use ReferenceProperty to refer to the long-lived objects. After many
 DeadlineExceededErrors I started using memcache to store the long-
 lived objects. That helped but not enough. Profiling showed that most
 of my time was now being spent in memcache, apparently because
 memcache goes to the network on every get() call. So I introduced an
 in-memory cache in front of memcache and now first search in-memory,
 then in memcache, then go to the datastore.

 That helped a lot and I'm now avoiding the DeadlineExceededErrors but
 many of my pages still take many seconds to load. Further profiling
 pointed out that the ReferenceProperty attributes (primarily in my
 frequently-created objects that refer to my long-lived objects) are
 hitting the datastore and causing most of my performance problems.
 That's the problem I'm trying to solve -- I want to be able to load
 all those objects and have them use the cached long-lived objects
 instead of going to the datastore to get them.

 I thought this would be a common problem but I haven't been able to
 find anyone else addressing this. Either I'm not looking in the right
 place, people haven't addressed this yet, those who have are not
 sharing, or my data model is screwed up and doesn't work with the
 AppEngine datastore very well!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---