Re: [google-appengine] Re: fastest way to serialize db.Model(s) to json?
so i opted for a custom solution where i pre-serialize the object into a dictionary. i added an objectproperty to the model which stores the dictionary to the blobstore. now i dont need to serialize the object(s) on request any more because i only need to access this single property. with this solution i got an approximately 50% speed improvement. On Jan 17, 2012, at 3:14 PM, Matthew Blain wrote: > the most concise code is probably db.to_dict. this is roughly the same > code path used in db.model_to_protobuf > > On Jan 17, 9:37 am, Andreas wrote: >> so i tried to port my app to python 2.7 but atm it is too complex to do and >> i really dont think there will be a massive speed improvement. >> think ill pre-serialize my objects (somehow like gregory suggested) which >> should bring the serialization down a few seconds. >> >> On Jan 16, 2012, at 8:46 PM, Kyle Finley wrote: >> >> >> >> >> >> >> >>> aschmid, >> >>> Are you running python27? If so you should see some improvement by >>> importing json instead of simplejson. >> >>> http://stackoverflow.com/a/8191970/236564 >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To view this discussion on the web >>> visithttps://groups.google.com/d/msg/google-appengine/-/GFQ13LeHVP4J. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group >>> athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: fastest way to serialize db.Model(s) to json?
the most concise code is probably db.to_dict. this is roughly the same code path used in db.model_to_protobuf On Jan 17, 9:37 am, Andreas wrote: > so i tried to port my app to python 2.7 but atm it is too complex to do and i > really dont think there will be a massive speed improvement. > think ill pre-serialize my objects (somehow like gregory suggested) which > should bring the serialization down a few seconds. > > On Jan 16, 2012, at 8:46 PM, Kyle Finley wrote: > > > > > > > > > aschmid, > > > Are you running python27? If so you should see some improvement by > > importing json instead of simplejson. > > >http://stackoverflow.com/a/8191970/236564 > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To view this discussion on the web > > visithttps://groups.google.com/d/msg/google-appengine/-/GFQ13LeHVP4J. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: fastest way to serialize db.Model(s) to json?
so i tried to port my app to python 2.7 but atm it is too complex to do and i really dont think there will be a massive speed improvement. think ill pre-serialize my objects (somehow like gregory suggested) which should bring the serialization down a few seconds. On Jan 16, 2012, at 8:46 PM, Kyle Finley wrote: > aschmid, > > Are you running python27? If so you should see some improvement by importing > json instead of simplejson. > > http://stackoverflow.com/a/8191970/236564 > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/GFQ13LeHVP4J. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: fastest way to serialize db.Model(s) to json?
aschmid, Are you running python27? If so you should see some improvement by importing *json* instead of *simplejson*. http://stackoverflow.com/a/8191970/236564 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/GFQ13LeHVP4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: fastest way to serialize db.Model(s) to json?
yes i did for my rpc calls im actually handling myself atm. i didn't look at how it serializes objects but i doubt it does something different than looping over the properties and serializing them like my other function does but i might be wrong. ill check it out tomorrow i think. On Jan 16, 2012, at 6:50 PM, alex wrote: > aschmid, this might be useless to you but if it's an API-related thing you're > serializing, have you considered ProtoRPC? > http://code.google.com/appengine/docs/python/tools/protorpc/overview.html > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/Io3crXFlGdsJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: fastest way to serialize db.Model(s) to json?
aschmid, this might be useless to you but if it's an API-related thing you're serializing, have you considered ProtoRPC? http://code.google.com/appengine/docs/python/tools/protorpc/overview.html -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Io3crXFlGdsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: fastest way to serialize db.Model(s) to json?
cant really believe this is the only solution. serializing objects is not an
uncommon thing to do.
if this is the best and quickest way to serialize objects then i might really
store not only a list of properties but the completely serialized object into a
dictionary in the blobstore.
On Jan 10, 2012, at 3:09 PM, Gregory Nicholas wrote:
> if it's as bad as 6 to 8 seconds, have you looked at creating a list
> of string for the property names? i know its more maintenance
> overhead, but might be worth the tradeoff..
>
> ex:
>
> class SerializableModel(object):
>
> serializable = []
>
> def to_json(self):
>result = []
>for x in self.serializable:
> result.append(self.__dict__[x])
>## append values from self.properties().items()
>
>
> class A(db.Model, SerializableModel):
> serializable = ['user_keys']
> _user_keys = db.StringListProperty(default=[], indexed=False)
>
> @property
> def user_keys(self):
>return [db.Key(x) for x in self._user_keys]
>
> @property
> def user_keys_count(self):
>"""not in serializable list"""
>return len(self._user_keys)
>
>
>
>
> On Jan 10, 9:05 am, Andreas wrote:
>> hi,
>>
>> im using more or less this
>> (http://www.awebcoder.com/post/91001/extended-jsonify-function-for-app...)
>> jsonify function to serialize whatever object i pass into the jsonify()
>> function and i have to say it does all i need and works pretty good.
>> the only downside i see right now is the speed. if i try to serialize 50
>> db.Model objects with around 10 properties each in production it takes 6-8
>> seconds which is not exactly what i would expect.
>>
>> is there a way to make this part of code more performant?
>> for property in dir(obj):
>> if property[0] != '_':
>> try:
>> value = obj.__getattribute__(property)
>> valueClass = str(value.__class__)
>> if not(('function' in valueClass) or ('built' in valueClass)
>> or ('method' in valueClass)):
>> value = dumps(value)
>> if value != None:
>> properties[property] = value
>> else:
>> properties[property] = ""
>> except: continue
>> if len(properties) == 0:
>> return str(obj)
>> else:
>> return properties
>>
>> i know i could use obj.properties().items() to loop over the properties but
>> i would not be able to serialize properties which are @property functions
>> because those don't get listed in the obj.properties() list.
>>
>> right now im caching the results so im gaining speed because i get the
>> result from memcache but when i flush the cache or the memcached result is
>> not available it takes too much until the result is ready.
>>
>> thx
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: fastest way to serialize db.Model(s) to json?
if it's as bad as 6 to 8 seconds, have you looked at creating a list
of string for the property names? i know its more maintenance
overhead, but might be worth the tradeoff..
ex:
class SerializableModel(object):
serializable = []
def to_json(self):
result = []
for x in self.serializable:
result.append(self.__dict__[x])
## append values from self.properties().items()
class A(db.Model, SerializableModel):
serializable = ['user_keys']
_user_keys = db.StringListProperty(default=[], indexed=False)
@property
def user_keys(self):
return [db.Key(x) for x in self._user_keys]
@property
def user_keys_count(self):
"""not in serializable list"""
return len(self._user_keys)
On Jan 10, 9:05 am, Andreas wrote:
> hi,
>
> im using more or less this
> (http://www.awebcoder.com/post/91001/extended-jsonify-function-for-app...)
> jsonify function to serialize whatever object i pass into the jsonify()
> function and i have to say it does all i need and works pretty good.
> the only downside i see right now is the speed. if i try to serialize 50
> db.Model objects with around 10 properties each in production it takes 6-8
> seconds which is not exactly what i would expect.
>
> is there a way to make this part of code more performant?
> for property in dir(obj):
> if property[0] != '_':
> try:
> value = obj.__getattribute__(property)
> valueClass = str(value.__class__)
> if not(('function' in valueClass) or ('built' in valueClass)
> or ('method' in valueClass)):
> value = dumps(value)
> if value != None:
> properties[property] = value
> else:
> properties[property] = ""
> except: continue
> if len(properties) == 0:
> return str(obj)
> else:
> return properties
>
> i know i could use obj.properties().items() to loop over the properties but i
> would not be able to serialize properties which are @property functions
> because those don't get listed in the obj.properties() list.
>
> right now im caching the results so im gaining speed because i get the result
> from memcache but when i flush the cache or the memcached result is not
> available it takes too much until the result is ready.
>
> thx
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
