hi brandon compression - not really, at least not with HRD. I remember it being useful in M/S but only for larger strings. I'm using this python property now, it only compresses over a certain size http://atastypixel.com/blog/wp-content/uploads/2011/01/compressible_text_property.py_.txt
great article jeff, I know I've read of few of your older posts about this in the past cheers brian On Jan 17, 9:21 am, "Brandon Wirtz" <drak...@digerat.com> wrote: > You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. > > > > > > > > -----Original Message----- > From: google-appengine@googlegroups.com > > [mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer > Sent: Tuesday, January 17, 2012 6:10 AM > To: google-appengine@googlegroups.com > Subject: Re: [google-appengine] Re: Do you Serialize your Data? > > In fact, I wrote up an explanation of this 6 months ago: > > http://blog.similarity.com/post/7541938593/how-to-build-an-online-dat... > e-nosql-edition > > Jeff > > On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer <j...@infohazard.org> wrote: > > I tend to do this a lot to minimize the cost of fetching data; if I > > need all the information about a Person then I get all that info by > > loading a single entity, usually pulled from memcache. Most of my > > analytic processes look like map/reduce so indexing these complex > > structures wouldn't really help. > > > If you're in Javaland you can use Objectify4 and annotate a field with > > @Serialize(zip=true) to get it automatically compressed. I haven't > > compared resulting entity sizes, however. > > > Jeff > > > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz <drak...@digerat.com> > wrote: > >> Have you weighed Compression vs uncompressed in terms of speed, and cost? > > >> I'll end up testing to see, but I like to know the answers beforehand > >> :-) > > >> -----Original Message----- > >> From: google-appengine@googlegroups.com > >> [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood > >> Sent: Tuesday, January 17, 2012 4:54 AM > >> To: Google App Engine > >> Subject: [google-appengine] Re: Do you Serialize your Data? > > >> hi brandon > > >> I do this with a bunch of models, keep a few properties for search > >> and then dump the bulk of data into a compressed, json text property. > >> Obviously the json part might be overkill but its a lot more flexible > >> going forward if (err, when) you need to add more data (especially > >> nested data) > > >> reads and writes are faster, YMMV > > >> YMMV > >> b > > >> On Jan 17, 4:44 am, "Brandon Wirtz" <drak...@digerat.com> wrote: > >>> We are looking at restructuring some of our calls, optimizing again > >>> for speed, we are thinking that rather than having a datastore call > >>> for a single cell, we'd serialize things. > > >>> This makes searching suck, and we have had a few issues where > >>> serialized and unserialized data didn't quite match when we were > >>> done (because of weird encoded characters from sites) > > >>> Serializing also locks you in to a single data structure for the > >>> most > >> part. > > >>> We are also just considering: > >>> data as a zipped, delimited array > >>> uncompressed, delimited > >>> Double storing, so that we pay twice for writes, and can run > >>> analytics, but so that reads are against the serialized data, and > >>> faster > >> and cost less. > > >>> What are anyone's thoughts? Have you had these discussions with your > >>> dev team? > > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "Google App Engine" group. > >> To post to this group, send email to google-appengine@googlegroups.com. > >> To unsubscribe from this group, send email to > >> google-appengine+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > >> To post to this group, send email to google-appengine@googlegroups.com. > >> To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > >> For more options, visit this group > >> athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
