[google-appengine] Re: Do you Serialize your Data?
hi brandon I do this with a bunch of models, keep a few properties for search and then dump the bulk of data into a compressed, json text property. Obviously the json part might be overkill but its a lot more flexible going forward if (err, when) you need to add more data (especially nested data) reads and writes are faster, YMMV YMMV b On Jan 17, 4:44 am, "Brandon Wirtz" wrote: > We are looking at restructuring some of our calls, optimizing again for > speed, we are thinking that rather than having a datastore call for a single > cell, we'd serialize things. > > This makes searching suck, and we have had a few issues where serialized and > unserialized data didn't quite match when we were done (because of weird > encoded characters from sites) > > Serializing also locks you in to a single data structure for the most part. > > We are also just considering: > data as a zipped, delimited array > uncompressed, delimited > Double storing, so that we pay twice for writes, and can run analytics, but > so that reads are against the serialized data, and faster and cost less. > > What are anyone's thoughts? Have you had these discussions with your dev > team? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Do you Serialize your Data?
hi brandon compression - not really, at least not with HRD. I remember it being useful in M/S but only for larger strings. I'm using this python property now, it only compresses over a certain size http://atastypixel.com/blog/wp-content/uploads/2011/01/compressible_text_property.py_.txt great article jeff, I know I've read of few of your older posts about this in the past cheers brian On Jan 17, 9:21 am, "Brandon Wirtz" wrote: > You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. > > > > > > > > -Original Message- > From: google-appengine@googlegroups.com > > [mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer > Sent: Tuesday, January 17, 2012 6:10 AM > To: google-appengine@googlegroups.com > Subject: Re: [google-appengine] Re: Do you Serialize your Data? > > In fact, I wrote up an explanation of this 6 months ago: > > http://blog.similarity.com/post/7541938593/how-to-build-an-online-dat... > e-nosql-edition > > Jeff > > On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer wrote: > > I tend to do this a lot to minimize the cost of fetching data; if I > > need all the information about a Person then I get all that info by > > loading a single entity, usually pulled from memcache. Most of my > > analytic processes look like map/reduce so indexing these complex > > structures wouldn't really help. > > > If you're in Javaland you can use Objectify4 and annotate a field with > > @Serialize(zip=true) to get it automatically compressed. I haven't > > compared resulting entity sizes, however. > > > Jeff > > > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz > wrote: > >> Have you weighed Compression vs uncompressed in terms of speed, and cost? > > >> I'll end up testing to see, but I like to know the answers beforehand > >> :-) > > >> -----Original Message----- > >> From: google-appengine@googlegroups.com > >> [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood > >> Sent: Tuesday, January 17, 2012 4:54 AM > >> To: Google App Engine > >> Subject: [google-appengine] Re: Do you Serialize your Data? > > >> hi brandon > > >> I do this with a bunch of models, keep a few properties for search > >> and then dump the bulk of data into a compressed, json text property. > >> Obviously the json part might be overkill but its a lot more flexible > >> going forward if (err, when) you need to add more data (especially > >> nested data) > > >> reads and writes are faster, YMMV > > >> YMMV > >> b > > >> On Jan 17, 4:44 am, "Brandon Wirtz" wrote: > >>> We are looking at restructuring some of our calls, optimizing again > >>> for speed, we are thinking that rather than having a datastore call > >>> for a single cell, we'd serialize things. > > >>> This makes searching suck, and we have had a few issues where > >>> serialized and unserialized data didn't quite match when we were > >>> done (because of weird encoded characters from sites) > > >>> Serializing also locks you in to a single data structure for the > >>> most > >> part. > > >>> We are also just considering: > >>> data as a zipped, delimited array > >>> uncompressed, delimited > >>> Double storing, so that we pay twice for writes, and can run > >>> analytics, but so that reads are against the serialized data, and > >>> faster > >> and cost less. > > >>> What are anyone's thoughts? Have you had these discussions with your > >>> dev team? > > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "Google App Engine" group. > >> To post to this group, send email to google-appengine@googlegroups.com. > >> To unsubscribe from this group, send email to > >> google-appengine+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > >> To post to this group, send email to google-appengine@googlegroups.com. > >> To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > >> For more options, visit this group > >> athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Do you Serialize your Data?
I use my own serialization to communicate with a flash app and also for the datastore. It is binary and gziped and allows for nested items and binary properties. The compression, decompression, serialize and deserialize time is negligible compared to the datastore request time and sending back the data. Btw, you seem to be the smart guy around here, What happens under the hood when list properties are modified, index and composite index wise ? Does it modify added and removed values only ? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Do you Serialize your Data?
Have you weighed Compression vs uncompressed in terms of speed, and cost? I'll end up testing to see, but I like to know the answers beforehand :-) -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood Sent: Tuesday, January 17, 2012 4:54 AM To: Google App Engine Subject: [google-appengine] Re: Do you Serialize your Data? hi brandon I do this with a bunch of models, keep a few properties for search and then dump the bulk of data into a compressed, json text property. Obviously the json part might be overkill but its a lot more flexible going forward if (err, when) you need to add more data (especially nested data) reads and writes are faster, YMMV YMMV b On Jan 17, 4:44 am, "Brandon Wirtz" wrote: > We are looking at restructuring some of our calls, optimizing again > for speed, we are thinking that rather than having a datastore call > for a single cell, we'd serialize things. > > This makes searching suck, and we have had a few issues where > serialized and unserialized data didn't quite match when we were done > (because of weird encoded characters from sites) > > Serializing also locks you in to a single data structure for the most part. > > We are also just considering: > data as a zipped, delimited array > uncompressed, delimited > Double storing, so that we pay twice for writes, and can run > analytics, but so that reads are against the serialized data, and faster and cost less. > > What are anyone's thoughts? Have you had these discussions with your > dev team? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Do you Serialize your Data?
I tend to do this a lot to minimize the cost of fetching data; if I need all the information about a Person then I get all that info by loading a single entity, usually pulled from memcache. Most of my analytic processes look like map/reduce so indexing these complex structures wouldn't really help. If you're in Javaland you can use Objectify4 and annotate a field with @Serialize(zip=true) to get it automatically compressed. I haven't compared resulting entity sizes, however. Jeff On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz wrote: > Have you weighed Compression vs uncompressed in terms of speed, and cost? > > I'll end up testing to see, but I like to know the answers beforehand :-) > > > -Original Message- > From: google-appengine@googlegroups.com > [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood > Sent: Tuesday, January 17, 2012 4:54 AM > To: Google App Engine > Subject: [google-appengine] Re: Do you Serialize your Data? > > hi brandon > > I do this with a bunch of models, keep a few properties for search and then > dump the bulk of data into a compressed, json text property. > Obviously the json part might be overkill but its a lot more flexible going > forward if (err, when) you need to add more data (especially nested data) > > reads and writes are faster, YMMV > > YMMV > b > > On Jan 17, 4:44 am, "Brandon Wirtz" wrote: >> We are looking at restructuring some of our calls, optimizing again >> for speed, we are thinking that rather than having a datastore call >> for a single cell, we'd serialize things. >> >> This makes searching suck, and we have had a few issues where >> serialized and unserialized data didn't quite match when we were done >> (because of weird encoded characters from sites) >> >> Serializing also locks you in to a single data structure for the most > part. >> >> We are also just considering: >> data as a zipped, delimited array >> uncompressed, delimited >> Double storing, so that we pay twice for writes, and can run >> analytics, but so that reads are against the serialized data, and faster > and cost less. >> >> What are anyone's thoughts? Have you had these discussions with your >> dev team? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Do you Serialize your Data?
In fact, I wrote up an explanation of this 6 months ago: http://blog.similarity.com/post/7541938593/how-to-build-an-online-dating-site-nosql-edition Jeff On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer wrote: > I tend to do this a lot to minimize the cost of fetching data; if I > need all the information about a Person then I get all that info by > loading a single entity, usually pulled from memcache. Most of my > analytic processes look like map/reduce so indexing these complex > structures wouldn't really help. > > If you're in Javaland you can use Objectify4 and annotate a field with > @Serialize(zip=true) to get it automatically compressed. I haven't > compared resulting entity sizes, however. > > Jeff > > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz wrote: >> Have you weighed Compression vs uncompressed in terms of speed, and cost? >> >> I'll end up testing to see, but I like to know the answers beforehand :-) >> >> >> -Original Message- >> From: google-appengine@googlegroups.com >> [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood >> Sent: Tuesday, January 17, 2012 4:54 AM >> To: Google App Engine >> Subject: [google-appengine] Re: Do you Serialize your Data? >> >> hi brandon >> >> I do this with a bunch of models, keep a few properties for search and then >> dump the bulk of data into a compressed, json text property. >> Obviously the json part might be overkill but its a lot more flexible going >> forward if (err, when) you need to add more data (especially nested data) >> >> reads and writes are faster, YMMV >> >> YMMV >> b >> >> On Jan 17, 4:44 am, "Brandon Wirtz" wrote: >>> We are looking at restructuring some of our calls, optimizing again >>> for speed, we are thinking that rather than having a datastore call >>> for a single cell, we'd serialize things. >>> >>> This makes searching suck, and we have had a few issues where >>> serialized and unserialized data didn't quite match when we were done >>> (because of weird encoded characters from sites) >>> >>> Serializing also locks you in to a single data structure for the most >> part. >>> >>> We are also just considering: >>> data as a zipped, delimited array >>> uncompressed, delimited >>> Double storing, so that we pay twice for writes, and can run >>> analytics, but so that reads are against the serialized data, and faster >> and cost less. >>> >>> What are anyone's thoughts? Have you had these discussions with your >>> dev team? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Do you Serialize your Data?
You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer Sent: Tuesday, January 17, 2012 6:10 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Do you Serialize your Data? In fact, I wrote up an explanation of this 6 months ago: http://blog.similarity.com/post/7541938593/how-to-build-an-online-dating-sit e-nosql-edition Jeff On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer wrote: > I tend to do this a lot to minimize the cost of fetching data; if I > need all the information about a Person then I get all that info by > loading a single entity, usually pulled from memcache. Most of my > analytic processes look like map/reduce so indexing these complex > structures wouldn't really help. > > If you're in Javaland you can use Objectify4 and annotate a field with > @Serialize(zip=true) to get it automatically compressed. I haven't > compared resulting entity sizes, however. > > Jeff > > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz wrote: >> Have you weighed Compression vs uncompressed in terms of speed, and cost? >> >> I'll end up testing to see, but I like to know the answers beforehand >> :-) >> >> >> -Original Message- >> From: google-appengine@googlegroups.com >> [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood >> Sent: Tuesday, January 17, 2012 4:54 AM >> To: Google App Engine >> Subject: [google-appengine] Re: Do you Serialize your Data? >> >> hi brandon >> >> I do this with a bunch of models, keep a few properties for search >> and then dump the bulk of data into a compressed, json text property. >> Obviously the json part might be overkill but its a lot more flexible >> going forward if (err, when) you need to add more data (especially >> nested data) >> >> reads and writes are faster, YMMV >> >> YMMV >> b >> >> On Jan 17, 4:44 am, "Brandon Wirtz" wrote: >>> We are looking at restructuring some of our calls, optimizing again >>> for speed, we are thinking that rather than having a datastore call >>> for a single cell, we'd serialize things. >>> >>> This makes searching suck, and we have had a few issues where >>> serialized and unserialized data didn't quite match when we were >>> done (because of weird encoded characters from sites) >>> >>> Serializing also locks you in to a single data structure for the >>> most >> part. >>> >>> We are also just considering: >>> data as a zipped, delimited array >>> uncompressed, delimited >>> Double storing, so that we pay twice for writes, and can run >>> analytics, but so that reads are against the serialized data, and >>> faster >> and cost less. >>> >>> What are anyone's thoughts? Have you had these discussions with your >>> dev team? >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Do you Serialize your Data?
Round one of my testing says that Serializing via JSON, or Pickle is SLOWER than just doing the reads from the datastore. I'm reading up on model_to_protobuf Which is supposed to be faster. I'm a little sad. It would be awesome if there was a .put(key,serialize(stuff_i_would_write) -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood Sent: Tuesday, January 17, 2012 6:52 AM To: Google App Engine Subject: [google-appengine] Re: Do you Serialize your Data? hi brandon compression - not really, at least not with HRD. I remember it being useful in M/S but only for larger strings. I'm using this python property now, it only compresses over a certain size http://atastypixel.com/blog/wp-content/uploads/2011/01/compressible_text_pro perty.py_.txt great article jeff, I know I've read of few of your older posts about this in the past cheers brian On Jan 17, 9:21 am, "Brandon Wirtz" wrote: > You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. > > > > > > > > -Original Message- > From: google-appengine@googlegroups.com > > [mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer > Sent: Tuesday, January 17, 2012 6:10 AM > To: google-appengine@googlegroups.com > Subject: Re: [google-appengine] Re: Do you Serialize your Data? > > In fact, I wrote up an explanation of this 6 months ago: > > http://blog.similarity.com/post/7541938593/how-to-build-an-online-dat... > e-nosql-edition > > Jeff > > On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer wrote: > > I tend to do this a lot to minimize the cost of fetching data; if I > > need all the information about a Person then I get all that info by > > loading a single entity, usually pulled from memcache. Most of my > > analytic processes look like map/reduce so indexing these complex > > structures wouldn't really help. > > > If you're in Javaland you can use Objectify4 and annotate a field > > with > > @Serialize(zip=true) to get it automatically compressed. I haven't > > compared resulting entity sizes, however. > > > Jeff > > > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz > wrote: > >> Have you weighed Compression vs uncompressed in terms of speed, and cost? > > >> I'll end up testing to see, but I like to know the answers > >> beforehand > >> :-) > > >> -Original Message- > >> From: google-appengine@googlegroups.com > >> [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood > >> Sent: Tuesday, January 17, 2012 4:54 AM > >> To: Google App Engine > >> Subject: [google-appengine] Re: Do you Serialize your Data? > > >> hi brandon > > >> I do this with a bunch of models, keep a few properties for search > >> and then dump the bulk of data into a compressed, json text property. > >> Obviously the json part might be overkill but its a lot more > >> flexible going forward if (err, when) you need to add more data > >> (especially nested data) > > >> reads and writes are faster, YMMV > > >> YMMV > >> b > > >> On Jan 17, 4:44 am, "Brandon Wirtz" wrote: > >>> We are looking at restructuring some of our calls, optimizing > >>> again for speed, we are thinking that rather than having a > >>> datastore call for a single cell, we'd serialize things. > > >>> This makes searching suck, and we have had a few issues where > >>> serialized and unserialized data didn't quite match when we were > >>> done (because of weird encoded characters from sites) > > >>> Serializing also locks you in to a single data structure for the > >>> most > >> part. > > >>> We are also just considering: > >>> data as a zipped, delimited array > >>> uncompressed, delimited > >>> Double storing, so that we pay twice for writes, and can run > >>> analytics, but so that reads are against the serialized data, and > >>> faster > >> and cost less. > > >>> What are anyone's thoughts? Have you had these discussions with > >>> your dev team? > > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "Google App Engine" group. > >> To post to this group, send email to google-appengine@googlegroups.com. > >> To unsubscribe from this group, send email to > >> google-appengine+unsubscr...@googlegroups.com. > >> For mor
Re: [google-appengine] Re: Do you Serialize your Data?
Hi Brandon, On Thu, Jan 19, 2012 at 11:25 AM, Brandon Wirtz wrote: > Round one of my testing says that Serializing via JSON, or Pickle is SLOWER > than just doing the reads from the datastore. This might be different in the Python 2.7 runtime, where cPickle and C-accelerated JSON are available. Cheers, Brian > I'm reading up on model_to_protobuf > Which is supposed to be faster. > > I'm a little sad. It would be awesome if there was a > .put(key,serialize(stuff_i_would_write) > > > > > > -Original Message- > From: google-appengine@googlegroups.com > [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood > Sent: Tuesday, January 17, 2012 6:52 AM > To: Google App Engine > Subject: [google-appengine] Re: Do you Serialize your Data? > > hi brandon > > compression - not really, at least not with HRD. I remember it being useful > in M/S but only for larger strings. I'm using this python property now, it > only compresses over a certain size > http://atastypixel.com/blog/wp-content/uploads/2011/01/compressible_text_pro > perty.py_.txt > > great article jeff, I know I've read of few of your older posts about this > in the past > > cheers > brian > > > On Jan 17, 9:21 am, "Brandon Wirtz" wrote: >> You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. >> >> >> >> >> >> >> >> -Original Message- >> From: google-appengine@googlegroups.com >> >> [mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer >> Sent: Tuesday, January 17, 2012 6:10 AM >> To: google-appengine@googlegroups.com >> Subject: Re: [google-appengine] Re: Do you Serialize your Data? >> >> In fact, I wrote up an explanation of this 6 months ago: >> >> http://blog.similarity.com/post/7541938593/how-to-build-an-online-dat... >> e-nosql-edition >> >> Jeff >> >> On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer > wrote: >> > I tend to do this a lot to minimize the cost of fetching data; if I >> > need all the information about a Person then I get all that info by >> > loading a single entity, usually pulled from memcache. Most of my >> > analytic processes look like map/reduce so indexing these complex >> > structures wouldn't really help. >> >> > If you're in Javaland you can use Objectify4 and annotate a field >> > with >> > @Serialize(zip=true) to get it automatically compressed. I haven't >> > compared resulting entity sizes, however. >> >> > Jeff >> >> > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz >> wrote: >> >> Have you weighed Compression vs uncompressed in terms of speed, and > cost? >> >> >> I'll end up testing to see, but I like to know the answers >> >> beforehand >> >> :-) >> >> >> -Original Message- >> >> From: google-appengine@googlegroups.com >> >> [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood >> >> Sent: Tuesday, January 17, 2012 4:54 AM >> >> To: Google App Engine >> >> Subject: [google-appengine] Re: Do you Serialize your Data? >> >> >> hi brandon >> >> >> I do this with a bunch of models, keep a few properties for search >> >> and then dump the bulk of data into a compressed, json text property. >> >> Obviously the json part might be overkill but its a lot more >> >> flexible going forward if (err, when) you need to add more data >> >> (especially nested data) >> >> >> reads and writes are faster, YMMV >> >> >> YMMV >> >> b >> >> >> On Jan 17, 4:44 am, "Brandon Wirtz" wrote: >> >>> We are looking at restructuring some of our calls, optimizing >> >>> again for speed, we are thinking that rather than having a >> >>> datastore call for a single cell, we'd serialize things. >> >> >>> This makes searching suck, and we have had a few issues where >> >>> serialized and unserialized data didn't quite match when we were >> >>> done (because of weird encoded characters from sites) >> >> >>> Serializing also locks you in to a single data structure for the >> >>> most >> >> part. >> >> >>> We are also just considering: >> >>> data as a zipped, delimited array >> >>> uncompressed, delimited >> >>> Double storing, so that we p
RE: [google-appengine] Re: Do you Serialize your Data?
Ah... Working off old docs that said cPickle didn't work so I didn't even try it. I'll try that. I moved on to put_async which looks like other low hanging fruit for my app. Especially since Cache Misses are already the slowest of my pages, (and cache miss = write) -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Brian Quinlan Sent: Wednesday, January 18, 2012 4:33 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Do you Serialize your Data? Hi Brandon, On Thu, Jan 19, 2012 at 11:25 AM, Brandon Wirtz wrote: > Round one of my testing says that Serializing via JSON, or Pickle is > SLOWER than just doing the reads from the datastore. This might be different in the Python 2.7 runtime, where cPickle and C-accelerated JSON are available. Cheers, Brian > I'm reading up on model_to_protobuf > Which is supposed to be faster. > > I'm a little sad. It would be awesome if there was a > .put(key,serialize(stuff_i_would_write) > > > > > > -Original Message- > From: google-appengine@googlegroups.com > [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood > Sent: Tuesday, January 17, 2012 6:52 AM > To: Google App Engine > Subject: [google-appengine] Re: Do you Serialize your Data? > > hi brandon > > compression - not really, at least not with HRD. I remember it being > useful in M/S but only for larger strings. I'm using this python > property now, it only compresses over a certain size > http://atastypixel.com/blog/wp-content/uploads/2011/01/compressible_te > xt_pro > perty.py_.txt > > great article jeff, I know I've read of few of your older posts about > this in the past > > cheers > brian > > > On Jan 17, 9:21 am, "Brandon Wirtz" wrote: >> You wrote "How to Use GAE to Get Laid" and didn't tell anyone? Evil. >> >> >> >> >> >> >> >> -Original Message- >> From: google-appengine@googlegroups.com >> >> [mailto:google-appengine@googlegroups.com] On Behalf Of Jeff >> Schnitzer >> Sent: Tuesday, January 17, 2012 6:10 AM >> To: google-appengine@googlegroups.com >> Subject: Re: [google-appengine] Re: Do you Serialize your Data? >> >> In fact, I wrote up an explanation of this 6 months ago: >> >> http://blog.similarity.com/post/7541938593/how-to-build-an-online-dat... >> e-nosql-edition >> >> Jeff >> >> On Tue, Jan 17, 2012 at 2:06 PM, Jeff Schnitzer > wrote: >> > I tend to do this a lot to minimize the cost of fetching data; if I >> > need all the information about a Person then I get all that info by >> > loading a single entity, usually pulled from memcache. Most of my >> > analytic processes look like map/reduce so indexing these complex >> > structures wouldn't really help. >> >> > If you're in Javaland you can use Objectify4 and annotate a field >> > with >> > @Serialize(zip=true) to get it automatically compressed. I haven't >> > compared resulting entity sizes, however. >> >> > Jeff >> >> > On Tue, Jan 17, 2012 at 1:27 PM, Brandon Wirtz >> > >> wrote: >> >> Have you weighed Compression vs uncompressed in terms of speed, >> >> and > cost? >> >> >> I'll end up testing to see, but I like to know the answers >> >> beforehand >> >> :-) >> >> >> -Original Message- >> >> From: google-appengine@googlegroups.com >> >> [mailto:google-appengine@googlegroups.com] On Behalf Of bFlood >> >> Sent: Tuesday, January 17, 2012 4:54 AM >> >> To: Google App Engine >> >> Subject: [google-appengine] Re: Do you Serialize your Data? >> >> >> hi brandon >> >> >> I do this with a bunch of models, keep a few properties for search >> >> and then dump the bulk of data into a compressed, json text property. >> >> Obviously the json part might be overkill but its a lot more >> >> flexible going forward if (err, when) you need to add more data >> >> (especially nested data) >> >> >> reads and writes are faster, YMMV >> >> >> YMMV >> >> b >> >> >> On Jan 17, 4:44 am, "Brandon Wirtz" wrote: >> >>> We are looking at restructuring some of our calls, optimizing >> >>> again for speed, we are thinking that rather than having a >> >>> datastore call for a single cell, we'd serialize things. &g
Re: [google-appengine] Re: Do you Serialize your Data?
i started pre-serializing my objects with an objectproperty which writes to the blobstore on put . while serializing 50 entities with around 10 properties each on request took around 5-6 seconds as an average, now i cut the time in half at least. On Jan 18, 2012, at 8:19 PM, Mahron wrote: > I use my own serialization to communicate with a flash app and also > for the datastore. > > It is binary and gziped and allows for nested items and binary > properties. > > The compression, decompression, serialize and deserialize time is > negligible compared to the datastore request time and sending back the > data. > > Btw, you seem to be the smart guy around here, > > What happens under the hood when list properties are modified, index > and composite index wise ? > Does it modify added and removed values only ? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Do you Serialize your Data?
I opted for cpickle. My Slowness the first pass was that I was testing with 250k binaries and Pickle escapes them... So they were very large and didn't actually fit (I didn't check they matched going in and coming out) Once I applied ZIP to the Pickle it appears to be fast. Which brings me to the next question... I use datastore like many people would use Memcache and store with date. Does anyone want to place bets on how much faster a 500 byte file reads than a 100k file? I'm thinking about storing the last modified date and checking it, before I read the actual data. But I need to bench the difference in speed an calculate for the cache miss ratio. But it would appear that with ASync Writes, Async reads for Initialization, and serialization my average latency is going to go from 600 MS to 185ms and knock another 50-60% off of my hosting bill. That will put me to less than the price I was at in beta. (in terms of price per traffic served) (and google bot is still the most expensive part of that) -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Andreas Sent: Wednesday, January 18, 2012 6:36 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Do you Serialize your Data? i started pre-serializing my objects with an objectproperty which writes to the blobstore on put . while serializing 50 entities with around 10 properties each on request took around 5-6 seconds as an average, now i cut the time in half at least. On Jan 18, 2012, at 8:19 PM, Mahron wrote: > I use my own serialization to communicate with a flash app and also > for the datastore. > > It is binary and gziped and allows for nested items and binary > properties. > > The compression, decompression, serialize and deserialize time is > negligible compared to the datastore request time and sending back the > data. > > Btw, you seem to be the smart guy around here, > > What happens under the hood when list properties are modified, index > and composite index wise ? > Does it modify added and removed values only ? > > -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Do you Serialize your Data?
Pickle is much faster if you set Highest Profile rather than taking the default. Something about compatibility... Any how... Also Zlib.compress on 1 is a lot faster than 6, and there was minimal difference in the size of the data after. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Brandon Wirtz Sent: Wednesday, January 18, 2012 10:35 PM To: google-appengine@googlegroups.com Subject: RE: [google-appengine] Re: Do you Serialize your Data? I opted for cpickle. My Slowness the first pass was that I was testing with 250k binaries and Pickle escapes them... So they were very large and didn't actually fit (I didn't check they matched going in and coming out) Once I applied ZIP to the Pickle it appears to be fast. Which brings me to the next question... I use datastore like many people would use Memcache and store with date. Does anyone want to place bets on how much faster a 500 byte file reads than a 100k file? I'm thinking about storing the last modified date and checking it, before I read the actual data. But I need to bench the difference in speed an calculate for the cache miss ratio. But it would appear that with ASync Writes, Async reads for Initialization, and serialization my average latency is going to go from 600 MS to 185ms and knock another 50-60% off of my hosting bill. That will put me to less than the price I was at in beta. (in terms of price per traffic served) (and google bot is still the most expensive part of that) -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Andreas Sent: Wednesday, January 18, 2012 6:36 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Do you Serialize your Data? i started pre-serializing my objects with an objectproperty which writes to the blobstore on put . while serializing 50 entities with around 10 properties each on request took around 5-6 seconds as an average, now i cut the time in half at least. On Jan 18, 2012, at 8:19 PM, Mahron wrote: > I use my own serialization to communicate with a flash app and also > for the datastore. > > It is binary and gziped and allows for nested items and binary > properties. > > The compression, decompression, serialize and deserialize time is > negligible compared to the datastore request time and sending back the > data. > > Btw, you seem to be the smart guy around here, > > What happens under the hood when list properties are modified, index > and composite index wise ? > Does it modify added and removed values only ? > > -- > You received this message because you are subscribed to the Google > Groups "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
