Re: [appengine-java] Re: Number of writes per second limitation
5 put() RPCs plus one more for the commit, although you could use a batch put to turn this into 1 put() RPC plus one more for the commit. These are implementation details though. We may at some point just hold on to the entire mutation until commit time (easy if you're not asking us to generate ids for any of your entities, harder if you are since we would then need to maintain a client-side cache of ids to avoid RPCs just for id-generation). -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/qSsxcAlgC9QJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Number of writes per second limitation
It is pseudo code. To be detailed, you have to store it as byte array inside a Blob. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/NMRkT87UhV0J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Number of writes per second limitation
On Sun, Nov 6, 2011 at 12:56 PM, de Witte wrote:
>
> HashMap map = new HashMap(4000);
> entity.setUnindexedProperty("dto", map) ;
Have you actually tried this? If it works, this behavior is totally
new to me, and I'd like to hear an official description of how this is
represented in the raw datastore.
I would expect an exception "HashMap cannot be stored in the datastore".
Jeff
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Number of writes per second limitation
You're still thinking of an RDBMS. Here is a more accurate mental model of the datastore: It's a big persistent HashMap. The key is a Key (tuple of ancestor keys, kind, and id), the value is a serialized (protobuf) blob of your properties. There is no update except loading an entity *whole* and writing an entity *whole* in a transaction. Indexes are separate - they live in a different BigTable. Querying for entities walks the index, then requires separate fetches for each entity blob. This is why keys-only queries are cheaper. This is also why unindexed properties are "free" - it really doesn't matter whether you write 20 bytes or 20 kilobytes into the serialized blob. Index updates are expensive because they require separate writes to other tables. Jeff On Sat, Nov 5, 2011 at 9:15 AM, J.Ganesan wrote: > Thank you, Ikai. Your comments were useful. I was fixated on "one > object one entity". Following your advice, I collapsed the entities. > Now, they are only handful. > > One last doubt - how does writing to the disk work in the following > scenario ? > > // transaction 1 > // set two objects as properties > // no doubt here > entity.setUnindexedProperty( string1, object1 ) ; > entity.setUnindexedProperty( string2, object2 ) ; > put(). > > > > // transaction 2 > // update object1 > entity.setUnindexedProperty( string1, object1 ) ; > put() ; > > App Engine stores bytes corresponding to object1 somewhere. I guess > that string1 points to the disk location, enabling overwriting with > updated bytes. My doubt is whether the second transaction forces > writing bytes corresponding to object2 also to the disk, which is > needless ? > > J.Ganesan > > On Nov 5, 3:56 am, "Ikai Lan (Google)" wrote: >> If all 4000 entites are in a single entity group, in theory you can do this >> because it counts as a single transactional write. There's a maximum RPC >> size of 11mb (implementation detail) so if you trip this, you're in some >> trouble - the RPC size include not only the size of the entity but also the >> size of all the indexes. >> >> The problem is that this is a bad design. App Engine charges for datastore >> ops, so you're already using 4000 datastore write ops per request + >> multiples for indexes. >> >> Instead, try to figure out how to can write the data in as few entities as >> possible. I suspect you're still thinking relationally. You didn't answer >> my question about what problem you're trying to solve. What are you >> building? Why would 4000 writes be needed? Why can't all the data fit into >> a single entity? >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App Engine >> plus.ikailan.com | twitter.com/ikai >> >> On Fri, Nov 4, 2011 at 8:31 AM, J.Ganesan wrote: >> >> >> >> >> >> >> >> > Thank you, Gerald. I will look for alternative implementations if I >> > can not call put() many times within a transaction. I am waiting for >> > Ikai's comments. >> >> > J.Ganesan >> >> > On Nov 4, 11:02 am, Gerald Tan wrote: >> > > If there is no need to reference the objects from outside the group, you >> > > would probably find it a lot more efficient to store the while array >> > > serialized as a byte array. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine for Java" group. >> > To post to this group, send email to >> > google-appengine-java@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine-java+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Number of writes per second limitation
On Fri, Nov 4, 2011 at 3:56 PM, Ikai Lan (Google) wrote: > If all 4000 entites are in a single entity group, in theory you can do this > because it counts as a single transactional write. There's a maximum RPC > size of 11mb (implementation detail) so if you trip this, you're in some > trouble - the RPC size include not only the size of the entity but also the > size of all the indexes. One question: If I write to 5 entities in a single entity group within one transaction, do these get batched up by the client library into a single RPC on commit or will this produce 5 RPCs plus one more for the commit? I'm curious to know what this does to the critical section, which could get large just from RPC latency. Thanks, Jeff -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Number of writes per second limitation
If all 4000 entites are in a single entity group, in theory you can do this because it counts as a single transactional write. There's a maximum RPC size of 11mb (implementation detail) so if you trip this, you're in some trouble - the RPC size include not only the size of the entity but also the size of all the indexes. The problem is that this is a bad design. App Engine charges for datastore ops, so you're already using 4000 datastore write ops per request + multiples for indexes. Instead, try to figure out how to can write the data in as few entities as possible. I suspect you're still thinking relationally. You didn't answer my question about what problem you're trying to solve. What are you building? Why would 4000 writes be needed? Why can't all the data fit into a single entity? -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Nov 4, 2011 at 8:31 AM, J.Ganesan wrote: > Thank you, Gerald. I will look for alternative implementations if I > can not call put() many times within a transaction. I am waiting for > Ikai's comments. > > J.Ganesan > > On Nov 4, 11:02 am, Gerald Tan wrote: > > If there is no need to reference the objects from outside the group, you > > would probably find it a lot more efficient to store the while array > > serialized as a byte array. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
