On Sat, Jun 27, 2009 at 4:14 PM, Andy Freeman<ana...@earthlink.net> wrote: > >> > Does that mean that db.put((e1, e2, e3,)) where all of the entities >> > are 500kb will fail? >> >> Yes. > > Thanks. > > I'll take this opportunity to promote a couple of related feature > requests. > > (1) We need a way to estimate entity sizes > http://code.google.com/p/googleappengine/issues/detail?id=1084
The 1MB limit is on the API call, rather than the entity itself, per-se, so index size doesn't count in the 1MB limit. You can always serialize the entity yourself and check its size, though that requires touching datastore-internal methods. > > (2) We need a way to help predict when datastore operations will fail > http://code.google.com/p/googleappengine/issues/detail?id=917 > > I assume that db.get((k1, k2,)) can fail because of size reasons when > db.get(k1) followed by db.get(k2) will succeed. Does db.get((k1, > k2,)) return at least one entity in that case? No, the operation will simply fail. Given that it's an invariant that the returned list has the same length as the passed list, there's no sensible way to return partial results without implying that certain entities didn't exist when they actually do. -Nick Johnson > > > > On Jun 26, 9:36 am, "Nick Johnson (Google)" <nick.john...@google.com> > wrote: >> On Fri, Jun 26, 2009 at 4:42 PM, Andy Freeman <ana...@earthlink.net> wrote: >> >> > > the 1MB limit applies only to single API calls >> >> > Does that mean that db.put((e1, e2, e3,)) where all of the entities >> > are 500kb will fail? >> >> Yes. >> >> >> >> > Where are limits on the total size per call documented? >> >> >http://code.google.com/appengine/docs/python/datastore/overview.html#... >> > only mentions a limit on the size of individual entities and the total >> > number of entities for batch methods. The batch method documentation >> > (http://code.google.com/appengine/docs/python/datastore/functions.html >> > andhttp://code.google.com/appengine/docs/python/memcache/functions.html) >> > does not mention any limits. >> >> You're right - we need to improve our documentation in that area. The 1MB >> limit applies to _all_ API calls. >> >> >> >> > Is there a documented limit on the number of entities per memcache >> > call? >> >> No. >> >> >> >> > BTW - There is a typo in >> >http://code.google.com/appengine/docs/python/memcache/overview.html#Q... >> > . >> > It says "In addition to quotas, the following limits apply to the use >> > of the Mail service:" instead of "Memcache service" >> >> Thanks for the heads-up. >> >> -Nick Johnson >> >> >> >> >> >> >> >> > On Jun 26, 7:28 am, "Nick Johnson (Google)" <nick.john...@google.com> >> > wrote: >> > > Hi tav, >> >> > > Batch puts aren't transactional unless all the entities are in the >> > > same entity group. Transactions, however, _are_ transactional, and the >> > > 1MB limit applies only to single API calls, so you can make multiple >> > > puts to the same entity group in a transaction. >> >> > > -Nick Johnson >> >> > > On Fri, Jun 26, 2009 at 8:53 AM, tav<t...@espians.com> wrote: >> >> > > > Hey guys and girls, >> >> > > > I've got a situation where I'd have to "transactionally" update >> > > > multiple entities which would cumulatively be greater than the 1MB >> > > > datastore API limit... is there a decent solution for this? >> >> > > > For example, let's say that I start off with entities E1, E2, E3 which >> > > > are all about 400kb each. All the entities are specific to a given >> > > > User. I grab them all on a "remote node" and do some calculations on >> > > > them to yield new "computed" entities E1', E2', and E3'. >> >> > > > Any failure of the remote node or the datastore is recoverable except >> > > > when the remote node tries to *update* the datastore... in that >> > > > situation, it'd have to batch the update into 2 separate .put() calls >> > > > to overcome the 1MB limit. And should the remote node die after the >> > > > first put(), we have a messy situation =) >> >> > > > My solution at the moment is to: >> >> > > > 1. Create a UserRecord entity which has a 'version' attribute >> > > > corresponding to the "latest" versions of the related entities for any >> > > > given User. >> >> > > > 2. Add a 'version' attribute to all the entities. >> >> > > > 3. Whenever the remote node creates the "computed" new set of >> > > > entities, it creates them all with a new version number -- applying >> > > > the same version for all the entities in the same "transaction". >> >> > > > 4. These new entities are actually .put() as totally separate and new >> > > > entities, i.e. they do not overwrite the old entities. >> >> > > > 5. Once a remote node successfully writes new versions of all the >> > > > entities relating to a User, it updates the UserRecord with the latest >> > > > version number. >> >> > > > 6. From the remote node, delete all Entities related to a User which >> > > > don't have the latest version number. >> >> > > > 7. Have a background thread check and do deletions of invalid versions >> > > > in case a remote node had died whilst doing step 4, 5 or 6... >> >> > > > I've skipped out the complications caused by multiple remote nodes >> > > > working on data relating to the same User -- but, overall, the >> > > > approach is pretty much the same. >> >> > > > Now, the advantage of this approach (as far as I can see) is that data >> > > > relating to a User is never *lost*. That is, data is never lost before >> > > > there is valid data to replace it. >> >> > > > However, the disadvantage is that for (unknown) periods of time, there >> > > > would be duplicate data sets for a given User... All of which is >> > > > caused by the fact that the datastore calls cannot exceed 1MB. =( >> >> > > > So queries will yield duplicate data -- gah!! >> >> > > > Is there a better approach to try at all? Thanks! >> >> > > > -- >> > > > love, tav >> >> > > > plex:espians/tav | t...@espians.com | +44 (0) 7809 569 369 >> > > >http://tav.espians.com|http://twitter.com/tav|<http://twitter.com/tav%7C>skype:tavespian >> >> > > -- >> > > Nick Johnson, App Engine Developer Programs Engineer >> > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >> > > Number: 368047- Hide quoted text - >> >> > > - Show quoted text - >> >> -- >> Nick Johnson, App Engine Developer Programs Engineer >> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: >> 368047- Hide quoted text - >> >> - Show quoted text - > > > -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
