Ahh, the first positive result of feedback. I forgot about ~20MB
transaction write limit, so that will have to be addressed. Thank you,
I'll address it soon, I'll enter it as an issue.
The details are in the source, but the general concept is as
straightforward as receiving the entity and splitting it up into
entities that are each less than 1MB. That's what the put(Entity)
method does (little more detail... it takes your original entity and
puts as much properties as it can in the first shard without going
over the size limit, then it goes on to the next shard and puts more
properties, if it comes across a large blob it splits it up among the
shards). The get(Key) method takes your key and queries the database
for all the shards, reassembles the original entity, and gives that
back to you, with the proper Entity.getKey() even so that you don't
care that the shards exist (that's why Text is not supported, which is
just a minor point, because I can't tell if something large I'm
reassembling is a Blob or a Text, since Text to Blob is just
Text.getValue.getbytes[], I thought that was a good compromise).
Nothing I can do about the 30sec request time limit (but when Task
Queue comes out, that will not be an issue! :) )
On Aug 26, 7:59 am, Stakka <henrik.lindqv...@gmail.com> wrote:
> How do you get past the datastore's 1 MB entity size limit, ~20MB
> transaction write limit and the 30 sec request time limit of writing
> this amount of data?
>
> On Aug 26, 9:41 am, Tristan <lugieb...@gmail.com> wrote:
>
>
>
> > Ok, it is far from complete, but I have implemented the put(Entity)
> > and the get(Key) methods around which the whole library will be based.
> > The source of those two methods shows how it is possible to store
> > large entities without having to worry about their sizes. If you can
> > upload it, you should be able to do something like:
>
> > Entity e = new Entity("myEntity", "myKey");
> > e.setProperty("myProperty", <200MB Blob>);
> > mindashDatastore.put(e);
>
> > Design considerations:
> > 1. exact implementation of the DatastoreService Interface so that
> > coding practices can remain the same. (only added
> > EntityCorruptException)
> > 2. Text type is not supported due to predictability issues (use Blob
> > instead)
>
> > I acknowledge that some of my sizing assumptions may be inappropriate,
> > those were my best guesses at the time.
>
> > The project can be found at:http://code.google.com/p/mindash-datastore/
>
> > I look forward to your feedback as the library becomes functional.
>
> > -Tristan
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~----------~----~----~----~------~----~------~--~---
