On 11 Jul 2010, at 04:30, jd wrote:
On Jul 11, 4:23 am, John Patterson wrote:
It is about the same speed. I've just uploaded 500,000 entities in
The new Mapping API sounds like the ideal solution but it is python
only at the moment.
Ikai wrote a great article introducing the Java version o
I am doing the exact same think you are mentioning, only I use google
spreadsheet to fetch the data.
some pointers:
use low level.
make sure you don't make many writes to the same entity group.
make parallel writes for different entity groups to speed things up.
have a retry mechanism for failed w
On Jul 11, 4:23 am, John Patterson wrote:
> It is about the same speed. I've just uploaded 500,000 entities in
> The new Mapping API sounds like the ideal solution but it is python
> only at the moment.
Oops no it is also available for Java now!
http://code.google.com/p/appengine-mapred
It is about the same speed. I've just uploaded 500,000 entities in
about 4 hours (over a very slow connection) but my batch size was only
50 - perhaps larger batches would be faster.
The new Mapping API sounds like the ideal solution but it is python
only at the moment. You could use the
John, is Remote Datastore faster than Python bulk insert? I tried to
load millions of entities using Python but had to give up as it took
about an hour to load just 100K. Peter
On Jul 10, 4:22 pm, John Patterson wrote:
> You can use the RemoteDatastore to do bulk puts directly from your
> deskt
Thanks for the feedback.
Btw, using low level api should have better performance than using jdo ? And
also, (regardless of jdo or low level api) making persistent a collection of
entities is *significantly* better than one at a time ?
On 10 July 2010 17:22, John Patterson wrote:
> You can use th
You can use the RemoteDatastore to do bulk puts directly from your
desktop to the live datastore. It is extremely fast as it simply
forwards on the binary protocol buffer data from one environment to
another.
http://code.google.com/p/remote-datastore/
On 10 Jul 2010, at 20:06, Robert Lanc
What I would do is create a servlet that accepts a row or rows of the
spreadsheet as params, and then create a local java app to read from
the spreadsheet and gradually feed the spreadsheet to the servlet, app
engine accepts up to 500 entities in a put.
On Jul 10, 7:56 am, Ice13ill wrote:
> I wan
