Ikai,
Maybe you are right. Maybe not. I'm not an expert in datastore
internals, but here is my point of view.
This paper claims that Berkeley DB Java edition can insert about
15,000 records per second.
http://www.oracle.com/database/docs/bdb-je-architecture-whitepaper.pdf
The graphic is on page
I think we can safely assume that the programmer was trying to speed
up things a little by writing 12 thousand objects in a single
operation.
Now if that gets to be faster or slower than writing each object
separately, it is a matter of the internal implementation of the data
store. I prefer to do
my experience with a relatively simple application via JDO
makePersistentAll() was that I got
DataStore Operation Timeout exceptions with batch sizes of approx
200-300 objects ...
On Feb 24, 1:48 pm, Guillermo Schwarz
wrote:
> I think we can safely assume that the programmer was trying to speed
>
I think there is a way to grab big chunks of oprations, put them in a
queue to be done asynchronously and that would be it.
My take would be that using proxies it would be easy to queue any long
operation transparently. I've done that with EJBs in the past, I don't
see the reason a QueingProxy cou
Simple key-only writes can definitely do it, but there's a few places where
you can introduce overhead:
- serialization
- network I/O
- indexes
My point wasn't necessarily that it wasn't possible. makePersistentAll does
use a batch write, and there are definitely sites that can do 12,000+ writes
On Wed, Feb 24, 2010 at 1:06 PM, Ikai L (Google) wrote:
> My point wasn't necessarily that it wasn't possible. makePersistentAll does
> use a batch write, and there are definitely sites that can do 12,000+ writes
> a second (and well above that), but I don't know of any that will attempt to
> do t
Jeff, point taken, but the original poster has been asking for three
different requirements:
- requirement to do all writes synchronously
- sub-some-couple-hundred-millisecond writes
- 12k entities being written
This just won't scale well if it's common. Messaging users can be done
asynchronously
I don't think the original poster had a requirement for synchronous
writes; he just didn't want to do the writes asynchronously because it
involved a lot more code.
I'm also perfectly fine with asynchronous writes and a very lax
interpretation of consistency. I don't even mind writing extra code.
Hi
I wrote importing geoip database in JSF. It has like 100k entries. Besides
appengine and datastore performance is awful and importing took around 1,5h
of total CPU its fairly easy to use TaskQueue. If there is above average
processing to be done then i suggest separating task into smaller batch
Guillermo,
Taskqueue items can only be 10K
(http://code.google.com/appengine/docs/java/taskqueue/overview.html#Quotas_and_Limits).
The basic idea is that if you have more data than that you put it into
an entity (in the data-store) and have the task pull it out and
process it. It might be that you
We have an issue for an asynchronous write API for the datastore:
http://code.google.com/p/googleappengine/issues/detail?id=2817
This is something that can fit into that model.
On Thu, Feb 25, 2010 at 12:26 PM, Scott Hernandez
wrote:
> Guillermo,
>
> Taskqueue items can only be 10K
> (
> http:/
11 matches
Mail list logo
