I actually only need 2 of my 8 columns of those 1000 posts. (the rest
is lazily loaded with ajax when clicked)
My line of thinking is that in a relational database, getting 2 vs 8
columns won't really matter. The database does the same query and
would just dump the extra 6 rows on the table before handing them to
me. Sure there is a tiny tiny bandwidth difference if it gets dumped
out at the database level or the application level, but they are
normally sitting right next to each other on gigabit, so it doesn't
really matter. I see how the app engine isn't exactly like this, but I
am still trying to figure out all of the performance characteristics
of the service.
For step 1 I will try cutting down to the 2 columns I care about in my
query. If that doesn't help, maybe I will just start caching the
result with the cache implementation.. it is a really small amount of
data I get back from the query (maybe 50KB tops). Really insane it
takes me 8 api and app cpu seconds every single time I get them.
On Feb 10, 12:52 pm, Ikai L (Google) ika...@google.com wrote:
The datastore isn't a relational database, and creating an index isn't the
same as it is in SQL - you aren't creating a B-tree index that allows a fast
query, you're creating an actual index so the datastore knows which entities
have the value you are looking for. Basically, this index has been created
already.
8 seconds is pretty high, but in general, it depends on the size of your
entities and whether offset is being used. The datastore is a distributed
key-value store. Rather than query for 1000 entities, is there a way you can
limit what you are querying for to a key-value retrieval? Think of the
datastore as a giant distributed, persistent hash table with additional
functionality: how would you optimize data retrieval for this type of store?
My guess is the functionality you're looking for can be either cached, or
you can do a keys only query and lazy query, or you can try to store the
data in sharded slabs - that is, store a denormalized cache of the data
you need from 200 temporally sharded lists of posts - it's unlikely that you
need all the data from each post.
A good project to study would be the Jaiku
engine:http://code.google.com/p/jaikuengine/
On Tue, Feb 9, 2010 at 9:10 PM, Brian bwa...@gmail.com wrote:
I am also getting and closing a new PersistenceManager around the
query, if that matters
On Feb 9, 11:08 pm, Brian bwa...@gmail.com wrote:
Hi guys,
I was wondering if anyone has found a less CPU intensive way to query.
I basically have a bunch of small objects in the database I want out.
I tried a few ways to get them out, including an extent and a query. I
ended up settling on this:
Query query = pm.newQuery(Post.class);
query.setOrdering(postedDate desc);
query.setRange(start, end);
ListPost posts = (ListPost) query.execute();
Doing a range of 0 to 1000.. figuring the 1000 newest items is good
enough.
I find that this tends to be pretty bad on my quota CPU.. like maybe 8
seconds of CPU time per query, if there are 1000 posts. Is this just
how it is? Or do I need to add an index to postedDate maybe?
I plan to play with it more tomorow, but if anyone has insight on
getting stuff out of the database fast (from both a time and CPU quota
standpoint), I would appreciate it. I have never seen a datbase take .1
seconds for such a stupid silly query, I have to be doing
something wrong.
Thanks!
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2B
unsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
Ikai Lan
Developer Programs Engineer, Google App
Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
