Re: [google-appengine] Local datastore is slow once loaded up, any suggestions?

2010-02-19 Thread Eli Jones 
I did some preliminary testing on this.. mainly, I used the cachepy.py from
here:

http://appengine-cookbook.appspot.com/recipe/cachepy-faster-than-memcache-and-unlimited-quota/

I
created a test page.. that I used to load 400,000 entities from one of my
Models whenever it received a flag from me to fill up the cache (I created
the entities on the fly and gave them key_names and values and then called
cachepy.set() on each one).

Then.. i removed the cache flag.. and just used the test page to get
entities from the cachepy cache by keyname to see how long they hung around.

I then went into the Interactive Console and began grabbing entities from
the cache as well.

The entities could be got from both locations.

I then started doing this:

result = cachepy.get(keyname)
result.prop1 = "newvalue"

etc..

I would then use my test page to check the cache for that keyname, and I
would see the "newvalue" for prop1.

It looks like.. as long as you are actively using the cache from the global
dictionary object that cachpy uses.. it will stay in memory. (It has for 60
minutes so far, and I'm guessing it will persist as long as you are using it
and you don't stick "too many" new objects into the cache.)

Since you aren't pickling the objects before sticking them in the global
dict.. you're able to use the Model entities with all the datastore type
checking etc.. so.. you have some sanity to what can and can't be done to
the entities in memory and you get proper error reporting if you do
something un-kosher to one of your entity properties like trying to set an
IntegerProperty to a String.. etc.

So, if you have a setup where you do a lot of accessing entities by key_name
and then "doin thangs" to that stuff.. this could work in the development
server.  You'd just need a bootstrap process to stuff your datastore into
the cache when you were ready to get to testing.. and then.. create some
process to dump that info to the datastore once you were done mucking
around.

Loading up the 400,000 entities into the cache gobbled up an extra 270MB of
RAM in the pythonw.exe process on my machine... and I'm guessing you can
just keep adding objects to the cache until Python crashes...

I guess if you really wanted to get punk rock... you could emulate searching
by entity properties by just having a function that looped through all
entities of a given Model and had it return the ones that matched your
"query".  That would be plenty fast.

On Fri, Feb 19, 2010 at 8:44 PM, Eli Jones  wrote:

> I noticed this issue as well when working from a bulkloaded backup up my
> live datastore (around the same number of entities).  For me, it seems that
> db.delete() takes forever.. db.puts() seem tolerable.  (I'm on Windows).
>
> I see that Nick is indicating that this may just be the way it is.
>
> It'd be nice to do offline processing and development working against a
> snapshot of my current datastore.. but I'm not going to whine too much for
> that.  I'd rather the App Engine Team keep focusing on sweetening the live
> store.
>
> One potential way around this (if you reay, really wanted to use the
> development environment with a large datastore).. you could cook up a way to
> have the datastore stuff loaded into a global Dictionary object.. and you
> could put(), delete() stuff from it while testing.
>
> That might work well since the development environment seems to run only
> one instance or thread or whatever they call it.. so as long as you didn't
> fill up the memory, it seems that maybe the global object would be there for
> long enough to do several tests..
>
> Then again, you'd probably need uncoupled Models.. where no class was
> related to any other one through references or ancestors or parents or
> children etc.  So, like I said, you'd have to really want to do this..
>
> For me, this would work since I want to do lots of cycling through 10s of
> thousands of entities.. updating and deleting them in memory (Model entities
> are relate to others only through their key_names and not through any
> defined properties)... then when I was done.. I could just have some process
> sync the ram datastore with the one on disk.. and go have a beer or whatever
> while it took its sweet time.
>
> There be dragons in the details though.
>
>
> On Fri, Feb 19, 2010 at 12:19 PM, obvious  wrote:
>
>> I've loaded up a local datastore with 40,000+ entries.  Unfortunately,
>> recalling any data from it at all is very slow on my fairly new
>> Macbook Pro.  Any suggestions on speeding things up, short of buying a
>> new piece of hardware?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appeng...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com
>> .
>> For more options, vi

Re: [google-appengine] Local datastore is slow once loaded up, any suggestions?

2010-02-19 Thread Eli Jones 
I noticed this issue as well when working from a bulkloaded backup up my
live datastore (around the same number of entities).  For me, it seems that
db.delete() takes forever.. db.puts() seem tolerable.  (I'm on Windows).

I see that Nick is indicating that this may just be the way it is.

It'd be nice to do offline processing and development working against a
snapshot of my current datastore.. but I'm not going to whine too much for
that.  I'd rather the App Engine Team keep focusing on sweetening the live
store.

One potential way around this (if you reay, really wanted to use the
development environment with a large datastore).. you could cook up a way to
have the datastore stuff loaded into a global Dictionary object.. and you
could put(), delete() stuff from it while testing.

That might work well since the development environment seems to run only one
instance or thread or whatever they call it.. so as long as you didn't fill
up the memory, it seems that maybe the global object would be there for long
enough to do several tests..

Then again, you'd probably need uncoupled Models.. where no class was
related to any other one through references or ancestors or parents or
children etc.  So, like I said, you'd have to really want to do this..

For me, this would work since I want to do lots of cycling through 10s of
thousands of entities.. updating and deleting them in memory (Model entities
are relate to others only through their key_names and not through any
defined properties)... then when I was done.. I could just have some process
sync the ram datastore with the one on disk.. and go have a beer or whatever
while it took its sweet time.

There be dragons in the details though.

On Fri, Feb 19, 2010 at 12:19 PM, obvious  wrote:

> I've loaded up a local datastore with 40,000+ entries.  Unfortunately,
> recalling any data from it at all is very slow on my fairly new
> Macbook Pro.  Any suggestions on speeding things up, short of buying a
> new piece of hardware?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Local datastore is slow once loaded up, any suggestions?

2010-02-19 Thread Nick Johnson (Google) 
Hi,

Unfortunately, the App Engine SDK datastore isn't really designed for
testing with such a large amount of data. You should try testing with a much
smaller dataset, if that's remotely possible.

-Nick Johnson


On Fri, Feb 19, 2010 at 5:19 PM, obvious  wrote:

> I've loaded up a local datastore with 40,000+ entries.  Unfortunately,
> recalling any data from it at all is very slow on my fairly new
> Macbook Pro.  Any suggestions on speeding things up, short of buying a
> new piece of hardware?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>


-- 
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Local datastore is slow once loaded up, any suggestions?

2010-02-19 Thread obvious 
I've loaded up a local datastore with 40,000+ entries.  Unfortunately,
recalling any data from it at all is very slow on my fairly new
Macbook Pro.  Any suggestions on speeding things up, short of buying a
new piece of hardware?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.