We are going to hack AAF a bit to make the optimize optional. Should we flush every time we bulk index if we don't optimize?
I'm running this application on EC2, so I think part of the problem is the poor IO performance on the VPS. Thanks. On Sep 29, 2007, at 12:12 PM, Jens Kraemer wrote: > On Fri, Sep 28, 2007 at 03:42:30PM -0400, Erik Morton wrote: >> Thanks Jens. >> >> I'm seeing very strange behavior. The ferret/drb server is running on >> the web server. I disable AAF in my environment.rb like so: >> >> MyModel.disable_ferret >> >> Then, every hour I run a script that grabs the records that have been >> updated and I call >> >> MyModel.bulk_index array_of_changed_objects >> >> When there are, say, 100 or so objects that have changed the server >> where DRB is will have approximately 50% of the CPU waiting on IO, >> and will drop from 120MB free of memory to 10MB free, though the >> ruby/ >> drb process doesn't seem to actually consume that memory -- or at >> least it's not reported to by top. > > Where do you get these numbers from? possibly it's just the os using > unused ram for filesystem buffers? > >> While the batch update is happening it seems like my entire site is >> locked up. Requests usually hang until the indexing completes. >> Further, I can't even run script/ console from a different machine >> until the indexing completes. I'm doing the bulk index on two >> indexes. One is about 41K records, the other is 1 million >> records. In >> both cases there has been at most 100 or so objects that needed >> to be >> indexed in bulk. The fact that script/ console, when run from a >> different server, doesn't load until the index stops makes me think >> that either something is blocking in the ferret/drb server, or the >> optimization of the 3GB index after the bulk_index of 100 records is >> consuming all of the web server's resources. > > While the batch update is running, nobody else is able to update the > index. So yes, every other request that wants to update the index will > hang. However requests not using aaf at all, or searches, should do > fine. > > If you feel like the DRb takes too much CPU, use renice or nice > when you > start it to lower it's priority. > > Another possibility that comes to mind is database locks, however I > can't imagine where these should come from. > > With your index size, the optimizing might be the culprit - just > comment > out that portion and look how it goes without it (in > ferret_extensions.rb). > > cheers, > Jens > > -- > Jens Krämer > http://www.jkraemer.net/ - Blog > http://www.omdb.org/ - The new free film database > _______________________________________________ > Ferret-talk mailing list > [email protected] > http://rubyforge.org/mailman/listinfo/ferret-talk _______________________________________________ Ferret-talk mailing list [email protected] http://rubyforge.org/mailman/listinfo/ferret-talk
