On Sat, Mar 21, 2009 at 12:03 PM, Kees Nuyt <k.n...@zonnet.nl> wrote: > > Hi Puneet, > > On Sat, 21 Mar 2009 10:47:44 -0500, P Kishor > <punk.k...@gmail.com> wrote: > >>I should have mentioned the page_size in my OP. It is 32768 set by me >>at the start of db creation. >> >>Yes, the db connection is kept open. > > Hm, apart from faster disks (15k RPM or high end SSD) I have > no idea what else can be done to improve performance.
Ok. That is a good "limit" to know. One scenario I would like to experiment with is doing a full fledged BLOB testing. For that, I need to be able to convert the entire dataset into the BLOBs table. As I mentioned in my original post, the BLOBs are taking about 430 KB per cell which is going to result in a 400+ GB db (problem, given my laptop has only 170 GB free... but I could do the testing on one my servers). The real problem is that the conversion is taking way too long. I set the process last night on my laptop, and this morning only about 1/4 of the table had converted. The conversion will be a one-time job, so it is not a big deal if it takes 4 days to do this, but still... would be nice to speed this up. Unfortunately, I can't use 15 K RPM disks, but I must add here that in the "real world" this db will reside on a 3 GHz quad core dual Xeon Xserve with 32 GB RAM and 2.73 TB hardware RAIDed "disk" (3 x 1 TB 7200 RPM disks that Apple supplies). If I can't improve 33 ms per query, then I can experiment with chopping up the task, which will be done via Xgrid anyway. The model will run on a 10-Xserve cluster, so I can create 10 copies of the db. This will allow each instance of the model run to query its own copy of the db, potentially divvying up the 9 hours of querying into 0.9 hours of querying. Of course, an entirely other route is to do the testing/development with SQLite, and then move to PostGres for production. That way we can have only one copy of the db, and query from the 10 cluster servers over IP. -- Puneet Kishor http://www.punkish.org/ Nelson Institute for Environmental Studies http://www.nelson.wisc.edu/ Carbon Model http://carbonmodel.org/ Open Source Geospatial Foundation http://www.osgeo.org/ Sent from: Madison WI United States. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
