Yes, my tests have been on 8-real-core (+8 hyperthreaded) Windows Server 2008 R2 Datacenter, and on 6-real-core (+6 hyperthreaded) Fedora 14.
In general Linux performs better than Windows, but both exhibit the seeming serialization when multiple independent Sqlite databases are in use. There is always spare CPU left over in these tests, and the CPU utilization isn't much higher in the two database test than in the single database. I didn't record the CPU utilization numbers, but just watched them with the process monitor or with top. This is a Java application that has a thin custom native interface into a Sqlite shared library. FWIW, the options I use when compiling the Sqlite (latest version) amalgamation are: -DSQLITE_OMIT_LOAD_EXTENSION=1 -DSQLITE_THREADSAFE=(I have tried both 1and 2) -DSQLITE_TEMP_STORE=2 -DSQLITE_DEFAULT_PAGE_SIZE=4096 -DHAVE_USLEEP=1 As I mentioned originally this may simply be a problem in my code, but comparing Java profiling between the single database and two databases shows an inordinate increase in the time spent in the code that interfaces with Sqlite. This, with all the unfruitful time debugging my code led me to ask this question to this group. I appreciate everyone's help. I will continue to debug and experiment and any advice is welcome! Bob ________________________________ From: Keith Medcalf <kmedc...@dessus.com> To: Bob Price <rjp_em...@yahoo.com>; General Discussion of SQLite Database <sqlite-users@sqlite.org> Sent: Saturday, September 22, 2012 10:14 PM Subject: RE: [sqlite] performance of one process using multiple independent sqlite databases Do you have multiple real CPU cores? Multiple fake cores (what intel calls hyperthreads)? When one thread in one process opens a database and runs it takes N seconds -- utilizing only one CPU of course. What is the CPU consumption while the process is running? When you use two processes, each with one thread opening one separate database each but it still takes N seconds what is the CPU consumption? When you use one process with two threads, each with an independent connection to independent databases, what is the CPU consumption? What operating system? --- () ascii ribbon campaign against html e-mail /\ www.asciiribbon.org > -----Original Message----- > From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users- > boun...@sqlite.org] On Behalf Of Bob Price > Sent: Saturday, 22 September, 2012 18:48 > To: sqlite-users@sqlite.org > Subject: [sqlite] performance of one process using multiple independent > sqlite databases > > I've been searching through the archive without much luck on this topic, so > I'll ask my question. If this has been answered somewhere else please point > me to that. > > I have a single process that independently opens and uses multiple Sqlite > databases, each with its own connection. The data in each database is > entirely independent of the data in any other database, so I never use the > ATTACH capability, just open each db separately. > > This all works correctly, but I am having some performance issues. It > appears that it might be the case that a write on the connection of one > database blocks any activity on the connection of the other databases. Some > things that led me to this conclusion are tests that show with only a single > db I can process some data in N seconds, but when I have two threads each > talking to their own db, the time jumps to around 2*N seconds, while if I > run two test processes each with its own db the time is only a little over > N. The host I am using has lots of spare CPU cycles during these tests, and > I have used normal disks, SSDs, and ram disks, all with synchronous=OFF. > > I have tried THREAD_SAFE=1 and 2, the default journal mode, the WAL journal > mode, and BEGIN DEFERRED and BEGIN EXCLUSIVE, on Linux and Windows. > > Does this make sense? Does Sqlite let two connections to different databases > impact each other? Of course it could be that I am doing something dumb > elsewhere, but I have literally spent many days debugging and profiling this > to no avail. > > Any suggestions or other configurations I might try? > > Any help is appreciated! > > Bob > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
