Alright so to sum up all the testing here it goes. I have a Driver doing this pulling in 32 queries aimed at randomness and different tables, much like that would be experienced in typical usage. Best performance comes from having 2 separate programs running on 2 separate files.
Single file different connection each thread: 1 thread: 64,100,000 Queries/HR 2 threds: 98,800,000 Queries/HR 4 threads: 115,600,000 Queries/Hr 8 threads: 119,200,000 Queries/HR 16 Threads: 129,600,000 Queries /HR Different file different hdd 2 programs 2 total threads : 102,300,000 Queries/HR 4 total threads: 156,320,000 Queries/ HR 8 total threads 155,440,000 Quereis/HR 2 Programs same file 2 thread total: 107,000,000 Quereis/HR 4 thread total: 117,800,000 Quereis/HR 8 thread total: 133,400,000 Queries/HR 16 thread total:133,840,000 Queries/HR 2 programs different file same hdd 2 thread total: 127,400,000 Queries/Hr 4 Threads total: 169,400,00 Queries/Hr 8 threads total: 180,400,000 Queries/HR Different file different hdd same program 50% of threads per file 2 threads: 87,000,000 4 threads: 117,200,000 8 Threads: 129,600,000 Different file same hdd same program 50% of threads per file 2 Threads: 110,400,000 4 Threads: 130,000,000 8 Threads: 136,000,000 On 11 Aug 2011, at 9:51pm, Drew Kozicki wrote: > Unfortunately reading from disk appears not to be the problem. Even at 32 > threads the IO appears to be very minimal. Our inability to scale appears to > be caused by a mutex in the caching. My CPU usage is at 30% and my disk is > near silent watching the performance monitor. This is on a 10 GB database > executing 32 different queries against 7 different tables. Purely as a method of research, having done some time trials and measured the results, as you obviously have done, a) Make a second copy of the data on the same disk, have half look at one copy, half look at the other copy, and see what that does to the times. b) If you have another shared physical volume available, do it again but this time put the second copy on the other hard disk. One tells you about contention for one file. The other tells you about contention for access to a single hard disk. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
