The bottleneck appears to be mutex's locking access to the file and cache inside the same program. I'm guessing that the reason why 2 programs on 2 files on 2 hdd's did not perform as well as the one too me suggests that both hdd's do not have the same performance and that it is not access to the physical disk that is the problem. Just by having 2 programs access 2 different files on the same disk give that much of a performance/scaling gain causes me to believe that there exists a vital area of code that is mutexed that is the bottleneck.
The source code, unfortunately, I can't post as it is the property of the company I work for. So I'll ask the question. Is there any compile time or pragma's engineered towards read only performance as locking a table is no longer needed. To save mutexing in cache we would also be ok if each thread would have it's cache so it doesn't have to obtain a lock going into it. Thank you all once again for your help, Drew Message: 17 Date: Fri, 12 Aug 2011 19:28:33 +0100 From: Simon Slavin <slav...@bigfraud.org> Subject: Re: [sqlite] Read only scaling optimization To: General Discussion of SQLite Database <sqlite-users@sqlite.org> Message-ID: <b4050f5b-3edb-4b79-a58c-3788d971c...@bigfraud.org> Content-Type: text/plain; charset=us-ascii On 12 Aug 2011, at 7:01pm, Drew Kozicki wrote: > I have a Driver doing this pulling in 32 queries aimed at randomness and > different tables, much like that would be experienced in typical usage. Best > performance comes from having 2 separate programs running on 2 separate > files. I'm no expert, but that suggests to me that your bottleneck is access to the physical file on disk. So your greatest speed increases will come not from more threads but from a very fast hard disk drive, lots of hard drive caching, etc.. That's a great set of benchmarks, by the way. Simon. ------------------------------ Message: 18 Date: Fri, 12 Aug 2011 14:35:29 -0400 From: Pavel Ivanov <paiva...@gmail.com> Subject: Re: [sqlite] Read only scaling optimization To: General Discussion of SQLite Database <sqlite-users@sqlite.org> Message-ID: <cag1a4rt+pkj2ingy0jeym6h2vfikownlnx1bqgn-9hydput...@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 >> I have a Driver doing this pulling in 32 queries aimed at randomness and >> different tables, much like that would be experienced in typical usage. Best >> performance comes from having 2 separate programs running on 2 separate >> files. > > I'm no expert, but that suggests to me that your bottleneck is access to the physical file on disk. ?So your greatest speed increases will come not from more threads but from a very fast hard disk drive, lots of hard drive caching, etc.. It's a little surprising to me that with all the same conditions 2 files residing on the same drive have better performance than the same files residing on different drives. Theoretically that shouldn't happen. Pavel Message: 2 Date: Sun, 14 Aug 2011 17:53:26 +0400 From: Alexey Pechnikov <pechni...@mobigroup.ru> Subject: Re: [sqlite] Read only scaling optimization To: General Discussion of SQLite Database <sqlite-users@sqlite.org> Message-ID: <CANMYFJ=7F87=f415337O+ HuywUmjZtNfNL-V5m8Un=rg3wk...@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 2011/8/12 Pavel Ivanov <paiva...@gmail.com>: > It's a little surprising to me that with all the same conditions 2 > files residing on the same drive have better performance than the same > files residing on different drives. Theoretically that shouldn't > happen. Yes, it's not right behaviour. Is needed the sources of the test programm. -- Best regards, Alexey Pechnikov. http://pechnikov.tel/ _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users