The bottleneck appears to be mutex's locking access to the file and cache inside the same program. I'm guessing that the reason why 2 programs on 2 files on 2 hdd's did not perform as well as the one too me suggests that both hdd's do not have the same performance and that it is not access to the physical disk that is the problem. Just by having 2 programs access 2 different files on the same disk give that much of a performance/scaling gain causes me to believe that there exists a vital area of code that is mutexed that is the bottleneck.
The source code, unfortunately, I can't post as it is the property of the company I work for. So I'll ask the question. Is there any compile time or pragma's engineered towards read only performance as locking a table is no longer needed. To save mutexing in cache we would also be ok if each thread would have it's cache so it doesn't have to obtain a lock going into it. Thank you all once again for your help, Drew Message: 17 Date: Fri, 12 Aug 2011 19:28:33 +0100 From: Simon Slavin <[email protected]> Subject: Re: [sqlite] Read only scaling optimization To: General Discussion of SQLite Database <[email protected]> Message-ID: <[email protected]> Content-Type: text/plain; charset=us-ascii On 12 Aug 2011, at 7:01pm, Drew Kozicki wrote: > I have a Driver doing this pulling in 32 queries aimed at randomness and > different tables, much like that would be experienced in typical usage. Best > performance comes from having 2 separate programs running on 2 separate > files. I'm no expert, but that suggests to me that your bottleneck is access to the physical file on disk. So your greatest speed increases will come not from more threads but from a very fast hard disk drive, lots of hard drive caching, etc.. That's a great set of benchmarks, by the way. Simon. ------------------------------ Message: 18 Date: Fri, 12 Aug 2011 14:35:29 -0400 From: Pavel Ivanov <[email protected]> Subject: Re: [sqlite] Read only scaling optimization To: General Discussion of SQLite Database <[email protected]> Message-ID: <cag1a4rt+pkj2ingy0jeym6h2vfikownlnx1bqgn-9hydput...@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 >> I have a Driver doing this pulling in 32 queries aimed at randomness and >> different tables, much like that would be experienced in typical usage. Best >> performance comes from having 2 separate programs running on 2 separate >> files. > > I'm no expert, but that suggests to me that your bottleneck is access to the physical file on disk. ?So your greatest speed increases will come not from more threads but from a very fast hard disk drive, lots of hard drive caching, etc.. It's a little surprising to me that with all the same conditions 2 files residing on the same drive have better performance than the same files residing on different drives. Theoretically that shouldn't happen. Pavel Message: 2 Date: Sun, 14 Aug 2011 17:53:26 +0400 From: Alexey Pechnikov <[email protected]> Subject: Re: [sqlite] Read only scaling optimization To: General Discussion of SQLite Database <[email protected]> Message-ID: <CANMYFJ=7F87=f415337O+ [email protected]> Content-Type: text/plain; charset=ISO-8859-1 2011/8/12 Pavel Ivanov <[email protected]>: > It's a little surprising to me that with all the same conditions 2 > files residing on the same drive have better performance than the same > files residing on different drives. Theoretically that shouldn't > happen. Yes, it's not right behaviour. Is needed the sources of the test programm. -- Best regards, Alexey Pechnikov. http://pechnikov.tel/ _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
