Thanks for the quick responses!
I am on a machine with many many cores, 500GB RAM, and lots of NVMe drives raided together, so the system should not be the issue. I have been testing with 16, 32, and 48 threads/databases at once, and the cumulative time it takes for all of the threads to just open all (millions) of the databases goes from 1200 seconds to 2200 seconds to 3300 seconds. As mentioned, this is likely to be something else, but I was hoping that I was somehow using SQLite wrong. Jason Lee ________________________________ From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on behalf of Jens Alfke <j...@mooseyard.com> Sent: Monday, April 22, 2019 12:52:28 PM To: SQLite mailing list Subject: Re: [sqlite] Multiple Independent Database Instances > On Apr 22, 2019, at 11:39 AM, Lee, Jason <jason...@lanl.gov> wrote: > > Hi. Are there any gotchas when opening multiple independent databases from > within one process using the C API? Do you mean different database files, or multiple connections to the same file? > I am opening one database per thread in my code, and noticed that > sqlite3_open_v2 and sqlite3_close slow down as the number of threads > increase, indicating there might be some resource contention somewhere, even > though the databases should be independent of each other. How many databases/threads? With huge numbers I’d expect slowdowns, since you’ll be bottlenecking on I/O. But otherwise there shouldn’t be any gotchas. I would troubleshoot this by profiling the running code to see where the time is being spent. Even without knowledge of the SQLite internals, looking at the call stacks of the hot-spots can help identify what the problem is. —Jens _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
