Thank you both for your answers. I am using the shared memory segment in application startup approach. It seems that the problem was with allocating memory for an iterator to my data structure each time on xOpen and freeing the memory on xClose. It seems that for one thread it was not a big issue, but with many threads it became a bottleneck. I now keep one instance and allocate memory at xConnect and just "resetting" that on xOpen, and the performance is very good. I am using different connections (and different table instances) for each thread.
________________________________ From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on behalf of Hick Gunter <h...@scigames.at> Sent: Monday, July 24, 2017 9:05 AM To: 'SQLite mailing list' Subject: Re: [sqlite] Concurrent reads for VTs with in-memory data structures We are using a shared memory segment (created during application startup) to contain the data records, but you could also use a memory mapped file. This will keep the static data identical across all connections and processes. When writing your VT module, consider giving it a readonly/readwrite switch parameter. This can be used to provide the transaction and update methods (or not) in the method table within xCreate/xConnect. Not providing an xUpdate method tells SQLite that the table is strictly readonly and this may improve concurrency. That way you can perform the inital load with SQL statements in the "bootstrap" program while the application itself can only read. Bootstrap: CREATE VIRTUAL TABLE my_table_rw USING my_module ( 'RW' ); INSERT INTO ... DROP TABLE my_table_rw; Applcation. CREATE VIRTUAL TABLE my_table USING my_module (); -----Ursprüngliche Nachricht----- Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Dimitris Bil Gesendet: Montag, 03. Juli 2017 19:14 An: sqlite-users@mailinglists.sqlite.org Betreff: [sqlite] Concurrent reads for VTs with in-memory data structures I have some virtual tables that keep in-memory data structures. Data loading is happening at table creation (xCreate and xConnect are the same) and after that, during querying, only read access is needed. Queries do not access any other tables. Is there a way to achieve concurrent execution without having to keep multiple copies of each data structure? _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___________________________________________ Gunter Hick Software Engineer Scientific Games International GmbH FN 157284 a, HG Wien Klitschgasse 2-4, A-1130 Vienna, Austria Tel: +43 1 80100 0 E-Mail: h...@scigames.at This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation. _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
