On 22 Apr 2019, at 10:25pm, Lee, Jason <[email protected]> wrote: > I have a set of several million database files sitting on my filesystem. Each > thread will open a previously unprocessed database file, do some queries, > close the database, and move on to the next unprocessed database file.
If this process is getting slower and slower, you have a resource leak somewhere in your program. It's possible to make SQLite do this using faulty programming. For instance, you may have a statement that reads a table. Statements much be finalized or reset using sqlite3_finalize() or sqlite3_reset() and you still need to do this even if SQLite returned SQLITE_DONE to tell you there are no more rows to return. If you do not do this, even though sqlite3_close() will run and return SQLITE_OK, it cannot release the resources used by the file because a statement is still pending. It waits until the statement is terminated and then automatically closes the file. So you get a resource leak until that's done. You can start analysis by using a monitoring program to monitor memory usage of the process. Does it gradually use more and more memory the longer it runs ? If so, it shouldn't be too difficult to figure out what memory isn't being released. _______________________________________________ sqlite-users mailing list [email protected] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
