Re: [Interest] SQLite database: avoid high memory usage?
Sorry, I was wrong. It is an issue of our app. On 1/24/2022 11:40 AM, Alexander Dyagilev wrote: Hello, Let's suppose we have a database file of 16GB in size. We open it using this code: m_db = QSqlDatabase::addDatabase( "QSQLITE", dbConnectionName()); m_db.setConnectOptions("PRAGMA jounal_mode=WAL;"); m_db.setDatabaseName(m_dbPath); m_db.open(); After this, our process will use 16GB of memory (at least, under Windows OS). I.e. the whole database file is loaded into the memory. Is there a way to avoid this? ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] SQLite database: avoid high memory usage?
On Monday, 24 January 2022 02:41:38 PST Alexander Dyagilev wrote: > Sorry, I don't really know (and I do not even care). But this is what > Windows Task Manager shows to our users, and they are concerned about this. Easy way to answer this: try to open a database that is bigger than the virtual memory total for your system. If you can, then it's just virtual address for the memory-mapped file, not committed memory. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel DPG Cloud Engineering ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] SQLite database: avoid high memory usage?
On 1/24/2022 12:39 PM, Hamish Moffatt via Interest wrote: Did you mean to misspell journal_mode? LOL. OK, fine... ROFL... No, of course no... Are you sure it's real memory usage and not just address space used by mmap? Sorry, I don't really know (and I do not even care). But this is what Windows Task Manager shows to our users, and they are concerned about this. ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] SQLite database: avoid high memory usage?
On Mon, Jan 24, 2022 at 11:40 AM Hamish Moffatt via Interest wrote: > > On 24/1/22 19:40, Alexander Dyagilev wrote: > > Hello, > > > > Let's suppose we have a database file of 16GB in size. > > > > We open it using this code: > > > > m_db = QSqlDatabase::addDatabase( > > "QSQLITE", > > dbConnectionName()); > > m_db.setConnectOptions("PRAGMA jounal_mode=WAL;"); > > m_db.setDatabaseName(m_dbPath); > > m_db.open(); > > > > After this, our process will use 16GB of memory (at least, under > > Windows OS). I.e. the whole database file is loaded into the memory. > > > > Is there a way to avoid this? > > > Did you mean to misspell journal_mode? > > Are you sure it's real memory usage and not just address space used by mmap? > > > Hamish > Indeed. Doubts if on Windows it will be all really mapped: https://docs.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases Besides that, to keep large areas mapped in RAM, one may need to lock the pages in memory with their own limits. Consider inspecting the sources of the SQL-db concerned. Kind regards, Robert ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] SQLite database: avoid high memory usage?
On 24/1/22 19:40, Alexander Dyagilev wrote: Hello, Let's suppose we have a database file of 16GB in size. We open it using this code: m_db = QSqlDatabase::addDatabase( "QSQLITE", dbConnectionName()); m_db.setConnectOptions("PRAGMA jounal_mode=WAL;"); m_db.setDatabaseName(m_dbPath); m_db.open(); After this, our process will use 16GB of memory (at least, under Windows OS). I.e. the whole database file is loaded into the memory. Is there a way to avoid this? Did you mean to misspell journal_mode? Are you sure it's real memory usage and not just address space used by mmap? Hamish ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest