Dear experts, In my application performance is of critical importance. I've chosen to preallocate a large contiguous block of memory for the page cache.
I make use of the following function to configure the page cache: sqlite3_config(SQLITE_CONFIG_PAGECACHE, void*, int sz, int N); The documentation for this function says the following: *This option specifies a static memory buffer that SQLite can use for the database page cache with the default page cache implementation. There are three arguments to this option: A pointer to 8-byte aligned memory, the size of each page buffer (sz), and the number of pages (N). The sz argument should be the size of the largest database page (a power of two between 512 and 32768) plus a little extra for each page header. The page header size is 20 to 40 bytes depending on the host architecture. It is harmless, apart from the wasted memory, to make sz a little too large.* Now, first of all, with my default database block size of 1024, the 20 to 40 bytes documented for the cache header size is wrong. On the Linux and HP-UX Itanium systems where I've tested this with SQLite 3.7.8, the cache header size was 248 bytes. This could be seen in the output from sqlite3_status(SQLITE_STATUS_PAGECACHE_SIZE) which returned a value of about 1272 bytes. (1272 - 1024 = 248). This means the page cache header size is already 25% of the size of the actual data block, or put differently, 20% of the preallocated cache buffer is 'lost' to the header component. With a cache buffer of 5 GB, this means a total of 1 GB for the header meta data only. SQLite is a little quirky in the way that even though I may allocate a block of 1 GB of memory for the page cache, if I specify a too small cache block size (sz value) in the call to sqlite3_config(SQLITE_CONFIG_PAGECACHE, void*, int sz, int N), then the cache page allocation from the preallocated buffer will fail and that memory won't be used for the cache at all, instead it falls back to the default allocator and simply allocates additional memory for the cache. The preallocated memory block is totally wasted. Now since my application performs a high number of transactions, the page cache grows to several GB of memory. Since memory is also a finite resource, I tried to limit the cache block size to the minimum required. When I first tried it with the documented 20 - 40 additional bytes for the header, it failed. However, using the 248 header bytes that resulted from the output of sqlite_status(SQLITE_STATUS_PAGECACHE_SIZE), it worked fine. That was with SQLite version 3.7.8. Now my dilemma is that it seems the page cache block header size was increased from 3.7.8 to 3.7.10. With 3.7.10 sqlite_status(SQLITE_STATUS_PAGECACHE_SIZE) now returns 1312 bytes, indicating a page cache header size of 288 bytes. This means that once again my application is not using the preallocated buffer at all, but is allocating cache with malloc() and is running out of memory. So I have to change the header size constant, recompile and redeploy the application. In order to avoid this scenario in future with future releases of SQLite, is there a way to determine the page cache header size within the code, keeping in mind that the page cache is configured using sqlite3_config() * before* the call to sqlite3_initialize()? Any advice on handling this problem would be much appreciated. Best regards, Jaco _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
