On Thu, 7 Feb 2013 09:56:27 +0000 (GMT) James Vanns <james.va...@framestore.com> wrote:
> Hello list. I'd like to ask someone with more SQLite experience than me a > simple question. First, some background; > > Distribution: Scientific Linux 6.3 > Kernel: 2.6.32-279.9.1.el6.x86_64 > SQLite version: 3.6.20 > > We have a single process that, given some data, does some processing and > writes it all to a single SQLite DB file. This is a write-once process. When > this task is finished, the file itself is marked as read only (0444). > > This file exists on an NFS share for multiple users to read - nothing further > is ever written to it. The problem we're seeing is that when this DB file is > read from (over NFS) none of the pages are cached (despite ~12GB free for > page cache use) or at least immediately evicted. This is quite detrimental to > performance because our resulting data files (SQLite DB files) are between > 100 to 400 MB in size. We *want* it to be cached - the whole thing. The page > cache would do this nicely for us and allow multiple processes on the same > machine to share that data without any complication. > > I understand that SQLite implements it's own internal page cache but why, on > a standard desktop machine, will it not use the page cache. Is there anyway > of forcing it or bypassing the internal page cache in favour of the job that > Linux already does? I cannot find any reference to O_DIRECT or madvise() or > favdise() etc. in the code. The following PRAGMAs don't help either; > > PRAGMA writable_schema = OFF > PRAGMA journal_mode = OFF > PRAGMA synchronous = OFF > > PRAGMA cache_size = -<size of DB file in kbytes> > > Obviously that last one works - but only for a single process and for the > lifetime of that process. We want the pages to reside in RAM afterwards. > > Anyone out there know how to correct this undesirable behaviour? You should use these pragmas too : PRAGMA temp_store = MEMORY; PRAGMA read_uncommited = TRUE; If not, a big select with a big sort could try to use temporal files on your nfs server. As you aren't doing any write, no need to wait for write locking. If you need cache being persistent between process on the same server, you can build a ram disk, write the db there and use it from any process. This way you read the db only once from nfs. Even better, you can shutdown nfs because a simple ftp/http server and wget/fetch can do what you want, serve/receive read only files. > Regards, > > Jim Vanns > > -- > Jim Vanns > Senior Software Developer > Framestore > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users --- --- Eduardo Morras <emorr...@yahoo.es> _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
