richard, comments inline:
----- Original Message ----- From: "D. Richard Hipp" <[EMAIL PROTECTED]> To: <sqlite-users@sqlite.org> Sent: Friday, June 17, 2005 8:54 PM Subject: Re: [sqlite] license to hack: pager in shmem > On Fri, 2005-06-17 at 20:16 -0400, Paul G wrote: > > i'd like to hack sqlite3 to use shared memory for the page cache. > > > > A more useful hack, perhaps, would be to change the pager to use > memmap() (when available) instead of read() and write(). This i'm going to assume you meant mmap() ;] indeed, i've already looked at doing this (and have the beginnings of an implementation) but was put off by the intrusiveness of the change. > The downside: You'll need to change the pager API around some, which > means changes to the BTree layer. The current pager allocates some > extra memory immediately before and after the data page. And it tracks > each page using just a pointer to the data - since it can always find > the header and tailer using pointer arithmetic. That won't work with > memmap(). You'd have to relocate this extra stuff into a separate > allocation which you then track separately. right. when i looked at it, it just didn't seem like it would be all that time-efficient (wrt implementation work) to decouple the page header and EXTRA from the page itself. > Also, you'll have to take > out an exclusive lock on the database before making any changes to the > data buffers in memory, not just before calling write() as in the > current > solution. That will cut your currency some because it means taking an > exclusive lock much earlier in the write process. personally, i don't specifically care, since my usage pattern will be heavily trended towards reads (if i do any writes at all). however, this is indeed a valid concern. i'm sure something can be thought up here if there's user demand, but i don't think it makes sense to deal with this from the start (premature optimization and all that). > A new pager API that works for memmap() would also continue to work > for the old read()/write() method. So presumably you could create > a new pragma that lets the programmer decide at run-time which method > to use. Use memmap() for faster reads and less memory usage or > use read/write for better concurrency. i was actually going to do something like ":mmap:/path/to/db", in keeping with the :memory: feature, since this whole excercise started with me trying to determine whether i can tweak the :memory: implementation to do what i want. i might put in some work on this, although it's a bit more work than i'd hoped (shm would have been faster afair). will this patch, assuming it looks decent and works (for a reasonable threshold of 'works'), be accepted into trunk? it's a bit too much to do just for myself, but assuming others would benefit as well, i might be up for it. cheers, -p
