John Stanton wrote:
> I wonder if members can help me with some advice. I have a program
> which is a multi-threaded application server with Sqlite embedded which
> runs on Unix and Windows. For an i/o buffer per thread I have the idea
> of using a mmap'd file so that it can be transferred using
> sendfile/TransmitFile with minimum overhead and no buffer shadowing.
> The program design is such that there is a pool of inactive threads
> waiting for a connection, and that pool could be quite large. Each one
> would have a mmap'd file with an open fd involved.
>
> Does anyone know the performance impact of having a considerable number
> of mmap'd files attached to a single process in Unix/Linux and Windows?
> Has anyone tried such a strategy? My guess is that the impact is not
> great, but I have no evidence in support.
> JS
>
You have a lot of work ahead of you if you want to modify
SQLite to use MMAP. SQLite was not designed for this. What
are you trying to accomplish?
I have no idea what the performance implications of using
MMAP are. The only way to know is to try it and see.
===
Thankyou for the response. I am not modifying Sqlite, just embedding it
in an application server, and looking at ways to optimally construct the
server. It seems to me ideal to use zero buffering capabilities of
the host OS if possible and to use a mmap'd file as a buffer and hold
its fd so that it can be sent to the network without further buffer
copying. Since it is difficult to set up realistic tests to simulate
high volumes and loads I was curious if others had tried this approach
and experienced any unexpected side effects in practice.
BTW, Sqlite fits very neatly into such an environment. This particular
application has hundreds of databases, each with a handful of users, a
situation where Sqlite's single file simplicity is a great asset and no
impediment to high performance. By having a purpose-designed server and
application specific language plus the unbloated Sqlite I am looking to
achieve higher performance and efficiency than by cobbling it together
from some general purpose language processor and generic web server.
The recent discussion about locality of reference made me cautious. My
concern is about having local memory references in the program held up
by thrashing in virtual memory and losing more than would be gained by
the more efficient sendfile type transfer.
The system I am replacing was put together on the basis that there were
plenty of resources to compensate for each added inefficiency, until one
day it did matter - the camel's back eventually broke.
JS
