In my experience under windows, using memory mapped files is no faster than opening and reading the file into memory then processing it. The limiting factor is the disk drives. I use both methods and the choice of method is always a tossup. If the API I'm using will accept an entire buffer for the file, say a JPG decode, I'll use mmap (windows equivalent) otherwise I may just read it a chunk at a time.
My impression is, he's asking a general MMAP versus fopen question and not planning on integrating MMAP into SQLite. C Sunday, April 16, 2006, 10:00:57 AM, you wrote: JS> John Stanton wrote: >> I wonder if members can help me with some advice. I have a program >> which is a multi-threaded application server with Sqlite embedded which >> runs on Unix and Windows. For an i/o buffer per thread I have the idea >> of using a mmap'd file so that it can be transferred using >> sendfile/TransmitFile with minimum overhead and no buffer shadowing. >> The program design is such that there is a pool of inactive threads >> waiting for a connection, and that pool could be quite large. Each one >> would have a mmap'd file with an open fd involved. >> >> Does anyone know the performance impact of having a considerable number >> of mmap'd files attached to a single process in Unix/Linux and Windows? >> Has anyone tried such a strategy? My guess is that the impact is not >> great, but I have no evidence in support. >> JS >> JS> You have a lot of work ahead of you if you want to modify JS> SQLite to use MMAP. SQLite was not designed for this. What JS> are you trying to accomplish? JS> I have no idea what the performance implications of using JS> MMAP are. The only way to know is to try it and see. JS> === JS> Thankyou for the response. I am not modifying Sqlite, just embedding it JS> in an application server, and looking at ways to optimally construct the JS> server. It seems to me ideal to use zero buffering capabilities of JS> the host OS if possible and to use a mmap'd file as a buffer and hold JS> its fd so that it can be sent to the network without further buffer JS> copying. Since it is difficult to set up realistic tests to simulate JS> high volumes and loads I was curious if others had tried this approach JS> and experienced any unexpected side effects in practice. JS> BTW, Sqlite fits very neatly into such an environment. This particular JS> application has hundreds of databases, each with a handful of users, a JS> situation where Sqlite's single file simplicity is a great asset and no JS> impediment to high performance. By having a purpose-designed server and JS> application specific language plus the unbloated Sqlite I am looking to JS> achieve higher performance and efficiency than by cobbling it together JS> from some general purpose language processor and generic web server. JS> The recent discussion about locality of reference made me cautious. My JS> concern is about having local memory references in the program held up JS> by thrashing in virtual memory and losing more than would be gained by JS> the more efficient sendfile type transfer. JS> The system I am replacing was put together on the basis that there were JS> plenty of resources to compensate for each added inefficiency, until one JS> day it did matter - the camel's back eventually broke. JS> JS -- Best regards, Teg mailto:[EMAIL PROTECTED]
