My experience differs on this one. When processing large amounts of data with a processing function, which takes longer than the I/O using a mapped file is faster than sequential I/O, as the Windows memory manager and caching system will pull in the I/O pages asynchronously. At least on Windows I'd say using mapped files for really large data transfers is reasonable and most likely faster, than using sequential I/O using any other method.
Mike -----Ursprüngliche Nachricht----- Von: Teg [mailto:[EMAIL PROTECTED] Gesendet: Sonntag, 16. April 2006 16:55 An: John Stanton Cc: sqlite-users@sqlite.org Betreff: Re[2]: [sqlite] MMAP In my experience under windows, using memory mapped files is no faster than opening and reading the file into memory then processing it. The limiting factor is the disk drives. I use both methods and the choice of method is always a tossup. If the API I'm using will accept an entire buffer for the file, say a JPG decode, I'll use mmap (windows equivalent) otherwise I may just read it a chunk at a time. My impression is, he's asking a general MMAP versus fopen question and not planning on integrating MMAP into SQLite. C Sunday, April 16, 2006, 10:00:57 AM, you wrote: JS> John Stanton wrote: >> I wonder if members can help me with some advice. I have a program >> which is a multi-threaded application server with Sqlite embedded which >> runs on Unix and Windows. For an i/o buffer per thread I have the idea >> of using a mmap'd file so that it can be transferred using >> sendfile/TransmitFile with minimum overhead and no buffer shadowing. >> The program design is such that there is a pool of inactive threads >> waiting for a connection, and that pool could be quite large. Each one >> would have a mmap'd file with an open fd involved. >> >> Does anyone know the performance impact of having a considerable number >> of mmap'd files attached to a single process in Unix/Linux and Windows? >> Has anyone tried such a strategy? My guess is that the impact is not >> great, but I have no evidence in support. >> JS >> JS> You have a lot of work ahead of you if you want to modify SQLite to JS> use MMAP. SQLite was not designed for this. What are you trying to JS> accomplish? JS> I have no idea what the performance implications of using MMAP are. JS> The only way to know is to try it and see. JS> === JS> Thankyou for the response. I am not modifying Sqlite, just JS> embedding it in an application server, and looking at ways to optimally construct the JS> server. It seems to me ideal to use zero buffering capabilities JS> of the host OS if possible and to use a mmap'd file as a buffer and JS> hold its fd so that it can be sent to the network without further JS> buffer copying. Since it is difficult to set up realistic tests to JS> simulate high volumes and loads I was curious if others had tried JS> this approach and experienced any unexpected side effects in practice. JS> BTW, Sqlite fits very neatly into such an environment. This JS> particular application has hundreds of databases, each with a JS> handful of users, a situation where Sqlite's single file simplicity JS> is a great asset and no impediment to high performance. By having a JS> purpose-designed server and application specific language plus the JS> unbloated Sqlite I am looking to achieve higher performance and JS> efficiency than by cobbling it together from some general purpose language processor and generic web server. JS> The recent discussion about locality of reference made me cautious. JS> My concern is about having local memory references in the program JS> held up by thrashing in virtual memory and losing more than would be JS> gained by the more efficient sendfile type transfer. JS> The system I am replacing was put together on the basis that there JS> were plenty of resources to compensate for each added inefficiency, JS> until one day it did matter - the camel's back eventually broke. JS> JS -- Best regards, Teg mailto:[EMAIL PROTECTED]
