Chris, Thanks for the very pertinent comments on segment linking.
I am not sending an Sqlite database. It is the output from my program,
principally assembled HTML/PDF/PostScript and similar.
I want to avoid buffer shadowing on the content my program creates. If
I use a malloc'd buffer it gets copied several times before arriving at
the network interface. If I use the zero copy mechanism with a sendfile
or TransmitFile on an open file descriptor/handle in conjunction with a
modern network adapter then the transfer is optimal.
My concern was in line with your comments on VM management by the OS.
In general terms the mapping of files into VM gives an OS a means of
sharing executables between processes and is not necessarily designed to
handle many more files particularly in READWRITE rather than READONLY
mode. With that in mind I was interested to hear if anyone had hit such
limits and their impact.
It has always been my experience that if you get the underlying
mechanisms used by your application to be optimal, then you have no
scaling problems and no walls to hit in the future.
JS
Christian Smith wrote:
On Sun, 16 Apr 2006, John Stanton wrote:
I wonder if members can help me with some advice. I have a program
which is a multi-threaded application server with Sqlite embedded which
runs on Unix and Windows. For an i/o buffer per thread I have the idea
of using a mmap'd file so that it can be transferred using
sendfile/TransmitFile with minimum overhead and no buffer shadowing.
The program design is such that there is a pool of inactive threads
waiting for a connection, and that pool could be quite large. Each one
would have a mmap'd file with an open fd involved.
Is the SQLite database file what you want to send down the socket? Or are
you just building up a buffer of your regular session protocol data to
send?
If the former, then you'll have problems with synchronisation unless you
stop all database processing while sending the file. You'll also have to
wait until the file is static (locked) before MMAP'ing it, so you'll know
how much to MMAP. There will be setup cost associated with MMAP the file.
MMAP is probably not optimised to be done on demand. Instead, it is
designed for up front persistant mappings of known size, like libraries
and memory mapped devices.
If the latter, assuming you don't need the session data to be persistent,
then what's wrong with using just an anonymous memory buffer from malloc?
The buffer is anonymous and so will only ever be swapped, so no file IO
at all unless you have memory pressure. sendfile will gain you nothing if
you don't require persistent data.
Does anyone know the performance impact of having a considerable number
of mmap'd files attached to a single process in Unix/Linux and Windows?
Has anyone tried such a strategy? My guess is that the impact is not
great, but I have no evidence in support.
The OS has to track each MMAP segment, which is usually a linear linked
list sorted by virtual address. As most processes don't have that many
segments mapped (10's of segments would be considered a lot) the O(n)
linear search is not considered much of a burden. If this increases to the
100's or 1000's of segments, the kernel will spend correspondingly more
time in VM management. Such linear searches can also have a detrimental
effect on the CPU caching.
All in all, my guess is that the complications would not be worth any
performance gains you may achieve. It sounds like a micro-optimisation
(sendfile itself is really only useful for static content.)
JS
Christian
