Hi Doug, I really like the idea of reading extra header!
I agree that an extra copy is very expensive and I too expect that the associated overhead would cancel the gains of reduced number of system calls. My previous email lacked an explanation of how to deliver buffers to application without copying. One idea I have been thinking about is to have a set of fixed size buffers and load messages there. Once packets are laid out in the buffer, the application would be delivered pointers to packets within these buffers. The buffers would be recycled as soon as the application is done with handling requests. The downsides of this approach would be dangling pointers, buffer management, etc. but I do not want to consider them for now in this e-mail. The upsides are smaller number of memory allocations, and lower number of system calls issued. Mateusz On Wed, Apr 28, 2010 at 2:11 PM, Doug Judd <[email protected]> wrote: > Hi Mateusz, > Good observation. The one problem I see with your proposal is that the > extra copy from the circular buffer into a buffer that can be delivered up > into the application would be too expensive. I suspect it would dwarf any > savings you got by reducing the number of system calls. One idea I've been > meaning to implement is to have the comm layer read an extra 36 bytes > (header size) when reading the payload so that the next header gets read > along with it. This would cut the number of read requests in half, under > heavy load. > - Doug > On Wed, Apr 28, 2010 at 1:46 PM, Mateusz Berezecki <[email protected]> > wrote: >> >> Hi, >> >> I've been tinkering around AsyncComm and I realized that performance >> of message reading is suboptimal. >> >> The current scheme works like this: >> >> 1. reactor calls data dispatcher's handle_event >> 2. message handler tries to read the header from socket (sometimes >> using partial reads) via read() >> 3. once the header is read, dispatcher tries to read the message body >> (possibly using partial reads) via read() >> >> There's also a memory allocation happening for every packet to make >> room for packet body. >> >> The performance degradation of this scheme starts becoming visible >> under heavy load (i.e. high volume of small messages being processed). >> >> The degradation is mostly due to excessive system calls and userspace >> / kernelspace context switches that are inseparably linked to read >> system calls. It's also worth noting that linux does not use SYSCALL >> instructions for x86 cpus inside the kernel, but generic interrupt >> driven mechanism unless CPU type is explicitly configured to the CPU >> type pentium 4 or better. Configuring kernel to use SYSCALL/SYSRET are >> somewhat faster but if the number of system calls is large the gain is >> negligible. >> >> Do you think it would be better to read as much data as possible into >> a fixed cyclic buffer, process messages from that buffer to also avoid >> excessive memory allocations and allocate event objects from some pool >> allocator and send them for dispatch ? >> >> What do you think? >> >> Mateusz >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Hypertable Development" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/hypertable-dev?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
