Hello James, It is hard to respond because I don't know what a PV driver is or how it is related to network drivers.
On Tuesday 14 April 2009 08:41:37 James Harper wrote: > I'm writing Windows PV drivers for Xen, and bacula is the only app that > I have found that can reliably trigger a certain bug in the network > driver. While debugging that, I have noticed that bacula may not be > sending data in the most efficient way possible... That depends on how you define "the most efficient way possible". Bacula makes a large number of small network I/O calls, but it expects the OS to deal with getting them efficiently across the network. That said, there were some important optimizations between 2.2.8 and 3.0.0, which reduce the number of network writes by a factor of 2. > > Basically, when my drivers receive a packet, the packet consists of a > whole bunch of buffers chained together. The packets I get from Bacula > contain quite a few small chunks, eg a sample packet consists of a large > number of 4 byte and 12 byte buffers (15 of each in a single 10k TCP > packet). It's fairly easy to see the performance difference of using > smaller buffer sizes by using the iperf tool. I'm not sure if Linux > handles things in the same way though so this might not apply there... I am not quite sure what you mean by small buffer sizes. There are several different parameters that one must clearly distinguish. Bacula does send network I/O in sizes that are convenient for it. These are not "buffered", but simply reads and writes of the socket. This involves a certain number of small I/O sizes (4 bytes +). However, once a file is being transferred, Bacula packs the file into network buffers and sends as much as it can. The network buffer size is determined by what the OS permits. By default Bacula starts with a 64K size and reduces it if the OS indicates that it is too large. This size can be defined by the user to optimize I/O performance. Perhaps across the Internet the small read/write requests could be inefficient, but within a 100Mb or Gigabit network, they don't have much importance (as far as I know). > > I think that there could be some performance benefit in keeping a > per-socket buffer and only sending data onto the wire when we have > enough, at least for the fd->sd connection. Has this been considered > before? There are very few circumstances in Bacula when you can buffer socket data, because most of the communications are interactive -- that is Bacula sends some data and expects a response. During saving or restoring a file, Bacula packs the data into buffers (the size can be determined by the user). We would be happy to consider something more, but I would recommend two things: 1. Do your testing with version 3.0.0. 2. Test real file transfers and get some performance data that shows that it is really a bottle neck. At least for a local network, I am sure you will find that the comm part of Bacula is not the restraining factor. > The system I'm seeing this on is running 2.2.8 so it's pretty > old... maybe buffering has already been implemented in later versions? Buffering of file data is implemented in all versions (the user can control the sizes, so some experimenting is needed -- perhaps on your network our defaults are not optimum). Reducing the number of socket writes by a factor of two was implemented in 3.0.0 (possibly before). > (A quick look at bnet.c and bsock.c doesn't reveal anything obvious > though...) > Unfortunately, not much is evident in Bacula -- simply because what it is doing is rather complicated. Regards, Kern ------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ Bacula-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/bacula-devel
