Hi, 14.04.2009 12:27, James Harper wrote: >> Hi, >> >> 14.04.2009 09:23, Kern Sibbald wrote: >>> Hello James, >>> >>> It is hard to respond because I don't know what a PV driver is or > how it is >>> related to network drivers. >> Neither do I :-) > > ParaVirtualised drivers. Much faster than an emulation of a PCI device > in a virtual machine.
Thanks... with the explanation I understand which meaning of PV you refer to :-) >>> On Tuesday 14 April 2009 08:41:37 James Harper wrote: >>>> I'm writing Windows PV drivers for Xen, and bacula is the only app > that >>>> I have found that can reliably trigger a certain bug in the network >>>> driver. While debugging that, I have noticed that bacula may not be >>>> sending data in the most efficient way possible... >>> That depends on how you define "the most efficient way possible". >>> >>> Bacula makes a large number of small network I/O calls, but it > expects the >> OS >>> to deal with getting them efficiently across the network. >>> That said, there were some important optimizations between 2.2.8 and > 3.0.0, >>> which reduce the number of network writes by a factor of 2. >> This seems to be present in 2.5.4x as I can't see what James describes >> in my test setup. > > If you are using wireshark then you won't. I'm talking about one level > below what you would see with a packet sniffer. Ah, ok... though I wonder how that works. I'm not a programmer, so please assume I'm stupid: FD puts something like "I want to start a job" into a packet to the SD and writes that packet to the socket. The FD needs the reply before it can continue it's work, right? So how can the network stack coalesce anything here? > When a windows app (probably similar to Linux) sends data to a socket, > windows will not necessarily send the data straight away but will > instead coalesce the data into (up to) a ~61K to send to the hardware > (or 1460 bytes if the hardware doesn't support large send offloading). > Because Bacula is doing lots of small writes, the network driver is > handed a chained list of buffers for the size of each write, so for > example in one case we have separate writes of 12, 195, 4, 11, 20, 4, > 11, 234, 4, 12, 194, 4, 11, 20, 4, 11, 234, 4, 12, 238, 4, 11, 20, 4, > 11, and 171 bytes, so we get a list of 26 fairly tiny buffers. That's something I understand, and I guess this really happens at a lower level inside the FD itself... I guess I kind of get the idea now. > My driver has a limit of 19 buffers that it can handle, and so in the > above case Windows has to copy a whole load of data around to make up a > 1514 byte packet. Also, there are only 256 'slots' in the communications > ring with the virtual hardware, and each buffer uses up one of those > 'slots' leaving less room for other packets. Sounds like some limitations in the driver framework / virtualization environment. Which does not imply that it's not something worth working on or rather around, obviously. > A physical network adapter probably has less restrictions on > scatter-gather lists but there is still going to be overhead with making > a packet out of lots of small buffers vs one big one. I'll not go through the source code :-) >>> We would be happy to consider something more, but I would recommend > two >>> things: >>> >>> 1. Do your testing with version 3.0.0. >> Seems well worth it, considering the difference of my observations to >> what James reports. >> >>> 2. Test real file transfers and get some performance data that shows > that it >>> is really a bottle neck. At least for a local network, I am sure > you will >>> find that the comm part of Bacula is not the restraining factor. >> Agreed :-) >> > > I'll try both of those. Now I'm curious what you find :-) Cheers, Arno > Thanks > > James > > -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück www.its-lehmann.de ------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ Bacula-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/bacula-devel
