RE: TCP Tuning
I have tested with a larger sample (40G) and the results were the same. There is a single route between the hosts so all traffic traverses it. There appears to be no resource issue on the backup client iostat/vmstat reports low utilisation. netstat reports not errors - but this would impact NFS as well. Thanks, Alan Date: Thu, 29 Oct 2009 16:09:16 -0400 From: martin...@zmanda.com To: ap_griffi...@hotmail.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: Re: TCP Tuning Alan, Most people report faster throughput without nfs, your result is weird. Your sample is small (1.5GB), is it possible it was cached by the nfs client? Do amanda and NFS use the same network route? Do the NFS server have enough memory and cpu to run the amanda client software? Check the network cards, do they report error? Jean-Louis Alan Griffiths wrote: This time with files *actually* attached! From: ap_griffi...@hotmail.com To: martin...@zmanda.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: RE: TCP Tuning Date: Thu, 22 Oct 2009 17:14:33 +0100 Just one dle. No compression - data is already compressed (gzip). No encryption. I am using holding disk. Attached files: - amdump.1 direct from client amdump.3 through NFS. _ Access your other email accounts and manage all your email from one place. http://clk.atdmt.com/UKM/go/167688463/direct/01/
Re: TCP Tuning
Alan, Most people report faster throughput without nfs, your result is weird. Your sample is small (1.5GB), is it possible it was cached by the nfs client? Do amanda and NFS use the same network route? Do the NFS server have enough memory and cpu to run the amanda client software? Check the network cards, do they report error? Jean-Louis Alan Griffiths wrote: This time with files *actually* attached! From: ap_griffi...@hotmail.com To: martin...@zmanda.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: RE: TCP Tuning Date: Thu, 22 Oct 2009 17:14:33 +0100 Just one dle. No compression - data is already compressed (gzip). No encryption. I am using holding disk. Attached files: - amdump.1 direct from client amdump.3 through NFS.
RE: TCP Tuning
amgtar with blocksize set to 512 performs at the same speed. If I mount the partition on the backup server via NFS then the backup runs about 6x quicker. So I'm assuming it must be something related to way AMANDA sets up up the TCP connection. Although obviously open to other suggestions. Alan Date: Wed, 21 Oct 2009 06:59:01 -0400 From: martin...@zmanda.com To: ap_griffi...@hotmail.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: Re: TCP Tuning Use the amgtar application and set the TAR-BLOCKSIZE to a bigger value (half the STREAM_BUFSIZE. Why do you believe it is bottleneck? Jean-Louis Alan Griffiths wrote: Thanks for the pointer. I've re-compiled with define STREAM_BUFSIZE (NETWORK_BLOCK_BYTES * 16) And I can see that has taken effect on the server dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Thanks, Alan Date: Mon, 19 Oct 2009 10:03:21 -0400 Subject: Re: TCP Tuning From: dus...@zmanda.com To: ap_griffi...@hotmail.com CC: amanda-users@amanda.org On Mon, Oct 19, 2009 at 5:33 AM, Alan Griffiths wrote: Is there a way to modify the size of the TCP buffers used by AMANDA? I am trying to improve performance over a relatively high latency link and this seems to be the only way. It's a source constant, unfortunately, set in stream.h (STREAM_BUFSIZE). A patch to make that configurable at compile time or, better, at runtime would be much appreciated! Dustin -- Open Source Storage Engineer http://www.zmanda.com Download Messenger onto your mobile for free. Learn more. _ New Windows 7: Find the right PC for you. Learn more. http://www.microsoft.com/windows/buy/
Re: TCP Tuning
You talk about one dle or multiple dle? Are you using compression or encryption? on client or server? Are you using holding disk? or dumping directly to tape? Post the amdump.n file for when it use NFS and when it doesn't use it. Jean-Louis Alan Griffiths wrote: amgtar with blocksize set to 512 performs at the same speed. If I mount the partition on the backup server via NFS then the backup runs about 6x quicker. So I'm assuming it must be something related to way AMANDA sets up up the TCP connection. Although obviously open to other suggestions. Alan Date: Wed, 21 Oct 2009 06:59:01 -0400 From: martin...@zmanda.com To: ap_griffi...@hotmail.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: Re: TCP Tuning Use the amgtar application and set the TAR-BLOCKSIZE to a bigger value (half the STREAM_BUFSIZE. Why do you believe it is bottleneck? Jean-Louis Alan Griffiths wrote: Thanks for the pointer. I've re-compiled with define STREAM_BUFSIZE (NETWORK_BLOCK_BYTES * 16) And I can see that has taken effect on the server dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Thanks, Alan Date: Mon, 19 Oct 2009 10:03:21 -0400 Subject: Re: TCP Tuning From: dus...@zmanda.com To: ap_griffi...@hotmail.com CC: amanda-users@amanda.org On Mon, Oct 19, 2009 at 5:33 AM, Alan Griffiths wrote: Is there a way to modify the size of the TCP buffers used by AMANDA? I am trying to improve performance over a relatively high latency link and this seems to be the only way. It's a source constant, unfortunately, set in stream.h (STREAM_BUFSIZE). A patch to make that configurable at compile time or, better, at runtime would be much appreciated! Dustin -- Open Source Storage Engineer http://www.zmanda.com Download Messenger onto your mobile for free. Learn more. _ New Windows 7: Find the right PC for you. Learn more. http://www.microsoft.com/windows/buy/
RE: TCP Tuning
Just one dle. No compression - data is already compressed (gzip). No encryption. I am using holding disk. Attached files: - amdump.1 direct from client amdump.3 through NFS. Date: Thu, 22 Oct 2009 11:58:57 -0400 From: martin...@zmanda.com To: ap_griffi...@hotmail.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: Re: TCP Tuning You talk about one dle or multiple dle? Are you using compression or encryption? on client or server? Are you using holding disk? or dumping directly to tape? Post the amdump. file for when it use NFS and when it doesn't use it. Jean-Louis Alan Griffiths wrote: amgtar with blocksize set to 512 performs at the same speed. If I mount the partition on the backup server via NFS then the backup runs about 6x quicker. So I'm assuming it must be something related to way AMANDA sets up up the TCP connection. Although obviously open to other suggestions. Alan Date: Wed, 21 Oct 2009 06:59:01 -0400 From: martin...@zmanda.com To: ap_griffi...@hotmail.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: Re: TCP Tuning Use the amgtar application and set the TAR-BLOCKSIZE to a bigger value (half the STREAM_BUFSIZE. Why do you believe it is bottleneck? Jean-Louis Alan Griffiths wrote: Thanks for the pointer. I've re-compiled with define STREAM_BUFSIZE (NETWORK_BLOCK_BYTES * 16) And I can see that has taken effect on the server dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Thanks, Alan Date: Mon, 19 Oct 2009 10:03:21 -0400 Subject: Re: TCP Tuning From: dus...@zmanda.com To: ap_griffi...@hotmail.com CC: amanda-users@amanda.org On Mon, Oct 19, 2009 at 5:33 AM, Alan Griffiths wrote: Is there a way to modify the size of the TCP buffers used by AMANDA? I am trying to improve performance over a relatively high latency link and this seems to be the only way. It's a source constant, unfortunately, set in stream.h (STREAM_BUFSIZE). A patch to make that configurable at compile time or, better, at runtime would be much appreciated! Dustin -- Open Source Storage Engineer http://www.zmanda.com Download Messenger onto your mobile for free. Learn more. _ New Windows 7: Find the right PC for you. Learn more. http://www.microsoft.com/windows/buy/ _ Chat to your friends for free on selected mobiles http://clk.atdmt.com/UKM/go/174426567/direct/01/
RE: TCP Tuning
This time with files *actually* attached! From: ap_griffi...@hotmail.com To: martin...@zmanda.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: RE: TCP Tuning Date: Thu, 22 Oct 2009 17:14:33 +0100 Just one dle. No compression - data is already compressed (gzip). No encryption. I am using holding disk. Attached files: - amdump.1 direct from client amdump.3 through NFS. Date: Thu, 22 Oct 2009 11:58:57 -0400 From: martin...@zmanda.com To: ap_griffi...@hotmail.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: Re: TCP Tuning You talk about one dle or multiple dle? Are you using compression or encryption? on client or server? Are you using holding disk? or dumping directly to tape? Post the amdump. file for when it use NFS and when it doesn't use it. Jean-Louis Alan Griffiths wrote: amgtar with blocksize set to 512 performs at the same speed. If I mount the partition on the backup server via NFS then the backup runs about 6x quicker. So I'm assuming it must be something related to way AMANDA sets up up the TCP connection. Although obviously open to other suggestions. Alan Date: Wed, 21 Oct 2009 06:59:01 -0400 From: martin...@zmanda.com To: ap_griffi...@hotmail.com CC: dus...@zmanda.com; amanda-users@amanda.org Subject: Re: TCP Tuning Use the amgtar application and set the TAR-BLOCKSIZE to a bigger value (half the STREAM_BUFSIZE. Why do you believe it is bottleneck? Jean-Louis Alan Griffiths wrote: Thanks for the pointer. I've re-compiled with define STREAM_BUFSIZE (NETWORK_BLOCK_BYTES * 16) And I can see that has taken effect on the server dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Thanks, Alan Date: Mon, 19 Oct 2009 10:03:21 -0400 Subject: Re: TCP Tuning From: dus...@zmanda.com To: ap_griffi...@hotmail.com CC: amanda-users@amanda.org On Mon, Oct 19, 2009 at 5:33 AM, Alan Griffiths wrote: Is there a way to modify the size of the TCP buffers used by AMANDA? I am trying to improve performance over a relatively high latency link and this seems to be the only way. It's a source constant, unfortunately, set in stream.h (STREAM_BUFSIZE). A patch to make that configurable at compile time or, better, at runtime would be much appreciated! Dustin -- Open Source Storage Engineer http://www.zmanda.com Download Messenger onto your mobile for free. Learn more. _ New Windows 7: Find the right PC for you. Learn more. http://www.microsoft.com/windows/buy/ _ Chat to your friends for free on selected mobiles http://clk.atdmt.com/UKM/go/174426567/direct/01/ _ New Windows 7: Simplify what you do everyday. Find the right PC for you. http://www.microsoft.com/windows/buy/ amdump.1 Description: Binary data amdump.3 Description: Binary data
Re: TCP Tuning
Use the amgtar application and set the TAR-BLOCKSIZE to a bigger value (half the STREAM_BUFSIZE. Why do you believe it is bottleneck? Jean-Louis Alan Griffiths wrote: Thanks for the pointer. I've re-compiled with define STREAM_BUFSIZE (NETWORK_BLOCK_BYTES * 16) And I can see that has taken effect on the server dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Thanks, Alan Date: Mon, 19 Oct 2009 10:03:21 -0400 Subject: Re: TCP Tuning From: dus...@zmanda.com To: ap_griffi...@hotmail.com CC: amanda-users@amanda.org On Mon, Oct 19, 2009 at 5:33 AM, Alan Griffiths ap_griffi...@hotmail.com wrote: Is there a way to modify the size of the TCP buffers used by AMANDA? I am trying to improve performance over a relatively high latency link and this seems to be the only way. It's a source constant, unfortunately, set in stream.h (STREAM_BUFSIZE). A patch to make that configurable at compile time or, better, at runtime would be much appreciated! Dustin -- Open Source Storage Engineer http://www.zmanda.com Download Messenger onto your mobile for free. Learn more. http://clk.atdmt.com/UKM/go/174426567/direct/01/
RE: TCP Tuning
Thanks for the pointer. I've re-compiled with define STREAM_BUFSIZE (NETWORK_BLOCK_BYTES * 16) And I can see that has taken effect on the server dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Thanks, Alan Date: Mon, 19 Oct 2009 10:03:21 -0400 Subject: Re: TCP Tuning From: dus...@zmanda.com To: ap_griffi...@hotmail.com CC: amanda-users@amanda.org On Mon, Oct 19, 2009 at 5:33 AM, Alan Griffiths ap_griffi...@hotmail.com wrote: Is there a way to modify the size of the TCP buffers used by AMANDA? I am trying to improve performance over a relatively high latency link and this seems to be the only way. It's a source constant, unfortunately, set in stream.h (STREAM_BUFSIZE). A patch to make that configurable at compile time or, better, at runtime would be much appreciated! Dustin -- Open Source Storage Engineer http://www.zmanda.com _ Download Messenger onto your mobile for free http://clk.atdmt.com/UKM/go/174426567/direct/01/
RE: TCP Tuning
In the amandad log from 2.5.0p2 I see the following amandad: try_socksize: send buffer size is 65536 amandad: try_socksize: receive buffer size is 65536 amandad: time 0.006: bind_portrange2: trying port=757 amandad: time 0.006: stream_server: waiting for connection: 0.0.0.0.45627 ...OK, I've just built a 2.5.0p2 client with a larger STERAM_BUFSIZE and now I see this in the amandad log amandad: try_socksize: send buffer size is 524288 amandad: try_socksize: receive buffer size is 524288 amandad: time 18.040: bind_portrange2: trying port=659 amandad: time 18.040: stream_server: waiting for connection: 0.0.0.0.36030 Unfortunately it does not result in the backup running any faster. I will have a look at hacking amandad.c as you suggest. My fallback option is to NFS mount the directories on the backup server. Would prefer to not to do that, but have to make these backups run quicker one way or another. Alan Date: Tue, 20 Oct 2009 12:43:24 -0400 Subject: Re: TCP Tuning From: dus...@zmanda.com To: ap_griffi...@hotmail.com CC: amanda-users@amanda.org On Tue, Oct 20, 2009 at 11:58 AM, Alan Griffiths ap_griffi...@hotmail.com wrote: dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Hmm, I can't find that indication in any of my older logs. From what I can tell, because amandad just uses the handles (x)inetd gives it, it never sents the buffer size. From my read, there is a bit of an abuse of the security API when amandad calls security_accept on an already-connected socket (rather than the half-open connection that the accept() syscall expects), so I expect the easiest fix for your purposes is to add try_socksize calls to protocol_accept in amandad.c. Do you want to give that a try? Dustin -- Open Source Storage Engineer http://www.zmanda.com _ Did you know you can get Messenger on your mobile? http://clk.atdmt.com/UKM/go/174426567/direct/01/
Re: TCP Tuning
On Tue, Oct 20, 2009 at 1:42 PM, Alan Griffiths ap_griffi...@hotmail.com wrote: Unfortunately it does not result in the backup running any faster. I will have a look at hacking amandad.c as you suggest. My fallback option is to NFS mount the directories on the backup server. Would prefer to not to do that, but have to make these backups run quicker one way or another. Ah, my old logs don't stretch back to 2.5.0 :) Did a tcpdump demonstrate that the buffer sizes were actually in effect? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: TCP Tuning
On Tue, Oct 20, 2009 at 11:58 AM, Alan Griffiths ap_griffi...@hotmail.com wrote: dumper: try_socksize: send buffer size is 524288 But backups are running no faster and I cannot see any indication on the client of the buffer size being used. Note: client also has the new binaries. In older versions of AMANDA amandad used to report buffer size, but this appears to not be the case in 2.6.1p1. Hmm, I can't find that indication in any of my older logs. From what I can tell, because amandad just uses the handles (x)inetd gives it, it never sents the buffer size. From my read, there is a bit of an abuse of the security API when amandad calls security_accept on an already-connected socket (rather than the half-open connection that the accept() syscall expects), so I expect the easiest fix for your purposes is to add try_socksize calls to protocol_accept in amandad.c. Do you want to give that a try? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: TCP Tuning
On Mon, Oct 19, 2009 at 5:33 AM, Alan Griffiths ap_griffi...@hotmail.com wrote: Is there a way to modify the size of the TCP buffers used by AMANDA? I am trying to improve performance over a relatively high latency link and this seems to be the only way. It's a source constant, unfortunately, set in stream.h (STREAM_BUFSIZE). A patch to make that configurable at compile time or, better, at runtime would be much appreciated! Dustin -- Open Source Storage Engineer http://www.zmanda.com
