Re: [BackupPC-users] Using NFS to increase backup speed
-Original Message- From: Koen Vermeer [mailto:k...@vermeer.tv] Sent: den 17 oktober 2014 21:49 To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] Using NFS to increase backup speed [root@cyndane ~]# yes | pv | ssh titan cat /dev/null 1.16GiB 0:01:29 [14.3MiB/s] [root@cyndane ~]# I searched and read som more after posting the question, and I realize it's not that simple fixing this, as I initially thought... I'd tunnel the port your use for iperf (ssh -L 5001:localhost:5001 cyndane) and run the same test as you initially did (iperf -c localhost). I expect that you'll see a value that's larger than the 35-45 Mbps you get for rsync+ssh, meaning that rsync is the bottleneck. And most of the time, that makes sense, because it's often doing other things than transferring data. I'll give it a go. Thanks! -- //Sorin -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
-Original Message- From: Les Mikesell [mailto:lesmikes...@gmail.com] Sent: den 17 oktober 2014 16:57 To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] Using NFS to increase backup speed I was worried the full backup wouldn't complete in the limited time the BPC server in online. For practical reasons (well, because of the hd-space available really), the server used is off during nights in order to save some electricity. The first backup took some 17 hours... Today, I just checked, the first incremental backup took just under three hours, which is completely okay. I'll need to see how the next full backup performs with the speed tweaks I've done so far. Incrementals will quickly skip over files where the timestamp and length match the copy in the previous full run. They do transfer more each time until the next full sets a new comparision base. So the times you need to be concerned about are the last (usually 6th) incremental in the set, and the 3rd full run after setting the checksum-seed option). These should show what to expect for subsequent times. Yupp, I noticed. Will take another three or four days till the 6th backup shows. So far, so good! -- //Sorin -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
-Original Message- From: Koen Vermeer [mailto:k...@vermeer.tv] Sent: den 16 oktober 2014 17:29 To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] Using NFS to increase backup speed What about testing the speed over the ssh tunnel? That may tell you whether ssh is slowing down your transfers or that it's due to rsync. If it is ssh, you could trade encryption strength for speed. Hi and thanks, Would the below do? [root@cyndane ~]# yes | pv | ssh titan cat /dev/null 1.16GiB 0:01:29 [14.3MiB/s] [root@cyndane ~]# I searched and read som more after posting the question, and I realize it's not that simple fixing this, as I initially thought... -- //Sorin -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
-Original Message- From: Colin Shorts [mailto:c.sho...@intrallect.com] Sent: den 16 oktober 2014 17:58 To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] Using NFS to increase backup speed You may wish to prevent ssh from using compression when using a fast link, the overhead probably isn't worth it and may give you a reasonable boost in throughput. Disabling compression in sshd_config didn't do much, maybe .1-.2 MiB increase. However using arcfour did produce a significant increase as seen below. [root@cyndane ~]# yes | pv | ssh titan -c arcfour cat /dev/null 389MiB 0:00:23 [20.8MiB/s] I'll add arcfour to the ssh-arg path in BPC and see how things pan out. Thanks for the pointers guys! -- //Sorin -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
-Original Message- From: Les Mikesell [mailto:lesmikes...@gmail.com] Sent: den 16 oktober 2014 18:09 To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] Using NFS to increase backup speed I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync over ssh. That sounds pretty good. But unless you have a lot of new files created daily, the bottleneck is usually disk speed, especially merging a lot of small changes into a big existing file. I did some more tests with the suggested ciphers: [root@cyndane ~]# yes | pv | ssh titan -c aes128-ctr cat /dev/null 0:00:10 [14.2MiB/s] [root@cyndane ~]# yes | pv | ssh titan -c aes128-cbc cat /dev/null 0:00:28 [ 23MiB/s] [root@cyndane ~]# yes | pv | ssh titan -c arcfour cat /dev/null 0:00:20 [30.9MiB/s] It would seem arcfour would be the better choice for my setup. Thanks again for the pointers all! -- //Sorin -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
-Original Message- From: Les Mikesell [mailto:lesmikes...@gmail.com] Sent: den 16 oktober 2014 18:09 To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] Using NFS to increase backup speed I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync over ssh. That sounds pretty good. But unless you have a lot of new files created daily, the bottleneck is usually disk speed, especially merging a lot of small changes into a big existing file. Not too many new files daily, the reading is done from a three- or four-disk raid0-array, so should be fairly fast I guess. The 35-45 Mbps mentioned above is a pretty rought figure I think. I took it from the netspeed Gnome applet. The numbers I posted just earlier are probably more exact(ish). But you think this speed is what you'd expect from a setup with a gigabit switch, gigabit-NICs on both ends and CPU:s (BPC: single-core Athlon64 3500+, host: Intel core 2 Quad Q8200@2,33 GHx) a few years old then? Running iperf below I notice the network should theoretically be capable of a bit more than that. I understand that ssh adds quite a bit of bottleneck. There is some overhead for encryption. Back when cpus were slow enough for it to matter I used to set blowfish as the preferred cipher. Now you probably want aes-128 where you have hardware support. Would aes-128 be faster than arcfour, roughly speaking? I'll need to give aes-128 a go as well it seems. After reading one of the answers at http://serverfault.com/questions/377598/why-is-my-rsync-so-slow, I kinda' wonder if the NFS-angle would work for use with BackupPC and how I would go about it. Is the tar transfer method the only one supported by BPC if using NFS for example? Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing rsync over ssh. Any thoughts on this? I'd guess there are too many variables to predict but I'd stick to ssh because of the forced validation (--ignore-times) on full runs. In the case that will ultimately matter the most (the third and subsequent full runs) you can avoid most of the server-side reads if you used checksum caching but you still have to perform a full read of the content on the client side. With ssh, that's a local disk access. with nfs you'll copy every file across the network just so rsync can compute its block checksum even where everything already matches. Some of the articles mentioned this, but didn't go too much into depth. Your clarification cleared that up. Thanks! -- //Sorin -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
Hi,
Sorin Srbu wrote on 2014-10-16 12:25:53 + [[BackupPC-users] Using NFS to
increase backup speed]:
I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync
over ssh.
it is a frequent misconception that you *want* to see anything close to the
speed the network is capable of delivering here. The whole point of the rsync
protocol is to cut down network usage by transferring only data the remote end
does not yet have.
In the context of BackupPC, you may be using rsync for other reasons, but that
does not change how it works. Ideally, your full backups will transfer only a
small part of your data set over the network, while at least the client
machine will still need to read all of it from disk. On a fast network,
network speed will *not* be the bottleneck. Consequentially, you will not be
utilising (almost) all of it. The data rate becomes more of a measure for how
efficient the rsync algorithm is, with lower data rate meaning more
efficiency.
If you actually *are* transferring a lot of new data (e.g. initial backup),
you need to keep in mind that BackupPC will need to compress it (presuming you
are using compression).
Whenever this topic comes up, the question I tend to ask is: are you fixing a
real problem or are you merely trying to get figures that *seem* better to
you, because you are misreading them?
[...]
Is the tar transfer method the only one supported by BPC if using NFS for
example?
The transfer method is totally independant of the location of the data. You
can use FTP to backup localhost (or something mounted there), if you feel so
inclined, or, even worse, smb. There is not much point to that, though.
As for rsync, there is a point, because it makes much more exact incremental
backups than tar does. I use
$Conf{RsyncClientCmd} = 'sudo $rsyncPath $argList';
and it works fine, just as expected.
Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing
rsync over ssh.
Les has a good point that this would read *all* data over the network vs. only
a small fraction. Well, that depends on your data set. If almost everything
changes every day, there won't be much difference.
Has anybody on this list maybe set up their systems using NFS and can share
their experience?
I *have* done that in the past, and it just worked. It didn't give me any
issues, but then, I didn't care how long the backups took as long as they were
fast enough, which they were. You'll almost definitely need a no_root_squash
export, though, and you'll need to think about the error case where your target
file system is not mounted (e.g. modified PingCmd).
I doubt it's worth the effort both of setting it up and of maintaining it.
rsync over ssh is a well-supported standard case. It works well over VPNs,
through firewalls and across administrative domains. It would always be my
first choice of backup method. Unless you have a real *need* for speeding
up your backups, I don't see the point in experimenting.
Regards,
Holger
--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
On Fri, Oct 17, 2014 at 5:38 AM, Sorin Srbu sorin.s...@orgfarm.uu.se wrote: That sounds pretty good. But unless you have a lot of new files created daily, the bottleneck is usually disk speed, especially merging a lot of small changes into a big existing file. Not too many new files daily, the reading is done from a three- or four-disk raid0-array, so should be fairly fast I guess. Do you have a problem completing backups in the time available? And if so, with all incrementals or just fulls? Without checksum caching, the server side will have to read and uncompress everything for the full comparison. With it, after the second full that includes a file, only the client side has to do the read. But in either case, the really slow operation is when a large existing file has a lot of small random changes. For that, the server has to uncompress the old file and create a new copy, merging in the changes from the remote so it may involve a lot of disk seeking. The 35-45 Mbps mentioned above is a pretty rought figure I think. I took it from the netspeed Gnome applet. The numbers I posted just earlier are probably more exact(ish). But you think this speed is what you'd expect from a setup with a gigabit switch, gigabit-NICs on both ends and CPU:s (BPC: single-core Athlon64 3500+, host: Intel core 2 Quad Q8200@2,33 GHx) a few years old then? Backuppc rarely loads the network using rsync except on the initial copy. The point of rsync is to only copy changed data. In fact I usually add a --bwlimit to the rsync args to constrain it to be sure it won't bother any other network traffic. There is some overhead for encryption. Back when cpus were slow enough for it to matter I used to set blowfish as the preferred cipher. Now you probably want aes-128 where you have hardware support. Would aes-128 be faster than arcfour, roughly speaking? I'll need to give aes-128 a go as well it seems. I think this depends on the ssh version and the processor type involved as to whether uses hardware support and how much it helps. -- Les Mikesell lesmikes...@gmail.com -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
-Original Message- From: Les Mikesell [mailto:lesmikes...@gmail.com] Sent: den 17 oktober 2014 14:55 To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] Using NFS to increase backup speed Not too many new files daily, the reading is done from a three- or four-disk raid0-array, so should be fairly fast I guess. Do you have a problem completing backups in the time available? And if so, with all incrementals or just fulls? Without checksum caching, the server side will have to read and uncompress everything for the full comparison. With it, after the second full that includes a file, only the client side has to do the read. But in either case, the really slow operation is when a large existing file has a lot of small random changes. For that, the server has to uncompress the old file and create a new copy, merging in the changes from the remote so it may involve a lot of disk seeking. I thought I had, and maybe I still do, with the very first full backup. I was worried the full backup wouldn't complete in the limited time the BPC server in online. For practical reasons (well, because of the hd-space available really), the server used is off during nights in order to save some electricity. The first backup took some 17 hours... Today, I just checked, the first incremental backup took just under three hours, which is completely okay. I'll need to see how the next full backup performs with the speed tweaks I've done so far. -- //Sorin -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
On Fri, Oct 17, 2014 at 8:29 AM, Sorin Srbu sorin.s...@orgfarm.uu.se wrote: I was worried the full backup wouldn't complete in the limited time the BPC server in online. For practical reasons (well, because of the hd-space available really), the server used is off during nights in order to save some electricity. The first backup took some 17 hours... Today, I just checked, the first incremental backup took just under three hours, which is completely okay. I'll need to see how the next full backup performs with the speed tweaks I've done so far. Incrementals will quickly skip over files where the timestamp and length match the copy in the previous full run. They do transfer more each time until the next full sets a new comparision base. So the times you need to be concerned about are the last (usually 6th) incremental in the set, and the 3rd full run after setting the checksum-seed option). These should show what to expect for subsequent times. -- Les Mikesell lesmikes...@gmail.com -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
On 10/17/2014 09:40 AM, Sorin Srbu wrote: What about testing the speed over the ssh tunnel? That may tell you whether ssh is slowing down your transfers or that it's due to rsync. If it is ssh, you could trade encryption strength for speed. Would the below do? [root@cyndane ~]# yes | pv | ssh titan cat /dev/null 1.16GiB 0:01:29 [14.3MiB/s] [root@cyndane ~]# I searched and read som more after posting the question, and I realize it's not that simple fixing this, as I initially thought... I'd tunnel the port your use for iperf (ssh -L 5001:localhost:5001 cyndane) and run the same test as you initially did (iperf -c localhost). I expect that you'll see a value that's larger than the 35-45 Mbps you get for rsync+ssh, meaning that rsync is the bottleneck. And most of the time, that makes sense, because it's often doing other things than transferring data. Best, Koen -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
What about testing the speed over the ssh tunnel? That may tell you whether ssh is slowing down your transfers or that it's due to rsync. If it is ssh, you could trade encryption strength for speed. Best, Koen On October 16, 2014 2:25:53 PM CEST, Sorin Srbu sorin.s...@orgfarm.uu.se wrote: Hi all, I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync over ssh. Running iperf below I notice the network should theoretically be capable of a bit more than that. I understand that ssh adds quite a bit of bottleneck. [root@titan ~]# iperf -c cyndane Client connecting to cyndane, TCP port 5001 TCP window size: 42.5 KByte (default) [ 3] local 192.168.0.8 port 51754 connected with 192.168.0.9 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 398 MBytes 334 Mbits/sec [root@titan ~]# Just to clarify, titan above is the BPC-server and has some local disk space where the backup pool is. Cyndane is the host being backed up. Both have gigabit-NIC's and a ditto switch between them. After reading one of the answers at http://serverfault.com/questions/377598/why-is-my-rsync-so-slow, I kinda' wonder if the NFS-angle would work for use with BackupPC and how I would go about it. Is the tar transfer method the only one supported by BPC if using NFS for example? Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing rsync over ssh. Any thoughts on this? Has anybody on this list maybe set up their systems using NFS and can share their experience? Thanks. -- BW, Sorin --- # Sorin Srbu, Sysadmin # Uppsala University # Dept of Medicinal Chemistry # Div of Org Pharm Chem # Box 574 # SE-75123 Uppsala # Sweden # # Phone: +46 (0)18-4714482 # Visit: BMC, Husargatan 3, D5:512b # Web: http://www.orgfarm.uu.se --- # () ASCII ribbon campaign - Against html E-mail # /\ # # This message was not sent from an iProduct! # # MotD follows: # CentOS: Because servers can be cheap. -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Sent from Kaiten Mail. Please excuse my brevity.-- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
You may wish to prevent ssh from using compression when using a fast link, the overhead probably isn't worth it and may give you a reasonable boost in throughput. Like Koen said, you'll want to benchmark the different scenarios. Regards, Colin On 16/10/14 16:28, Koen Vermeer wrote: What about testing the speed over the ssh tunnel? That may tell you whether ssh is slowing down your transfers or that it's due to rsync. If it is ssh, you could trade encryption strength for speed. Best, Koen On October 16, 2014 2:25:53 PM CEST, Sorin Srbu sorin.s...@orgfarm.uu.se wrote: Hi all, I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync over ssh. Running iperf below I notice the network should theoretically be capable of a bit more than that. I understand that ssh adds quite a bit of bottleneck. [root@titan ~]# iperf -c cyndane Client connecting to cyndane, TCP port 5001 TCP window size: 42.5 KByte (default) [ 3] local192.168.0.8 http://192.168.0.8 port 51754 connected with192.168.0.9 http://192.168.0.9 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 398 MBytes 334 Mbits/sec [root@titan ~]# Just to clarify, titan above is the BPC-server and has some local disk space where the backup pool is. Cyndane is the host being backed up. Both have gigabit-NIC's and a ditto switch between them. After reading one of the answers at http://serverfault.com/questions/377598/why-is-my-rsync-so-slow, I kinda' wonder if the NFS-angle would work for use with BackupPC and how I would go about it. Is the tar transfer method the only one supported by BPC if using NFS for example? Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing rsync over ssh. Any thoughts on this? Has anybody on this list maybe set up their systems using NFS and can share their experience? Thanks. -- Sent from Kaiten Mail. Please excuse my brevity. -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Colin Shorts c.sho...@intrallect.com +44 (0) 131 292 0104 http://www.intrallect.com 137A George Street, Edinburgh, EH2 4JY, Scotland Intrallect is part of the Leading Software Group -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Using NFS to increase backup speed
On Thu, Oct 16, 2014 at 7:25 AM, Sorin Srbu sorin.s...@orgfarm.uu.se wrote: I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync over ssh. That sounds pretty good. But unless you have a lot of new files created daily, the bottleneck is usually disk speed, especially merging a lot of small changes into a big existing file. Running iperf below I notice the network should theoretically be capable of a bit more than that. I understand that ssh adds quite a bit of bottleneck. There is some overhead for encryption. Back when cpus were slow enough for it to matter I used to set blowfish as the preferred cipher. Now you probably want aes-128 where you have hardware support. After reading one of the answers at http://serverfault.com/questions/377598/why-is-my-rsync-so-slow, I kinda' wonder if the NFS-angle would work for use with BackupPC and how I would go about it. Is the tar transfer method the only one supported by BPC if using NFS for example? Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing rsync over ssh. Any thoughts on this? I'd guess there are too many variables to predict but I'd stick to ssh because of the forced validation (--ignore-times) on full runs. In the case that will ultimately matter the most (the third and subsequent full runs) you can avoid most of the server-side reads if you used checksum caching but you still have to perform a full read of the content on the client side. With ssh, that's a local disk access. with nfs you'll copy every file across the network just so rsync can compute its block checksum even where everything already matches. -- Les Mikesell lesmikes...@gmail.com -- Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
