[BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

Hi all,

I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync 
over ssh.

Running iperf below I notice the network should theoretically be capable of a 
bit more than that. I understand that ssh adds quite a bit of bottleneck.

[root@titan ~]# iperf -c cyndane

Client connecting to cyndane, TCP port 5001
TCP window size: 42.5 KByte (default)

[  3] local 192.168.0.8 port 51754 connected with 192.168.0.9 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec   398 MBytes   334 Mbits/sec
[root@titan ~]#

Just to clarify, titan above is the BPC-server and has some local disk space 
where the backup pool is. Cyndane is the host being backed up. Both have 
gigabit-NIC's and a ditto switch between them.

After reading one of the answers at 
<>, I kinda' 
wonder if the NFS-angle would work for use with BackupPC and how I would go 
about it. Is the tar transfer method the only one supported by BPC if using NFS 
for example?
Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing rsync 
over ssh.

Any thoughts on this? 
Has anybody on this list maybe set up their systems using NFS and can share 
their experience?

Thanks.

-- 
BW,
Sorin
---
# Sorin Srbu, Sysadmin
# Uppsala University
# Dept of Medicinal Chemistry
# Div of Org Pharm Chem
# Box 574
# SE-75123 Uppsala
# Sweden
#
# Phone: +46 (0)18-4714482
# Visit: BMC, Husargatan 3, D5:512b
# Web: http://www.orgfarm.uu.se
---
# ()  ASCII ribbon campaign - Against html E-mail 
# /\
#
# This message was not sent from an iProduct!
#
# MotD follows:
# CentOS: Because servers can be cheap.


--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Koen Vermeer]

What about testing the speed over the ssh tunnel? That may tell you whether ssh 
is slowing down your transfers or that it's due to rsync. If it is ssh, you 
could trade encryption strength for speed. 

Best, 
Koen 



On October 16, 2014 2:25:53 PM CEST, Sorin Srbu  
wrote:
>Hi all,
>
>I'm seeing network speeds at about 35-45 Mbps when using BackupPC and
>rsync over ssh.
>
>Running iperf below I notice the network should theoretically be
>capable of a bit more than that. I understand that ssh adds quite a bit
>of bottleneck.
>
>[root@titan ~]# iperf -c cyndane
>
>Client connecting to cyndane, TCP port 5001
>TCP window size: 42.5 KByte (default)
>
>[  3] local 192.168.0.8 port 51754 connected with 192.168.0.9 port 5001
>[ ID] Interval   Transfer Bandwidth
>[  3]  0.0-10.0 sec   398 MBytes   334 Mbits/sec
>[root@titan ~]#
>
>Just to clarify, titan above is the BPC-server and has some local disk
>space where the backup pool is. Cyndane is the host being backed up.
>Both have gigabit-NIC's and a ditto switch between them.
>
>After reading one of the answers at
><>, I
>kinda' wonder if the NFS-angle would work for use with BackupPC and how
>I would go about it. Is the tar transfer method the only one supported
>by BPC if using NFS for example?
>Rsyncing a NFS-mounted remote host supposedly is a lot faster than
>doing rsync over ssh.
>
>Any thoughts on this? 
>Has anybody on this list maybe set up their systems using NFS and can
>share their experience?
>
>Thanks.
>
>-- 
>BW,
>   Sorin
>---
># Sorin Srbu, Sysadmin
># Uppsala University
># Dept of Medicinal Chemistry
># Div of Org Pharm Chem
># Box 574
># SE-75123 Uppsala
># Sweden   
>#
># Phone: +46 (0)18-4714482
># Visit: BMC, Husargatan 3, D5:512b
># Web: http://www.orgfarm.uu.se
>---
># ()  ASCII ribbon campaign - Against html E-mail 
># /\
>#
># This message was not sent from an iProduct!
>#
># MotD follows:
># CentOS: Because servers can be cheap.
>
>
>--
>Comprehensive Server Monitoring with Site24x7.
>Monitor 10 servers for $9/Month.
>Get alerted through email, SMS, voice calls or mobile push
>notifications.
>Take corrective actions from your mobile device.
>http://p.sf.net/sfu/Zoho
>___
>BackupPC-users mailing list
>BackupPC-users@lists.sourceforge.net
>List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
>Wiki:http://backuppc.wiki.sourceforge.net
>Project: http://backuppc.sourceforge.net/

-- 
Sent from Kaiten Mail. Please excuse my brevity.--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Colin Shorts]

You may wish to prevent ssh from using compression when using a fast 
link, the overhead probably isn't worth it and may give you a reasonable 
boost in throughput.


Like Koen said, you'll want to benchmark the different scenarios.

Regards,
Colin

On 16/10/14 16:28, Koen Vermeer wrote:


What about testing the speed over the ssh tunnel? That may tell you 
whether ssh is slowing down your transfers or that it's due to rsync. 
If it is ssh, you could trade encryption strength for speed.


Best,
Koen



On October 16, 2014 2:25:53 PM CEST, Sorin Srbu 
 wrote:


Hi all,

I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync 
over ssh.

Running iperf below I notice the network should theoretically be capable of 
a bit more than that. I understand that ssh adds quite a bit of bottleneck.

[root@titan ~]# iperf -c cyndane


Client connecting to cyndane, TCP port 5001
TCP window size: 42.5 KByte (default)


[  3] local192.168.0.8    port 51754 connected 
with192.168.0.9    port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec   398 MBytes   334 Mbits/sec
[root@titan ~]#

Just to clarify, titan above is the BPC-server and has some local disk 
space where the backup pool is. Cyndane is the host being backed up. Both have 
gigabit-NIC's and a ditto switch between them.

After reading one of the answers at 
<>, I kinda' 
wonder if the NFS-angle would work for use with BackupPC and how I would go about it. Is 
the tar transfer method the only one supported by BPC if using NFS for example?
Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing 
rsync over ssh.

Any thoughts on this?
Has anybody on this list maybe set up their systems using NFS and can share 
their experience?

Thanks.


--
Sent from Kaiten Mail. Please excuse my brevity.


--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


--
Colin Shorts
c.sho...@intrallect.com
+44 (0) 131 292 0104

http://www.intrallect.com
137A George Street, Edinburgh, EH2 4JY, Scotland
Intrallect is part of the Leading Software Group

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Les Mikesell]

On Thu, Oct 16, 2014 at 7:25 AM, Sorin Srbu  wrote:
>
> I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync 
> over ssh.
>

That sounds pretty good.  But unless you have a lot of new files
created daily, the bottleneck is usually disk speed, especially
merging a lot of small changes into a big existing file.

> Running iperf below I notice the network should theoretically be capable of a 
> bit more than that. I understand that ssh adds quite a bit of bottleneck.

There is some overhead for encryption.  Back when cpus were slow
enough for it to matter I used to set blowfish as the preferred
cipher.  Now you probably want aes-128 where you have hardware
support.

>
> After reading one of the answers at 
> <>, I kinda' 
> wonder if the NFS-angle would work for use with BackupPC and how I would go 
> about it. Is the tar transfer method the only one supported by BPC if using 
> NFS for example?
> Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing 
> rsync over ssh.
>
> Any thoughts on this?

I'd guess there are too many variables to predict but I'd stick to ssh
because of the forced validation (--ignore-times) on full runs.  In
the case that will ultimately matter the most (the third and
subsequent full runs)  you can avoid most of the server-side reads if
you used checksum caching but you still have to perform a full read of
the content on the client side.  With ssh, that's a local disk access.
with nfs you'll copy every file across the network just so rsync can
compute its block checksum even where everything already matches.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

> -Original Message-
> From: Koen Vermeer [mailto:k...@vermeer.tv]
> Sent: den 16 oktober 2014 17:29
> To: General list for user discussion, questions and support
> Subject: Re: [BackupPC-users] Using NFS to increase backup speed
> 
> What about testing the speed over the ssh tunnel? That may tell you
> whether ssh is slowing down your transfers or that it's due to rsync. If it is
> ssh, you could trade encryption strength for speed.

Hi and thanks,

Would the below do? 

[root@cyndane ~]# yes | pv | ssh titan "cat > /dev/null"
1.16GiB 0:01:29 [14.3MiB/s]

[root@cyndane ~]#

I searched and read som more after posting the question, and I realize it's not 
that simple fixing this, as I initially thought...
--
//Sorin
--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

> -Original Message-
> From: Colin Shorts [mailto:c.sho...@intrallect.com]
> Sent: den 16 oktober 2014 17:58
> To: backuppc-users@lists.sourceforge.net
> Subject: Re: [BackupPC-users] Using NFS to increase backup speed
> 
> You may wish to prevent ssh from using compression when using a fast link,
> the overhead probably isn't worth it and may give you a reasonable boost in
> throughput.

Disabling compression in sshd_config didn't do much, maybe .1-.2 MiB increase.

However using arcfour did produce a significant increase as seen below.

[root@cyndane ~]# yes | pv | ssh titan -c arcfour "cat > /dev/null"
 389MiB 0:00:23 [20.8MiB/s]

I'll add arcfour to the ssh-arg path in BPC and see how things pan out.

Thanks for the pointers guys!

--
//Sorin

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

> -Original Message-
> From: Les Mikesell [mailto:lesmikes...@gmail.com]
> Sent: den 16 oktober 2014 18:09
> To: General list for user discussion, questions and support
> Subject: Re: [BackupPC-users] Using NFS to increase backup speed
> 
> > I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync 
> > over ssh.
> >
> 
> That sounds pretty good.  But unless you have a lot of new files
> created daily, the bottleneck is usually disk speed, especially
> merging a lot of small changes into a big existing file.

I did some more tests with the suggested ciphers:

[root@cyndane ~]# yes | pv | ssh titan -c aes128-ctr "cat > /dev/null"
0:00:10 [14.2MiB/s]
 
[root@cyndane ~]# yes | pv | ssh titan -c aes128-cbc "cat > /dev/null"
0:00:28 [  23MiB/s]

[root@cyndane ~]# yes | pv | ssh titan -c arcfour "cat > /dev/null"
0:00:20 [30.9MiB/s]

It would seem arcfour would be the better choice for my setup.

Thanks again for the pointers all!

--
//Sorin

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

> -Original Message-
> From: Les Mikesell [mailto:lesmikes...@gmail.com]
> Sent: den 16 oktober 2014 18:09
> To: General list for user discussion, questions and support
> Subject: Re: [BackupPC-users] Using NFS to increase backup speed
> 
> > I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync 
> > over ssh.
> >
> 
> That sounds pretty good.  But unless you have a lot of new files
> created daily, the bottleneck is usually disk speed, especially
> merging a lot of small changes into a big existing file.

Not too many new files daily, the reading is done from a three- or four-disk 
raid0-array, so should be fairly fast I guess.

The 35-45 Mbps mentioned above is a pretty rought figure I think. I took it 
from the netspeed Gnome applet. The numbers I posted just earlier are probably 
more exact(ish).

But you think this speed is what you'd expect from a setup with a gigabit 
switch, gigabit-NICs on both ends and CPU:s (BPC: single-core Athlon64 3500+, 
host: Intel core 2 Quad Q8200@2,33 GHx) a few years old then?


> > Running iperf below I notice the network should theoretically be capable of
> > a bit more than that. I understand that ssh adds quite a bit of bottleneck.
> 
> There is some overhead for encryption.  Back when cpus were slow
> enough for it to matter I used to set blowfish as the preferred
> cipher.  Now you probably want aes-128 where you have hardware
> support.

Would aes-128 be faster than arcfour, roughly speaking?

I'll need to give aes-128 a go as well it seems.


> > After reading one of the answers at
> <<http://serverfault.com/questions/377598/why-is-my-rsync-so-slow>>, I
> kinda' wonder if the NFS-angle would work for use with BackupPC and how I
> would go about it. Is the tar transfer method the only one supported by BPC
> if using NFS for example?
> > Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing
> rsync over ssh.
> >
> > Any thoughts on this?
> 
> I'd guess there are too many variables to predict but I'd stick to ssh
> because of the forced validation (--ignore-times) on full runs.  In
> the case that will ultimately matter the most (the third and
> subsequent full runs)  you can avoid most of the server-side reads if
> you used checksum caching but you still have to perform a full read of
> the content on the client side.  With ssh, that's a local disk access.
> with nfs you'll copy every file across the network just so rsync can
> compute its block checksum even where everything already matches.

Some of the articles mentioned this, but didn't go too much into depth. 
Your clarification cleared that up. Thanks!


--
//Sorin


--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Holger Parplies]

Hi,

Sorin Srbu wrote on 2014-10-16 12:25:53 + [[BackupPC-users] Using NFS to 
increase backup speed]:
> I'm seeing network speeds at about 35-45 Mbps when using BackupPC and rsync
> over ssh.

it is a frequent misconception that you *want* to see anything close to the
speed the network is capable of delivering here. The whole point of the rsync
protocol is to cut down network usage by transferring only data the remote end
does not yet have.

In the context of BackupPC, you may be using rsync for other reasons, but that
does not change how it works. Ideally, your full backups will transfer only a
small part of your data set over the network, while at least the client
machine will still need to read all of it from disk. On a fast network,
network speed will *not* be the bottleneck. Consequentially, you will not be
utilising (almost) all of it. The data rate becomes more of a measure for how
efficient the rsync algorithm is, with lower data rate meaning more
efficiency.

If you actually *are* transferring a lot of new data (e.g. initial backup),
you need to keep in mind that BackupPC will need to compress it (presuming you
are using compression).

Whenever this topic comes up, the question I tend to ask is: are you fixing a
real problem or are you merely trying to get figures that *seem* better to
you, because you are misreading them?

> [...]
> Is the tar transfer method the only one supported by BPC if using NFS for
> example?

The transfer method is totally independant of the location of the data. You
can use FTP to backup localhost (or something mounted there), if you feel so
inclined, or, even worse, smb. There is not much point to that, though.

As for rsync, there is a point, because it makes much more exact incremental
backups than tar does. I use

$Conf{RsyncClientCmd} = 'sudo $rsyncPath $argList';

and it works fine, just as expected.

> Rsyncing a NFS-mounted remote host supposedly is a lot faster than doing
> rsync over ssh.

Les has a good point that this would read *all* data over the network vs. only
a small fraction. Well, that depends on your data set. If almost everything
changes every day, there won't be much difference.

> Has anybody on this list maybe set up their systems using NFS and can share
> their experience?

I *have* done that in the past, and it just worked. It didn't give me any
issues, but then, I didn't care how long the backups took as long as they were
"fast enough", which they were. You'll almost definitely need a no_root_squash
export, though, and you'll need to think about the error case where your target
file system is not mounted (e.g. modified PingCmd).

I doubt it's worth the effort both of setting it up and of maintaining it.
rsync over ssh is a well-supported standard case. It works well over VPNs,
through firewalls and across administrative domains. It would always be my
first choice of backup method. Unless you have a real *need* for speeding
up your backups, I don't see the point in experimenting.

Regards,
Holger

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Les Mikesell]

On Fri, Oct 17, 2014 at 5:38 AM, Sorin Srbu  wrote:
>>
>> That sounds pretty good.  But unless you have a lot of new files
>> created daily, the bottleneck is usually disk speed, especially
>> merging a lot of small changes into a big existing file.
>
> Not too many new files daily, the reading is done from a three- or four-disk 
> raid0-array, so should be fairly fast I guess.

Do you have a problem completing backups in the time available?   And
if so, with all incrementals or just fulls?   Without checksum
caching, the server side will have to read and uncompress everything
for the full comparison.   With it, after the second full that
includes a file, only the client side has to do the read.   But in
either case, the really slow operation is when a large existing file
has a lot of small random changes.  For that, the server has to
uncompress the old file and create a new copy, merging in the changes
from the remote so it may involve a lot of disk seeking.

> The 35-45 Mbps mentioned above is a pretty rought figure I think. I took it 
> from the netspeed Gnome applet. The numbers I posted just earlier are 
> probably more exact(ish).
>
> But you think this speed is what you'd expect from a setup with a gigabit 
> switch, gigabit-NICs on both ends and CPU:s (BPC: single-core Athlon64 3500+, 
> host: Intel core 2 Quad Q8200@2,33 GHx) a few years old then?

Backuppc rarely loads the network using rsync except on the initial
copy.  The point of rsync is to only copy changed data.  In fact I
usually add a --bwlimit to the rsync args to constrain it to be sure
it won't bother any other network traffic.

>> There is some overhead for encryption.  Back when cpus were slow
>> enough for it to matter I used to set blowfish as the preferred
>> cipher.  Now you probably want aes-128 where you have hardware
>> support.
>
> Would aes-128 be faster than arcfour, roughly speaking?
>
> I'll need to give aes-128 a go as well it seems.

I think this depends on the ssh version and the processor type
involved as to whether uses hardware support and how much it helps.

-- 
  Les Mikesell
 lesmikes...@gmail.com

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

> -Original Message-
> From: Les Mikesell [mailto:lesmikes...@gmail.com]
> Sent: den 17 oktober 2014 14:55
> To: General list for user discussion, questions and support
> Subject: Re: [BackupPC-users] Using NFS to increase backup speed
> 
> > Not too many new files daily, the reading is done from a three- or four-disk
> raid0-array, so should be fairly fast I guess.
> 
> Do you have a problem completing backups in the time available?   And
> if so, with all incrementals or just fulls?   Without checksum
> caching, the server side will have to read and uncompress everything
> for the full comparison.   With it, after the second full that
> includes a file, only the client side has to do the read.   But in
> either case, the really slow operation is when a large existing file
> has a lot of small random changes.  For that, the server has to
> uncompress the old file and create a new copy, merging in the changes
> from the remote so it may involve a lot of disk seeking.

I thought I had, and maybe I still do, with the very first full backup. 

I was worried the full backup wouldn't complete in the limited time the BPC 
server in online. 
For practical reasons (well, because of the hd-space available really), the 
server used is off during nights in order to save some electricity.

The first backup took some 17 hours...

Today, I just checked, the first incremental backup took just under three 
hours, which is completely okay. 
I'll need to see how the next full backup performs with the speed tweaks I've 
done so far.

--
//Sorin


--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Les Mikesell]

On Fri, Oct 17, 2014 at 8:29 AM, Sorin Srbu  wrote:
>
> I was worried the full backup wouldn't complete in the limited time the BPC 
> server in online.
> For practical reasons (well, because of the hd-space available really), the 
> server used is off during nights in order to save some electricity.
>
> The first backup took some 17 hours...
>
> Today, I just checked, the first incremental backup took just under three 
> hours, which is completely okay.
> I'll need to see how the next full backup performs with the speed tweaks I've 
> done so far.

Incrementals will quickly skip over files where the timestamp and
length match the copy in the previous full run.  They do transfer more
each time until the next full sets a new comparision base.  So the
times you need to be concerned about are the last (usually 6th)
incremental in the set, and the 3rd full run after setting the
checksum-seed option).  These should show what to expect for
subsequent times.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Koen Vermeer]

On 10/17/2014 09:40 AM, Sorin Srbu wrote:
>> What about testing the speed over the ssh tunnel? That may tell you
>> whether ssh is slowing down your transfers or that it's due to rsync. If it 
>> is
>> ssh, you could trade encryption strength for speed.
> Would the below do?
>
> [root@cyndane ~]# yes | pv | ssh titan "cat > /dev/null"
> 1.16GiB 0:01:29 [14.3MiB/s]
>
> [root@cyndane ~]#
>
> I searched and read som more after posting the question, and I realize it's 
> not that simple fixing this, as I initially thought...

I'd tunnel the port your use for iperf (ssh -L 5001:localhost:5001 
cyndane) and run the same test as you initially did (iperf -c 
localhost). I expect that you'll see a value that's larger than the 
35-45 Mbps you get for rsync+ssh, meaning that rsync is the bottleneck. 
And most of the time, that makes sense, because it's often doing other 
things than transferring data.

Best,
Koen


--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

> -Original Message-
> From: Koen Vermeer [mailto:k...@vermeer.tv]
> Sent: den 17 oktober 2014 21:49
> To: backuppc-users@lists.sourceforge.net
> Subject: Re: [BackupPC-users] Using NFS to increase backup speed
> 
> > [root@cyndane ~]# yes | pv | ssh titan "cat > /dev/null"
> > 1.16GiB 0:01:29 [14.3MiB/s]
> >
> > [root@cyndane ~]#
> >
> > I searched and read som more after posting the question, and I realize it's
> not that simple fixing this, as I initially thought...
> 
> I'd tunnel the port your use for iperf (ssh -L 5001:localhost:5001
> cyndane) and run the same test as you initially did (iperf -c
> localhost). I expect that you'll see a value that's larger than the
> 35-45 Mbps you get for rsync+ssh, meaning that rsync is the bottleneck.
> And most of the time, that makes sense, because it's often doing other
> things than transferring data.

I'll give it a go. Thanks!

--
//Sorin

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using NFS to increase backup speed

[Sorin Srbu]

> -Original Message-
> From: Les Mikesell [mailto:lesmikes...@gmail.com]
> Sent: den 17 oktober 2014 16:57
> To: General list for user discussion, questions and support
> Subject: Re: [BackupPC-users] Using NFS to increase backup speed
> 
> > I was worried the full backup wouldn't complete in the limited time the BPC
> server in online.
> > For practical reasons (well, because of the hd-space available really), the
> server used is off during nights in order to save some electricity.
> >
> > The first backup took some 17 hours...
> >
> > Today, I just checked, the first incremental backup took just under three
> hours, which is completely okay.
> > I'll need to see how the next full backup performs with the speed tweaks
> I've done so far.
> 
> Incrementals will quickly skip over files where the timestamp and
> length match the copy in the previous full run.  They do transfer more
> each time until the next full sets a new comparision base.  So the
> times you need to be concerned about are the last (usually 6th)
> incremental in the set, and the 3rd full run after setting the
> checksum-seed option).  These should show what to expect for
> subsequent times.

Yupp, I noticed.

Will take another three or four days till the 6th backup shows.

So far, so good!

--
//Sorin

--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/