On 6/14/2018 6:04 PM, Holger Parplies wrote:
> Hi,
>
> Bowie Bailey wrote on 2018-06-14 15:38:44 -0400 [Re: [BackupPC-users] Slow
> local backup]:
>> On 6/14/2018 3:27 PM, Michael Stowe wrote:
>>> On 2018-06-14 10:05, Bowie Bailey wrote:
>>>
>>> I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I
>>> am trying to back up a local 318G filesystem. I am using rsyncd
>>> over the loopback connection. It has been running for 17 hours so
>>> far and has backed up less than half of the directory (based on
>>> the size of the backup filesystem). Running top does not show any
>>> excessive cpu or iowait. ???free??? shows no swap usage and 1.5G
>>> available memory.
> if I'm not miscalculating, that is roughly 10GB/hour or 3MB/s. From what I've
> read about BackupPCv3 performance, that wouldn't seem extremely unreasonable,
> especially for a first backup. For V4, I have no idea what the common figures
> are. The C implementation (rsync_bpc) might have a performance benefit. But I
> would expect one core of your CPU to be almost 100% busy, while the others may
> be idle. Depending on which figure you are looking at, that might not seem
> excessive, but it's the most you can get for single threaded compression
> performance (you *are* using compression, right?).
It finally finished after 24 hours. That gives about 13G/hour or about
3.8M/s.
The CPUs were not busy. That's what I was confused about. I would have
expected to see a bottleneck at some point, but nothing seemed to be
busy. The CPUs were all at or below 20% and iowait was close to 0 most
of the time. I'm not sure how I would determine if the loopback was
saturated.
>
>> [...]
>> I use rsyncd rather than rsync to avoid the ssh overhead.
> For a local backup, you can achieve the same by using 'sudo' instead of 'ssh'.
> At least you could with V3. I've forgotten if and how you can use 'sudo' with
> rsync in V4.
>
>> I expected a backup done via the loopback interface to be fast since it
>> doesn't have the normal networking bandwidth limitations.
> Well, yes, but you still have a network stack and probably at least two
> copies
> between kernel and user space. loopback networking does not come for free. A
> quick 'netperf' test gives me a bit less than 9 Gbit/s throughput. That's
> certainly faster than Gbit networking, but only by a factor of 10.
My network is 100M, so that is significantly faster than what I'm used
to. Also, *only* 10 times faster?? :)
> More importantly, the other limitations don't change - disk and compression
> speed, for instance. If your bottleneck is not the network, a faster network
> won't change anything. Keep in mind that a local backup means that client
> and server are the same computer, so a loopback backup might actually be
> *slower* than a remote backup. Your source file system and BackupPC pool are
> on different physical disks, hopefully?
Yes, the OS and BackupPC pool are on one pair of mirrored SATA disks and
the data that I'm backing up is on another pair of disks.
>
>>> Is it normal for the backup to take this long?
>>>
>>> While that's hard to guess without knowing the particulars of your
>>> system, I'm going to go out on a limb and say, no. No it is not.
> I believe it is still important whether it is the first backup or not. The
> recommendation used to be "ignore the timing of the first backup - fix your
> problem only if the second (or third) backup is still too slow". That would
> still seem to apply to BackupPCv4.
True. I haven't done a first backup in quite a while, so I guess I've
forgotten how slow they can be.
>
>>> Is there a better way to back up a local filesystem?
>>>
>>> Personally, I use rsync (not rsyncd) and in the cases where I have
>>> experienced slowness, it was due to poorly chosen rsync parameters (I
>>> note that this would not differ between rsync and rsyncd), a broken
>>> filesystem, or a specific bug in a specific version of rsync.
>> How do you go about setting up rsync so that it does a local copy rather
>> than going through ssh over the network?
> I would try setting $Conf {RsyncSshArgs} = [ '-e', '/usr/bin/sudo' ], but
> you might run into quoting problems. You could then try using a script
> containing something like 'exec /usr/bin/sudo $@'. Or see if using 'ssh'
> makes much of a difference at all ...
That sounds like it might work. I'll set up a test host to backup a
single directory and play around with it.
I can't see how rsync over ssh could possibly be any faster than a
direct connection to rsyncd.
--
Bowie
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
