Gandalf-
The server you're trying to back up doesn't seem that large. A similarly
sized server in my environment (210GB, 3M files) does a full backup in
~80-90 minutes, and incrementals run 10-15 minutes. Monitoring suggests
that the server never exceed 1Gbps outbound on the network connection.
This indicates to me that BackupPC is capable of doing the job, so that
leaves elements of your environment that are variables.
Here's a list that comes to mind:
1) target disk performance:
You indicate that your ZFS store does 50-70MB/s. That's pretty slow in
today's world. I get bothered when storage is slower than a single 10K RPM
drive (~100-120MB/sec). I wonder how fast metadata operations are.
bonnie++ benchmarks might indicate an issue here as BackupPC is metadata
intensive, and has to read a lot of metadata to properly place files in the
CPOOL. Compare those results with other storage to gauge how well your
ZFS is performing. I'm not a ZFS expert.
2) rsyncd vs rsync:
When BackupPC uses the 'rsync' method, it uses ssh to start a dedicated
rsync server on the client system with parameters picked by BackupPC
developers.
When you use the 'rsyncd' method, the options on the client side were
picked by you, and may not play well with BackupPC. It would be easy to
test around this by setting up backupPC to use the 'rsync' method instead
(setting up ssh correctly of course) and seeing if you note any
improvement. That will isolate any issues with your rsyncd configs.
A 4x 1Gbps network link will look exactly like a single 1Gbps per network
channel (stream) unless you've got some really nice port aggregation
hardware that can spray data at 4Gbps across those. As such, unless you
have parallel jobs running (multithreaded), I wouldn't expect to see any
product do better than 1Gbps from any single client in your environment.
The BackupPC server, running multiple backup jobs could see a benefit from
the bonded connection, being able to manage 4 1Gpbs streams at the same
time, under optimal conditions, which never happens.
On Wed, Sep 20, 2017 at 8:34 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:
> 2017-09-20 16:14 GMT+02:00 Ray Frush <fr...@rams.colostate.edu>:
> > Question: Just how big is the host you're trying to backup? GB?
> number
> > of files?
>
> From BackupPC web page: 147466.4 MB, 3465344 files.
>
> > What is the network connection between the client and the backup
> > server?
>
> 4x 1GbE bonded on both sides.
>
>
> > I'm curious about what it is about your environment that is making
> > it so hard to back up.
>
> It's the same with ALL servers that i'm trying to backup.
> BPC is about 12 times slower than any other tested solution.
> Obviously, same backup server, same source servers.
>
> > I believe I've mentioned my largest, hairiest server is 770GB with 6.8
> > Million files. Full backups on that system take 8.5 hours to run.
> > Incrementals take 20-30 minutes. I have no illusions that the
> > infrastructure I'm using to back things up is the fastest, but it's fast
> > enough for the job.
>
> The long running backup (about 38 hours, still running) is an incremental.
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Time flies like an arrow, but fruit flies like a banana.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
