I have a bit of hard data to offer on this subject, as I recently switched a
backup from tar+ssh (over cygwin) to rsyncd.
The backuppc server is on the same physical LAN, and connect to each other
via a 192.168 address. All the cabling and switches support 100 MB full
duplex communications, and the servers have gigabit NICs. The backuppc
server is dumping the data to the /var partition which is on the 2 80Gb
satas in the case, running in a RAID 1 software array, under mdadm on Debian
stable.
These are the current stats, using rsyncd -
Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files
Size/MB
30<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=30>
full 168480
41758.5 5.66 168333 41639.8 310 118.9
35<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=35>
incr 730
191.7 0.19 564 103.6 223 88.2
36<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=36>
incr 785
198.4 0.18 725 123.0 106 75.4
37<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=37>
full 169010
41836.6 5.73 168876 41750.2 276 86.5
38<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=38>
incr 0
0.0 0.00 0 0.0 29 0.0
39<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=39>
incr 0
0.0 0.00 0 0.0 0 0.0
40<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=40>
incr 155
89.4 0.04 42 5.5 169 83.9
41<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=41>
incr 321
124.2 0.05 142 19.8 234 104.4
Backup# Type Filled Level Start Date Duration/mins Age/days Server
Backup Path
30<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=30>
full yes 0 3/16 18:00
122.9 11.2 /var/lib/backuppc/pc/sarah/30
35<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=35>
incr no 1 3/21 18:00
16.9 6.2 /var/lib/backuppc/pc/sarah/35
36<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=36>
incr no 1 3/22 18:00
17.9 5.2 /var/lib/backuppc/pc/sarah/36
37<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=37>
full yes 0 3/23 18:00
121.7 4.2 /var/lib/backuppc/pc/sarah/37
38<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=38>
incr no 1 3/24 18:00
16.5 3.2 /var/lib/backuppc/pc/sarah/38
39<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=39>
incr no 1 3/25 18:00
15.4 2.2 /var/lib/backuppc/pc/sarah/39
40<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=40>
incr no 1 3/26 18:00
33.1 1.2 /var/lib/backuppc/pc/sarah/40
41<http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browse&host=sarah&num=41>
incr no 1 3/27 18:00
38.6 0.2
When it was doing tar, the full backups took far longer, in the neighborhood
of 600 minutes. The incremental backups took around an hour most days. So I
clearly made out better with rsyncd. Just as additional info, the server
being backed up is a file server for a small company. It is backing up the
directory where they store .jpg images of the products they sell. They
organize it by date, so obviously everything in the current day's directory
is new, but previous directories aren't modified most of the time.
I hope that helps.
Peace,
Jim
On 3/27/07, Holger Parplies <[EMAIL PROTECTED]> wrote:
Hi,
Les Mikesell wrote on 27.03.2007 at 01:03:32 [Re: [BackupPC-users] RSync
v. Tar]:
> Jesse Proudman wrote:
> > I've got one customer who's server has taken 3600 minutes to
> > backup. 77 Gigs of Data. 1,972,859 small files. Would tar be
> > better or make this faster? It's directly connected via 100 Mbit to
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > the backup box.
^^^^^^^^^^^^^^
> If the files don't change frequently, tar incremental runs will be much
> faster because they are based only on the target timestamps while rsync
> will load the entire directory at both ends and compare them.
if you ask me, regardless of how your data changes, tar is the way to go,
not rsync, especially for *full* backups. With a direct 100 MBit
connection,
there's not much point in spending (lots of) CPU time for saving bandwidth
- not with 2 million files. rsync is good for low bandwidth connections,
where the link severely limits the transfer and speeding it up makes a
real
difference. In your case, your link speed is in the same order of
magnitude
as your disk I/O performance (considering our favorite topic, the "seek
times"
on the pool file system, the network link may in fact not even be the
limiting
factor - it clearly isn't, as 77 GB would take slightly more than 2 hours
to
transfer over a 100 MBit link, and rsync is not making it go faster than
that ;-).
rsync has additional benefits concerning finding and backing up new (or
moved) files with old timestamps and deleted files on *incremental*
backups,
but keeping the list of 2 million files in memory will probably be a
problem,
as it possibly was with your full (?) backup (how much memory do the
BackupPC
server and the backed up host have?).
(Les: if the files *do* change frequently, there's even less speedup to
get
from using rsync. Only frequent metadata changes without file content
changes would give rsync an advantage - assuming it is faster to figure
out
that the file is identical than to simply send it over the network.)
Regards,
Holger
P.S.: rsync checksum caching *might* make a difference starting from the
third backup, but I read that it's less improvement than one might
expect.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
BackupPC-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
BackupPC-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
