Re: [BackupPC-users] spot-checking backups and rsync on cygwin
Nigel Kendrick wrote at about 09:21:03 +0100 on Wednesday, August 19, 2009: Morning, Happy to report that first MSSQL 700MB database dump across ADSL took 433 mins to complete and resulted in a 130MB compressed file, but the night's first full backup took 22mins to sync the changes! A few things, then I'll leave you all in peace for a while (maybe!): 1) Spot-checking backups: Any in-built or documented way to generate or view the MD5 hash of a backed-up file so that files in a backup set can be spot-checked against the MD5 of the original? -- if not, would be a nice feature to be able to click a button or link to ask the for hash of a file - I presume this would have to trigger a temporary un-compress and MD5 generation procedure I wrote the following bare-bones routine that may or not be helpful to you. If you are using rsync-based backups then md4sums are stored in a digest at the end of the file. The attached program can be run against either your cpool or your pc heirarchy or against individual files/directories in either the cpool or pc heirarchy. I have used the program to check for corruption in my pool. BackupPC_digestVerify.pl Description: Binary data 2) The FAQ states Rsync running on Cygwin is limited to either 2GB or 4GB file sizes. Does this still hold true (page last modified in 2006) - I am concerned because our database dumps are going to grow! 3) I have a very sturdy Dell PowerEdge 1300 - a dual PIII-450 box - that *could* be a backup server, but am I expecting too much for the horsepower? This box would sit on a remote site and backup around 10 servers via ADSL VPNs Thanks again - have to say that based on feedback so far you are a friendly bunch! Hope I can contribute to the project in some way. Nigel -- -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] spot-checking backups and rsync on cygwin
Morning, Happy to report that first MSSQL 700MB database dump across ADSL took 433 mins to complete and resulted in a 130MB compressed file, but the night's first full backup took 22mins to sync the changes! A few things, then I'll leave you all in peace for a while (maybe!): 1) Spot-checking backups: Any in-built or documented way to generate or view the MD5 hash of a backed-up file so that files in a backup set can be spot-checked against the MD5 of the original? -- if not, would be a nice feature to be able to click a button or link to ask the for hash of a file - I presume this would have to trigger a temporary un-compress and MD5 generation procedure 2) The FAQ states Rsync running on Cygwin is limited to either 2GB or 4GB file sizes. Does this still hold true (page last modified in 2006) - I am concerned because our database dumps are going to grow! 3) I have a very sturdy Dell PowerEdge 1300 - a dual PIII-450 box - that *could* be a backup server, but am I expecting too much for the horsepower? This box would sit on a remote site and backup around 10 servers via ADSL VPNs Thanks again - have to say that based on feedback so far you are a friendly bunch! Hope I can contribute to the project in some way. Nigel -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] spot-checking backups and rsync on cygwin
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nigel Kendrick wrote: 1) Spot-checking backups: Any in-built or documented way to generate or view the MD5 hash of a backed-up file so that files in a backup set can be spot-checked against the MD5 of the original? -- if not, would be a nice feature to be able to click a button or link to ask the for hash of a file - I presume this would have to trigger a temporary un-compress and MD5 generation procedure 2) The FAQ states Rsync running on Cygwin is limited to either 2GB or 4GB file sizes. Does this still hold true (page last modified in 2006) - I am concerned because our database dumps are going to grow! By default, every full backup means the local file is uncompressed so that we can compare the local rsync checksum with the remote file rsync checksum. However, if you want to speed up your full backup, you can add the --checksum-seed=32761 to your rsync command, but if you still want to occasionally check the checksums to make sure your pool'ed file is identical to the remote one then you also need to look at the option RsyncCsumCacheVerifyProb. In my experience, the files that are verified in this way are marked pool in the backup, even though they are not actually re-transferred (I can tell, because they complete too quickly to re-download the file). All the other unchanged files will be marked as same. 3) I have a very sturdy Dell PowerEdge 1300 - a dual PIII-450 box - that *could* be a backup server, but am I expecting too much for the horsepower? This box would sit on a remote site and backup around 10 servers via ADSL VPNs Is this box also doing the VPN encrytion/decryption? How many concurrent backups are you doing? Do you have compression enabled? Basically, with two CPU's you could do two backups at once, or use one for compression + the other for everything else needed in a backup. Personally, I use a single CPU P4 1.8GHz system, limited to two concurrent backups. The main thing I found was that I needed to upgrade my disk array performance, and memory (1.5G) to ensure I didn't end up swapping during backups. For me, this system is fast enough for my backup hosts (approx 30), and available backup periods (some hosts are blackout 7am to 7pm, some are blackout from 7am to 11pm, and some are allowed anytime (couple of local desktops)... Hope that info is useful, but really, all you can do is wait and see if your box handles it, if it doesn't, find the bottleneck, and solve it. Or just get a new box. You can't tell me new toys aren't fun :) Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkqLwkgACgkQGyoxogrTyiXS6gCgrlLOq1u03CW7SPJIhh3iuNFX 5F4An3YTwTNYVDOYzekCRrZEvuC639J8 =74nw -END PGP SIGNATURE- -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] spot-checking backups and rsync on cygwin
Cheers Adam, All our sites use hardware VPNs between Draytek routers so that load's taken care of. Looks like scheduling's the key here so as you say it's a case of try it and see. With reference to the possible cygwin/rsync file size limit, I am looking at a command-line file splitter that would be useful if needed - I found this one and thought I'd share because it seems to have a ton of useful other functions in a single executable: Swiss File Knife Windows+Linux file tree processor, binary grep, tree size list, instant ftp server, line filter, text replace, dupfind, join files, md5 lists, run command on all files, extract strings, detab, patch, tail, hexdump. no installation, ideal for usb stick. http://sourceforge.net/projects/swissfileknife/ Nigel -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] spot-checking backups and rsync on cygwin
Nigel Kendrick wrote: 2) The FAQ states Rsync running on Cygwin is limited to either 2GB or 4GB file sizes. Running cygwin rsync, I was able to back up (very slowly -- see other thread) files 13G and larger, so I don't think this is a concern any more. Does this still hold true (page last modified in 2006) - I am concerned because our database dumps are going to grow! 3) I have a very sturdy Dell PowerEdge 1300 - a dual PIII-450 box - that *could* be a backup server, but am I expecting too much for the horsepower? This box would sit on a remote site and backup around 10 servers via ADSL VPNs If your bandwidth is extremely low, it could manage. If you're doing full database dumps every night, then compression helps a lot BUT then the server probably doesn't have the horsepower. You can build new machines that have nearly 10x the performance of that old Dell for about $150 (including RAM). Spend the money. -- Jim Leonard (trix...@oldskool.org)http://www.oldskool.org/ Help our electronic games project: http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/ A child borne of the home computer wars: http://trixter.wordpress.com/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
