Re: [BackupPC-users] Advice on creating duplicate backup server
Stuart Luscombe wrote: > Hi there, > > > > I’ve been struggling with this for a little while now so I thought it > about time I got some help! > > > > We currently have a server running BackupPC v3.1.0 which has a pool of > around 3TB and we’ve got to a stage where a tape backup of of the pool > is taking 1-2 weeks, which isn’t effective at all. The decision was > made to buy a server that is an exact duplicate of our current one and > have it hosted in another building, as a 2 week old backup isn’t ideal > in the event of a disaster. > > > > I’ve got the OS (CentOS) installed on the new server and have installed > BackupPC v3.1.0, but I’m having problems working out how to sync the > pool with the main backup server. I managed to rsync the cpool folder > without any real bother, but the pool folder is the problem, if I try an > rsync it eventually dies with an ‘out of memory’ error (the server has > 8GB), and a cp –a didn’t seem to work either, as the server filled up, > assumedly as it’s not copying the hard links correctly? > > > > So my query here really is am I going the right way about this? If not, > what’s the best method to take so that say once a day the duplicate > server gets updated. > > > > Many Thanks > All of the folders have to be on the same filesystem, and they should all be synced all at once. Otherwise rsync won't know about hardlinks. Also, have you tried 'cp -a --preserve=all'? It should be mostly redundant but may be worth a shot. Best regards, Johan -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Advice on creating duplicate backup server
Hi there, I've been struggling with this for a little while now so I thought it about time I got some help! We currently have a server running BackupPC v3.1.0 which has a pool of around 3TB and we've got to a stage where a tape backup of of the pool is taking 1-2 weeks, which isn't effective at all. The decision was made to buy a server that is an exact duplicate of our current one and have it hosted in another building, as a 2 week old backup isn't ideal in the event of a disaster. I've got the OS (CentOS) installed on the new server and have installed BackupPC v3.1.0, but I'm having problems working out how to sync the pool with the main backup server. I managed to rsync the cpool folder without any real bother, but the pool folder is the problem, if I try an rsync it eventually dies with an 'out of memory' error (the server has 8GB), and a cp -a didn't seem to work either, as the server filled up, assumedly as it's not copying the hard links correctly? So my query here really is am I going the right way about this? If not, what's the best method to take so that say once a day the duplicate server gets updated. Many Thanks -- Stuart Luscombe Systems Administrator Dementia Research Centre 8-11 Queen Square WC1N 3BG London Direct: 08451 555 000 72 3875 Web : http://www.dementia.ion.ucl.ac.uk -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Advice on creating duplicate backup server
Hi, Nils Breunese (Lemonbit) wrote on 2008-12-08 12:23:40 +0100 [Re: [BackupPC-users] Advice on creating duplicate backup server]: > Stuart Luscombe wrote: > > > I?ve got the OS (CentOS) installed on the new server and have > > installed BackupPC v3.1.0, but I?m having problems working out how > > to sync the pool with the main backup server. I can't help you with *keeping* the pools in sync (other than recommending to run the backups from both servers, like Nils said), but I may be able to help you with an initial copy - presuming 'dd' doesn't work, which would be the preferred method. Can you mount either the old pool on the new machine or the new pool on the old machine via NFS? Or even better, put both disk sets in one machine for copying? You would need to shut down BackupPC for the duration of the copy - is that feasible? 3TB means you're facing about 10 hours even with 'dd', fast hardware and no intervening network - anything more complicated will obviously take longer. Your pool size is 3TB - how large is the file system it is on? Is the destination device at least the same size? How many files are there in your pool? > > I managed to rsync the > > cpool folder without any real bother, but the pool folder is the > > problem, Err, 'pool' or 'pc'? ;-) > > and a cp ?a didn?t seem to work > > either, as the server filled up, assumedly as it?s not copying the > > hard links correctly? That is an interesting observation. I was always wondering exactly in which way cp would fail. > > So my query here really is am I going the right way about this? If > > not, what?s the best method to take so that say once a day the > > duplicate server gets updated. Well, Dan, zfs? ;-) Presuming we can get an initial copy done (does anyone have any ideas on how to *verify* a 3TB pool copy?), would migrating the BackupPC servers to an Opensolaris kernel be an option, or is that too "experimental"? > Check the archives for a *lot* of posts on this subject. The general > conclusion is that copying or rsyncing big pools just doesn't work > because of the large number of hardlinks used by BackupPC. Using rsync > 3.x instead of 2.x seems to need a lot less memory, but it just ends > at some point. Because the basic problem for *any general purpose tool* remains: you need a full inode number to file name mapping for *all files* (there are next to no files with only one link in a pool FS), meaning *at least* something like 50 bytes per file, probably significantly more. You do the maths. Apparently, cp simply ignores hardlinks once malloc() starts failing, but I'm just guessing. This doesn't mean it can't be done. It just means *general purpose tools* will start to fail at some point. > A lot of people run into this when they want to migrate > their pool to another machine or bigger hard drive. In that case the > usual advice is to use dd to copy the partition and then grow the > filesystem once it's copied over. The only problem being that this limits you to the same FS with the same parameters (meaning if you've set up an ext3 FS with too high or too low inodes to block ratio, you can't fix it this way). And the fact remains that copying huge amounts of data simply takes time. > Instead of trying to sync the pool, can't you just run a second > BackupPC server that also backs up your machines? If you don't need the current backup history on the redundant server, save yourself the pain of the initial pool copy and just follow this path - presuming network and client load constraints allow you to. One other thing: is your pool size due to the amount of backed up data or due to a long backup history? If you just want to ensure you have a recent version of your data (but not the complete backup history) in the event of a catastrophe, archives (rather than a copy of the complete pool) may be what you're looking for. Regards, Holger -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Advice on creating duplicate backup server
Stuart Luscombe wrote: > > I’ve got the OS (CentOS) installed on the new server and have > installed BackupPC v3.1.0, but I’m having problems working out how > to sync the pool with the main backup server. I managed to rsync the > cpool folder without any real bother, but the pool folder is the > problem, if I try an rsync it eventually dies with an ‘out of > memory’ error (the server has 8GB), and a cp –a didn’t seem to work > either, as the server filled up, assumedly as it’s not copying the > hard links correctly? > > So my query here really is am I going the right way about this? If > not, what’s the best method to take so that say once a day the > duplicate server gets updated. Check the archives for a *lot* of posts on this subject. The general conclusion is that copying or rsyncing big pools just doesn't work because of the large number of hardlinks used by BackupPC. Using rsync 3.x instead of 2.x seems to need a lot less memory, but it just ends at some point. A lot of people run into this when they want to migrate their pool to another machine or bigger hard drive. In that case the usual advice is to use dd to copy the partition and then grow the filesystem once it's copied over. dd'ing your complete pool every day isn't going to work either I guess. Instead of trying to sync the pool, can't you just run a second BackupPC server that also backs up your machines? Nils Breunese. -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Advice on creating duplicate backup server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Holger Parplies wrote: > Hi, > > Nils Breunese (Lemonbit) wrote on 2008-12-08 12:23:40 +0100 [Re: > [BackupPC-users] Advice on creating duplicate backup server]: >> Stuart Luscombe wrote: >> >>> I?ve got the OS (CentOS) installed on the new server and have >>> installed BackupPC v3.1.0, but I?m having problems working out how >>> to sync the pool with the main backup server. > > I can't help you with *keeping* the pools in sync (other than recommending to How about my personal favourite - enbd or nbd I've had success using this to mirror a fileserver with it's "hot standby" partner... Simply setup all your drives as needed in your "master", format, use as needed (as you already have), then setup your drives in your slave (ie, raid1/5/6/etc) but don't format them. Then setup nbd/enbd (very simple, run one command on the slave and one on the master). Now, the tricky part, follow carefully: 1) umount the drive on the master. 2) create a raid1 array on the master with one device being your filesystem you unmounted in (1) and the second device "missing" 3) hot-add the device from nbd (/dev/nbd/0) to your new raid1 array 4) Use mdadm to configure the device in (3) as a write-mostly or write-only if possible. Now you have a real-time mirror on a remote machine. If everything goes pear-shaped, you do something like this: 1) Make sure the master is dead... 2) kill nbd on the slave 3) mount the device you used for nbd as /var/lib/backuppc 4) start backuppc on the slave (PS, this assumes you have some method of sync'ing the other system configs between the two machines (hint - rsync)... You may need to experiment a bit, but perhaps LVM + snapshots might help as well Of course, the simplest method to ensure off-site and up to date backups is to simple run a second independant backuppc server, assuming you have enough time + bandwidth... Hope that helps... Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkk9JTsACgkQGyoxogrTyiVQxQCdG3oRrrHYj4b5WY+TzkBNxDIh x50AoLYsfYeE1qYjdbC81CQAuCR0Tw/a =Gto6 -END PGP SIGNATURE- -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Advice on creating duplicate backup server
Hi, > > Instead of trying to sync the pool, can't you just run a second > > BackupPC server that also backs up your machines? > If you don't need the current backup history on the redundant server, save > yourself the pain of the initial pool copy and just follow this path - > presuming network and client load constraints allow you to. What do you guys think of DRBD as a solution to this? Benefits: - fast synchronization - backup server failover possible - no changes to BackupPC itself or the backup strategy Caveats: - the very same device parameters on both servers: identical file system and size etc. - if not on a cluster file system, only one side gets to read/modify the data Thomas signature.asc Description: This is a digitally signed message part -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Advice on creating duplicate backup server
Stuart Luscombe wrote at about 10:02:04 + on Monday, December 8, 2008: > Hi there, > > > > I've been struggling with this for a little while now so I thought it about > time I got some help! > > > > We currently have a server running BackupPC v3.1.0 which has a pool of > around 3TB and we've got to a stage where a tape backup of of the pool is > taking 1-2 weeks, which isn't effective at all. The decision was made to > buy a server that is an exact duplicate of our current one and have it > hosted in another building, as a 2 week old backup isn't ideal in the event > of a disaster. > > > > I've got the OS (CentOS) installed on the new server and have installed > BackupPC v3.1.0, but I'm having problems working out how to sync the pool > with the main backup server. I managed to rsync the cpool folder without any > real bother, but the pool folder is the problem, if I try an rsync it > eventually dies with an 'out of memory' error (the server has 8GB), and a cp > -a didn't seem to work either, as the server filled up, assumedly as it's > not copying the hard links correctly? > > > > So my query here really is am I going the right way about this? If not, > what's the best method to take so that say once a day the duplicate server > gets updated. > > > > Many Thanks It just hit me that given the known architecture of the pool and cpool directories shouldn't it be possible to come up with a scheme that works better than either rsync (which can choke on too many hard links) and 'dd' (which has no notion of incremental and requires you to resize the filesystem etc.). My thought is as follows: 1. First, recurse through the pc directory to create a list of files/paths and the corresponding pool links. Note that finding the pool links can be done in one of several ways: - Method 1: Create a sorted list of pool files (which should be significantly shorter than the list of all files due to the nature of pooling and therefore require less memory than rsyn) and then look up the links. - Method 2: Calculate the md5sum file path of the file to determine out where it is in the pool. Where necessary, determine among chain duplicates - Method 3: Not possible yet but would be possible if the md5sum file paths were appended to compressed backups. This would add very little to the storage but it would allow you to very easily determine the right link. If so then you could just read the link path from the file. Files with only 1 link (i.e. no hard links) would be tagged for straight copying. 2. Then rsync *just* the pool -- this should be no problem since by definition there are no hard links within the pool itself 3. Finally, run through the list generated in #1 to create the new pc directory by creating the necessary links (and for files with no hard links, just copy/rsync them) The above could also be easily adapted to allow for "incremental" syncing. Specifically, in #1, you would use rsync to just generate a list of *changed* files in the pc directory. In #2, you would continue to use rsync to just sync *changed* pool entries. In #3 you would only act on the shortened incremental sync list generated in #1. The more I think about it, the more I LIKE the idea of appending the md5sums file paths to compressed pool files (Method #3) since this would make the above very fast. (Note if I were implementing this, I would also include the chain number in cases where there are multiple files with the same md5sum path and of course then BackupPC_nightly would have to adjust this any time it changed around the chain numbering). Even without the above, Method #1 would still be much less memory intensive than rsync and Method #2 while potentially a little slow would require very little memory and wouldn't be nearly that bad if you are doing incremental backups. -- Just as any FYI, if anyone wants to implement method #2, here is the routine I use to generate the md5sum file path from a (compressed) file (note that it is based on the analogous uncompressed version in Lib.pm). use BackupPC::Lib; use BackupPC::Attrib; use BackupPC::FileZIO; use constant _128KB => 131072; use constant _1MB => 1048576; # Compute the MD5 digest of a compressed file. This is the compressed # file version of the Lib.pm function File2MD5. # For efficiency we don't use the whole file for big files # - for files <= 256K we use the file size and the whole file. # - for files <= 1M we use the file size, the first 128K and # the last 128K. # - for files > 1M, we use the file size, the first 128K and # the 8th 128K (ie: the 128K up to 1MB). # See the documentation for a discussion of the tradeoffs in # how much data we use and how many collisions we get. # # Returns the MD5 digest (a hex s
[BackupPC-users] scheduling question
I have some clients that take 3 days or better to do a full backup, i would like to do 1 full backup a month, and 1 incr backup every day, and keep 1 week worth of incr and only 1 full backup. What happens when there is still a full backup going on and i schedule and incr everyday? will it wait until the full is done or will it try to do an incr with a full backup still in progress? I would assume that the incr would fail if it doesnt have a full backup to work off of correct? Is there a certain way im suppose to schedule the backups to accomplish what i want to do? Also, im working with a machine with 2gigs or ram 800mhz cpu P3. The machines im backing up have pretty large sql db's 7 to 12 gigs, will this machine be able to handle that doing all the compression with rsync and over ssh? From some of the other posts ive read on this list, you can run into problems if the machine runs out of ram, even though it has a swap partition. Thanks for any input. -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] rsync - fileowner lost during backup or restore
Hi, I try to backup my windows clients with backuppc as well as cygwin/rsync. The backup/restore with rsyncd at windows and rsync on linux (run as root) works well. Only as normal (linux) user it didn't work because of insufficient privileges for chmod and chgrp. With backuppc (run as user backuppc) I have problems. In the XferLOG the uid/gid of the files will be reported. But in the backuppc datastore (/var/lib/backuppc/pc/...) the fileowner is set to backuppc:backuppc. After restore of files to the windows client they will be owned by the rsyncd process owner (SYSTEM:SYSTEM). I use the following flags for backup: --super --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --one-file-system --bwlimit=2840 and the following for restore: --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --relative --ignore-times --recursive Did backuppc store the fileowner in the filesystem or in the attrib file? Did backuppc need to be run as root to run chmod/chgrp succesfull? Thanks in advance Matthias -- Don't panic -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] I broke perfection ...
If you're being prompted for a password, then the public key is being ignored for some reason. This could be due to permissions. The .ssh directory should have 700 permissions (chmod 700 .ssh). Some distros are picky about the permissions of the individual files in .ssh. Just to be safe, give every file in .ssh 600 permissions (chmod 600 .ssh/*) -Rob Kenneth L. Owen wrote: > Mike, > > I have answered yes to the prompt to accept the key and immediately get > asked for a password. I have tried entering the password thinking that it > might be 'just this once', but when I rerun the command, I am prompted for > the password again. When I set this up before, this test ran without a > hitch, and I simply got a response of 'root' to the whoami command. -- ken > > -Original Message- > From: Les Mikesell [mailto:[EMAIL PROTECTED] > Sent: Thursday, December 04, 2008 3:38 PM > To: General list for user discussion,questions and support > Subject: Re: [BackupPC-users] I broke perfection ... > > Kenneth L. Owen wrote: >> The problem is that when all is working properly, and I run the command >> -bash-3.2$ ssh -l root Winserver whoami >> (bash-3.2$ prompt since user 'backuppc' was setup on Fedora 8 'no-logon') >> WinServer should not ask for a password or pass-phrase, but only respond >> 'root' > > Did you ever answer 'yes' at the point where your earlier message showed > a ^C instead? You do have to answer yes to that once (and only once). > Then if it goes on to ask for a password your keys aren't set up right. > >> The process is not generating password/phrase-less keys! > > This is determined by how you respond during the keygen step for the > backuppc user. If you just hit enter at the prompt for a passphrase you > won't have one. After that you just have to get the pub part of the key > into the /root/.ssh/authorized_keys2 file on the target with permissions > set correctly. > >> It did on the original setup back in October before the crash. I am >> wondering if some file or program update issued in the last month or so > may >> be affecting the way the system is working now. > > An update would affect everyone. It hasn't. > The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the addressee, any disclosure, reproduction, copying, distribution, or other dissemination or use of this transmission in error please notify the sender immediately and then delete this e-mail. E-mail transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard copy version. -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Block-level rsync-like hashing dd?
Hello! In all of the myriad of "how do I replicate a server", someone one time suggested using a netcat/dd like command with block-level hashing to copy only changed blocks. Since then, I have searched *extensively* for this, and I cannot find it. Does anyone know of such a beast? Also, in the interest of "improved dd", using Clonezilla to clone a backup pool has advantages over DD if your pool is substantially smaller than the size of the disk. However, in cases where it's not, dd would be easier. Tim Massey -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Block-level rsync-like hashing dd?
[EMAIL PROTECTED] wrote: > Hello! > > In all of the myriad of "how do I replicate a server", someone one time > suggested using a netcat/dd like command with block-level hashing to copy > only changed blocks. Since then, I have searched *extensively* for this, > and I cannot find it. > > Does anyone know of such a beast? Someone might have mentioned the possibility of hacking the check for devices out of rsync and then using it with the option to modify in place to directly duplicate a disk or partition. I'm not sure if anyone has actually tried it. If you have intermediate disk space you could dd your image to a local file, rsync that to a remote copy, then dd back to a device (and if it is just for backup purposes you could defer the copy to a device until you needed it). > Also, in the interest of "improved dd", using Clonezilla to clone a backup > pool has advantages over DD if your pool is substantially smaller than the > size of the disk. However, in cases where it's not, dd would be easier. Clonezilla also offers intermediate image file storage but it is compressed and would not rsync updates as well. -- Les Mikesell [EMAIL PROTECTED] -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Block-level rsync-like hashing dd?
On Mon, Dec 08, 2008 at 12:17:11PM -0500, [EMAIL PROTECTED] wrote: > In all of the myriad of "how do I replicate a server", someone one time > suggested using a netcat/dd like command with block-level hashing to copy > only changed blocks. Since then, I have searched *extensively* for this, > and I cannot find it. I'd just rsync the raw devices. It's actually smilar to rsync'ing DVD images, just a bit larger. Never checked whether that's faster - probably depends on your network bandwidth and I/O speeds. HTH, Tino. -- "What we nourish flourishes." - "Was wir nähren erblüht." www.lichtkreis-chemnitz.de www.craniosacralzentrum.de -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Block-level rsync-like hashing dd?
On Mon, Dec 08, 2008 at 06:49:27PM +0100, Tino Schwarze wrote: > > In all of the myriad of "how do I replicate a server", someone one time > > suggested using a netcat/dd like command with block-level hashing to copy > > only changed blocks. Since then, I have searched *extensively* for this, > > and I cannot find it. > > I'd just rsync the raw devices. It's actually smilar to rsync'ing DVD > images, just a bit larger. Never checked whether that's faster - > probably depends on your network bandwidth and I/O speeds. D'oh, sorry! Of course, rsync will sync the device file, not the content of the device... Tino. -- "What we nourish flourishes." - "Was wir nähren erblüht." www.lichtkreis-chemnitz.de www.craniosacralzentrum.de -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] backups failing due to errors under /sys... BackupFilesExclude is not taking effect. What gives?
BackupPC 3.1.0 on CentOS 5.2
Backups of a single host are aborting after lots of errors like
sys/block/hdc/capability: md4 doesn't match: will retry in phase 1; file removed
Remote[1]: rsync: read errors mapping "/sys/block/hdc/capability": No
data available (61)
sys/block/hdc/dev: md4 doesn't match: will retry in phase 1; file removed
Remote[1]: rsync: read errors mapping "/sys/block/hdc/dev": No data
available (61)
[ skipped 4 lines ]
sys/block/hdc/queue/iosched/back_seek_max: md4 doesn't match: will
retry in phase 1; file removed
Remote[1]: rsync: read errors mapping
"/sys/block/hdc/queue/iosched/back_seek_max": No data available (61)
sys/block/hdc/queue/iosched/back_seek_penalty: md4 doesn't match: will
retry in phase
Always referencing files under /sys
My site-wide config.pl includes
$Conf{BackupFilesExclude} = ['/proc', '/sys'];
My error log for this host shows:
Sent exclude: /proc
Why didn't /sys get excluded, as per the configuration? There is no
overriding $Conf{BackupFilesExclude} directive in the per-host
config file.
I've got rsync 3.0.3 (protocol version 30) on both ends.
Any ideas?
Thanks,
Aleksey
--
Aleksey Tsalolikhin
UNIX System Administrator
"I get stuff done!"
http://www.lifesurvives.com/
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] BackupFilesExcludes for Linux
Greetings. Me again.
I upgraded my backup server and configured two hard drives into a single
logical volume with LVM for my TopDir, and successfully configured
BackupPC to backup a share on one of my host machines. Rejoicing was
heard throughout the LAN.
Now, I am faced with the necessity of backing up my /etc and some of my
user's home directory on that machine.
I want to exclude several directories under my user's home directory,
but the information I've seen is all about configuring BackupPC for a
Windows host. I am assuming I need to change this:
$Conf{BackupFilesOnly} = {
'*' => [
'images',
'Photos'
]
};
$Conf{RsyncShareName} = [
'/mnt/share'
];
$Conf{ClientNameAlias} = '192.168.1.106';
To this:
$Conf{BackupFilesOnly} = {
'/mnt/share' => [
'images',
'Photos'
]
'/' => [
'etc'
]
'/home' => [
'madams'
]
};
$Conf{BackupFilesExclude} = {
'/home/madams' => ['/torrent', '/Download'],
};
$Conf{RsyncShareName} = [
'/mnt/share', '/home/madams', '/etc'
];
$Conf{ClientNameAlias} = '192.168.1.106';
Is this right, or do I misunderstand this configuration?
Thanks.
--
Mark E. Adams
Random Musing:
Save the whales. Collect the whole set.
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes for Linux
I think I spotted a mistake in my own speculative config. See below.
Mark Adams wrote:
> Greetings. Me again.
>
> I upgraded my backup server and configured two hard drives into a single
> logical volume with LVM for my TopDir, and successfully configured
> BackupPC to backup a share on one of my host machines. Rejoicing was
> heard throughout the LAN.
>
> Now, I am faced with the necessity of backing up my /etc and some of my
> user's home directory on that machine.
>
> I want to exclude several directories under my user's home directory,
> but the information I've seen is all about configuring BackupPC for a
> Windows host. I am assuming I need to change this:
>
>
> $Conf{BackupFilesOnly} = {
> '*' => [
> 'images',
> 'Photos'
> ]
> };
> $Conf{RsyncShareName} = [
> '/mnt/share'
> ];
> $Conf{ClientNameAlias} = '192.168.1.106';
>
>
>
> To this:
>
>
> $Conf{BackupFilesOnly} = {
> '/mnt/share' => [
> 'images',
> 'Photos'
> ]
> '/' => [
>'etc'
> ]
> '/home' => [
> 'madams'
> ]
> };
> $Conf{BackupFilesExclude} = {
> '/home/madams' => ['/torrent', '/Download'],
> };
> $Conf{RsyncShareName} = [
> '/mnt/share', '/home/madams', '/etc'
> ];
> $Conf{ClientNameAlias} = '192.168.1.106';
>
>
> Is this right, or do I misunderstand this configuration?
>
> Thanks.
>
>
I believe the $Conf{RsyncShareName} statement should involve '/' rather
than '/etc'. The directory I want to backup is /etc which is a
subdirectory of /. I also believe the /home/madams directory should be
/home. Am I right?
--
Mark E. Adams
Random Musing:
And do you think (fop that I am) that I could be the Scarlet Pumpernickel?
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backups failing due to errors under /sys... BackupFilesExclude is not taking effect. What gives?
Aleksey Tsalolikhin wrote:
> BackupPC 3.1.0 on CentOS 5.2
>
> Backups of a single host are aborting after lots of errors like
>
> sys/block/hdc/capability: md4 doesn't match: will retry in phase 1; file
> removed Remote[1]: rsync: read errors mapping "/sys/block/hdc/capability":
> No data available (61)
> sys/block/hdc/dev: md4 doesn't match: will retry in phase 1; file removed
> Remote[1]: rsync: read errors mapping "/sys/block/hdc/dev": No data
> available (61)
> [ skipped 4 lines ]
> sys/block/hdc/queue/iosched/back_seek_max: md4 doesn't match: will
> retry in phase 1; file removed
> Remote[1]: rsync: read errors mapping
> "/sys/block/hdc/queue/iosched/back_seek_max": No data available (61)
> sys/block/hdc/queue/iosched/back_seek_penalty: md4 doesn't match: will
> retry in phase
>
> Always referencing files under /sys
>
> My site-wide config.pl includes
>
> $Conf{BackupFilesExclude} = ['/proc', '/sys'];
>
> My error log for this host shows:
>
> Sent exclude: /proc
>
> Why didn't /sys get excluded, as per the configuration? There is no
> overriding $Conf{BackupFilesExclude} directive in the per-host
> config file.
>
Try
$Conf{BackupFilesExclude} = {
'*' => [
'/proc',
'/sys'
]
};
> I've got rsync 3.0.3 (protocol version 30) on both ends.
>
> Any ideas?
>
> Thanks,
> Aleksey
>
>
--
Don't panic
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Solution to keeping files selectively by "age" of backups
On Sun, Nov 23, 2008 at 05:00:01AM -0500, Jeffrey J. Kosowsky wrote: > Several people have recently asked about whether it is possible to > have some files backed up only occasionally or to keep some files for > only a limited amount of time. > > One example is perhaps you want to always have the most recent couple > of backups of your temp files so that if something crashes you can do > a full restore but you don't want to save these files (nearly) > forever. So, for example maybe you want the ability to have some > files only saved for the last 'x' backups. > > Well, this can be done with some new logic that I added to my > BackupPC_deleteFile program. Does anybody have a URL for dowloading the latest copy of this? -- -- rouilj John Rouillard System Administrator Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111 -- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes for Linux
Mark Adams wrote:
>
> I believe the $Conf{RsyncShareName} statement should involve '/' rather
> than '/etc'. The directory I want to backup is /etc which is a
> subdirectory of /. I also believe the /home/madams directory should be
> /home. Am I right?
>
$Conf{RsyncShareName} should contain the directories which should backuped.
If it contain '/' than all subdirectories, including /etc will be backuped,
if it is not on another filesystem and you use --one-file-system.
If it contain '/etc' than only /etc and all subdirectories of /etc will be
backuped.
br
Matthias
--
Don't panic
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes for Linux
Hi,
Mark Adams wrote on 2008-12-08 14:37:54 -0700 [Re: [BackupPC-users]
BackupFilesExcludes for Linux]:
> I think I spotted a mistake in my own speculative config. See below.
right, and there's a second one along the same line ...
> [...]
> > $Conf{BackupFilesOnly} = {
> > '/mnt/share' => [
> > 'images',
> > 'Photos'
> > ]
> > '/' => [
> >'etc'
> > ]
> > '/home' => [
> > 'madams'
> > ]
> > };
> > $Conf{BackupFilesExclude} = {
> > '/home/madams' => ['/torrent', '/Download'],
This should be
'/home' => [ '/maddams/torrent', '/maddams/Download' ],
> > };
> > $Conf{RsyncShareName} = [
> > '/mnt/share', '/home', '/'
[edited to match your correction]
> > ];
> > $Conf{ClientNameAlias} = '192.168.1.106';
As Matthias probably meant to point out, you can achieve the same with
$Conf {RsyncShareName} = [ '/etc', '/home/maddams', '/mnt/share/images',
'/mnt/share/Photos' ];
$Conf {BackupFilesExclude} = { '/home/maddams' => [ '/torrent',
'/Download'] };
or even
$Conf {RsyncShareName} = [ '/' ];
$Conf {BackupFilesOnly} = {
'/' => [ '/etc', '/home/maddams', '/mnt/share/images',
'/mnt/share/Photos' ],
};
$Conf {BackupFilesExclude} = {
'/' => [ '/home/maddams/torrent', '/home/maddams/Download' ],
};
This is, of course, just as speculative as your config, but I believe it
*should* work that way :-).
The important part is that the hash keys in BackupFilesOnly and
BackupFilesExclude need to match your {RsyncShareName}s exactly, else they
will simply be ignored (there is no magic that determines that '/home/maddams'
lies within the '/home' share). Aside from that, *with rsync and tar* (not
rsyncd and smb) you are rather free to split your paths into share names and
BackupFilesOnly entries any way you like. Less shares probably means less
overhead and a backup definition that is easier to understand. If you use
'--one-file-system', you will need extra shares for anything on different file
systems (probably '/home' and '/mnt/share' in your case).
Regards,
Holger
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Advice on creating duplicate backup server
you could mess around with LVM snapshots. I hear that you can make an LVM snapshot and rsync that over, then restore it to the backup LVM. I have not tried this but have seen examples around the net. have you tried rsync3? it works for me. I dont quite have 3TB so I cant really advise you on that size, Im not sure where the line is on file count that rsync3 cant handle. ZFS would be ideal for this but you have to make the leap to a solaris/opensolaris kernel. ZFS Fuse is completely non-functional for backuppc as it will crash as soon as you starting hitting the filesystem and the delayed write caching kicks in. ZFS on freebsd is not mature enough and tends to crash out with heavy IO. with zfs it works something like this: http://blogs.sun.com/clive/resource/zfs_repl.ksh you can send a full zfs snapshot like zfs send /pool/[EMAIL PROTECTED] | ssh remotehost zfs recv -v /remotepool/remotefs or send an incremental afterwards with zfs send -i /pool/[EMAIL PROTECTED] | ssh remotehost zfs recv -F -v /remotepool/remotefs feel free to compress the ssh stream with -C if you like, but I would first check your bandwidth usage and see if you are using the whole thing. If not, then the compression will slow you down. The real downside here is the switch to solaris if you are a linux person. You can also try nexenta which is the opensolaris kernel on a debian/ubuntu userland complete with apt. You also get filesystem level compression with ZFS so you dont need to compress your pool. This should make recovering files outside of backuppc a little more convenient . how is a tape taking 1-2weeks? 1 week = 5.2KB/s. If you are that IO constrained, nothing is going to work right for you. How full is your pool? you could also consider not keeping a copy of the pool remotely be rather pullting a tar backup off the backuppc system on some schedule and sending that to the remote machine for storage. The problem with using NBD or anything like that and using 'dd' is that there is no resume support and with 3TB you are likely to get errors every now and then. even with a full T1 you are stuck at at least 6hour with theoritical numbers and are probably looking at %50 more than that. as far as some other scheme to syncing up the pools, hardlinks will get you. You could use find to traverse the entire pool and take some info down on each file such as name, size, type etc etc and then use some fancy perl to sort this out into managable groups and then use rsync on individual files. On Mon, Dec 8, 2008 at 7:37 AM, Jeffrey J. Kosowsky <[EMAIL PROTECTED]>wrote: > Stuart Luscombe wrote at about 10:02:04 + on Monday, December 8, 2008: > > Hi there, > > > > > > > > I've been struggling with this for a little while now so I thought it > about > > time I got some help! > > > > > > > > We currently have a server running BackupPC v3.1.0 which has a pool of > > around 3TB and we've got to a stage where a tape backup of of the pool > is > > taking 1-2 weeks, which isn't effective at all. The decision was made > to > > buy a server that is an exact duplicate of our current one and have it > > hosted in another building, as a 2 week old backup isn't ideal in the > event > > of a disaster. > > > > > > > > I've got the OS (CentOS) installed on the new server and have installed > > BackupPC v3.1.0, but I'm having problems working out how to sync the > pool > > with the main backup server. I managed to rsync the cpool folder without > any > > real bother, but the pool folder is the problem, if I try an rsync it > > eventually dies with an 'out of memory' error (the server has 8GB), and > a cp > > -a didn't seem to work either, as the server filled up, assumedly as > it's > > not copying the hard links correctly? > > > > > > > > So my query here really is am I going the right way about this? If not, > > what's the best method to take so that say once a day the duplicate > server > > gets updated. > > > > > > > > Many Thanks > > It just hit me that given the known architecture of the pool and cpool > directories shouldn't it be possible to come up with a scheme that > works better than either rsync (which can choke on too many hard > links) and 'dd' (which has no notion of incremental and requires you > to resize the filesystem etc.). > > My thought is as follows: > 1. First, recurse through the pc directory to create a list of > files/paths and the corresponding pool links. > Note that finding the pool links can be done in one of several > ways: > - Method 1: Create a sorted list of pool files (which should be > significantly shorter than the list of all files due to the > nature of pooling and therefore require less memory than rsyn) > and then look up the links. > - Method 2: Calculate the md5sum file path of the file to determine > out where it is in the pool. Where necessary, determine among > chain duplicates > - Method 3: Not possible ye
Re: [BackupPC-users] BackupFilesExcludes for Linux
Holger Parplies wrote at about 00:24:56 +0100 on Tuesday, December 9, 2008:
> Hi,
>
> Mark Adams wrote on 2008-12-08 14:37:54 -0700 [Re: [BackupPC-users]
> BackupFilesExcludes for Linux]:
> > I think I spotted a mistake in my own speculative config. See below.
>
> right, and there's a second one along the same line ...
>
> > [...]
> > > $Conf{BackupFilesOnly} = {
> > > '/mnt/share' => [
> > > 'images',
> > > 'Photos'
> > > ]
> > > '/' => [
> > >'etc'
> > > ]
> > > '/home' => [
> > > 'madams'
> > > ]
> > > };
> > > $Conf{BackupFilesExclude} = {
> > > '/home/madams' => ['/torrent', '/Download'],
>
> This should be
>
> '/home' => [ '/maddams/torrent', '/maddams/Download' ],
>
> > > };
> > > $Conf{RsyncShareName} = [
> > > '/mnt/share', '/home', '/'
> [edited to match your correction]
> > > ];
> > > $Conf{ClientNameAlias} = '192.168.1.106';
>
> As Matthias probably meant to point out, you can achieve the same with
>
> $Conf {RsyncShareName} = [ '/etc', '/home/maddams', '/mnt/share/images',
> '/mnt/share/Photos' ];
> $Conf {BackupFilesExclude} = { '/home/maddams' => [ '/torrent',
> '/Download'] };
>
> or even
>
> $Conf {RsyncShareName} = [ '/' ];
> $Conf {BackupFilesOnly} = {
> '/' => [ '/etc', '/home/maddams', '/mnt/share/images',
> '/mnt/share/Photos' ],
> };
> $Conf {BackupFilesExclude} = {
> '/' => [ '/home/maddams/torrent', '/home/maddams/Download' ],
> };
>
> This is, of course, just as speculative as your config, but I believe it
> *should* work that way :-).
>
> The important part is that the hash keys in BackupFilesOnly and
> BackupFilesExclude need to match your {RsyncShareName}s exactly, else they
> will simply be ignored (there is no magic that determines that
> '/home/maddams'
> lies within the '/home' share). Aside from that, *with rsync and tar* (not
> rsyncd and smb) you are rather free to split your paths into share names and
> BackupFilesOnly entries any way you like. Less shares probably means less
> overhead and a backup definition that is easier to understand. If you use
> '--one-file-system', you will need extra shares for anything on different
> file
> systems (probably '/home' and '/mnt/share' in your case).
>
Also (I believe that) using multiple share names on the same
filesystem will not preserve hard links that are split across the
share names. So, if you are backing up your linux partitions and want
to preserve hard links then you should probably keep all the key root
partitions on the same share (please correct me if I am wrong but I
don't see how rsync would otherwise know about the hard links since
each share is done independently).
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes for Linux
Jeffrey J. Kosowsky wrote:
> Holger Parplies wrote at about 00:24:56 +0100 on Tuesday, December 9, 2008:
> > Hi,
> >
> > Mark Adams wrote on 2008-12-08 14:37:54 -0700 [Re: [BackupPC-users]
> BackupFilesExcludes for Linux]:
> > > I think I spotted a mistake in my own speculative config. See below.
> >
> > right, and there's a second one along the same line ...
> >
> > > [...]
> > > > $Conf{BackupFilesOnly} = {
> > > > '/mnt/share' => [
> > > > 'images',
> > > > 'Photos'
> > > > ]
> > > > '/' => [
> > > >'etc'
> > > > ]
> > > > '/home' => [
> > > > 'madams'
> > > > ]
> > > > };
> > > > $Conf{BackupFilesExclude} = {
> > > > '/home/madams' => ['/torrent', '/Download'],
> >
> > This should be
> >
> > '/home' => [ '/maddams/torrent', '/maddams/Download' ],
> >
> > > > };
> > > > $Conf{RsyncShareName} = [
> > > > '/mnt/share', '/home', '/'
> > [edited to match your correction]
> > > > ];
> > > > $Conf{ClientNameAlias} = '192.168.1.106';
> >
> > As Matthias probably meant to point out, you can achieve the same with
> >
> > $Conf {RsyncShareName} = [ '/etc', '/home/maddams',
> '/mnt/share/images',
> > '/mnt/share/Photos' ];
> > $Conf {BackupFilesExclude} = { '/home/maddams' => [ '/torrent',
> >'/Download'] };
> >
> > or even
> >
> > $Conf {RsyncShareName} = [ '/' ];
> > $Conf {BackupFilesOnly} = {
> >'/' => [ '/etc', '/home/maddams', '/mnt/share/images',
> > '/mnt/share/Photos' ],
> > };
> > $Conf {BackupFilesExclude} = {
> >'/' => [ '/home/maddams/torrent', '/home/maddams/Download' ],
> > };
> >
> > This is, of course, just as speculative as your config, but I believe it
> > *should* work that way :-).
> >
> > The important part is that the hash keys in BackupFilesOnly and
> > BackupFilesExclude need to match your {RsyncShareName}s exactly, else they
> > will simply be ignored (there is no magic that determines that
> '/home/maddams'
> > lies within the '/home' share). Aside from that, *with rsync and tar* (not
> > rsyncd and smb) you are rather free to split your paths into share names
> and
> > BackupFilesOnly entries any way you like. Less shares probably means less
> > overhead and a backup definition that is easier to understand. If you use
> > '--one-file-system', you will need extra shares for anything on different
> file
> > systems (probably '/home' and '/mnt/share' in your case).
> >
>
> Also (I believe that) using multiple share names on the same
> filesystem will not preserve hard links that are split across the
> share names. So, if you are backing up your linux partitions and want
> to preserve hard links then you should probably keep all the key root
> partitions on the same share (please correct me if I am wrong but I
> don't see how rsync would otherwise know about the hard links since
> each share is done independently).
>
>
Thanks for the tip Jeff. I think I understand what you're saying here
about preserving hard links, but I'm still just learning a thing or two
about rsync. Just for the sake of simplicity, I adapted Holger's design
as below. I stated an incremental backup of that config about 30
minutes ago and things look good at this point. I believe it's a winner.
I've got two more machines to configure for backup and then I just ride
it for another six months until I have to upgrade again. Thanks to all
you folks for the help. I really do appreciate it.
# cat spike.pl
$Conf {RsyncShareName} = [ '/' ];
$Conf {BackupFilesOnly} = {
'/' => [ '/etc', '/home/madams', '/mnt/share/images',
'/mnt/share/Photos' ],
};
$Conf {BackupFilesExclude} = {
'/' => [ '/home/madams/torrent', '/home/madams/Download' ],
};
--
Mark E. Adams
Random Musing:
Did you know that for the price of a 280-Z you can buy two Z-80's?
-- P.J. Plauger
--
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Advice on creating duplicate backup server
Hi, Jeffrey J. Kosowsky wrote on 2008-12-08 09:37:16 -0500 [Re: [BackupPC-users] Advice on creating duplicate backup server]: > > It just hit me that given the known architecture of the pool and cpool > directories shouldn't it be possible to come up with a scheme that > works better than either rsync (which can choke on too many hard > links) and 'dd' (which has no notion of incremental and requires you > to resize the filesystem etc.). yes, that hit someone on the list several years ago (I don't remember the name, sorry). I implemented the idea he sketched (well, more or less, there's some work left to make it really useful). > My thought is as follows: > 1. First, recurse through the pc directory to create a list of >files/paths and the corresponding pool links. >Note that finding the pool links can be done in one of several >ways: >- Method 1: Create a sorted list of pool files (which should be > significantly shorter than the list of all files due to the >nature of pooling and therefore require less memory than rsyn) >and then look up the links. Wrong. You need one entry per inode that points to an arbitrary path (the first one you copy). Every file(*) is in the pool, meaning a list of all pool files is exactly what you need. A different way to look at it: every file with a link count > 1 is a pooled file, and it's these files that cause rsync&co problems, not single link files. (Well, yes, rsync pre-3 needed a complete list of all files.) (*) Files that are not in the pool: 1.) 0-byte files. They take up no file system blocks, so pooling them saves only inodes. Not pooling them makes things simpler. 2.) log files (they get appended to; that would make pooling somewhat difficult; besides, what chance is there of a pool hit?), backups files (including backups.old) attrib files are pooled, contrary to popular belief, and that makes sense, because they are often identical with the same attrib file from the previous backup(s). The algorithm I implemented is somewhat similar: 1.) Walk pool/, cpool/ and pc/, printing information on the files and directories to a file (which will be quite large; by default I put it on the destination pool FS, because there should be large amounts of space there). 2.) Sort the file with the 'sort' command. The lines in the file are designed such that they will be sorted into a meaningful order: - directories first, so I can create them and subsequently not worry about whether the place I want to copy/link a file to already exists or not - files next, sorted by inode number, with the (c)pool file preceeding its pc/ links The consequence is that I get all references to one inode on adjacent lines. The first time, I copy the file. For the repetitions, I link to the first copy. All I need to keep in memory is something like one line from the file list, one "previous inode number", one "file name of previous inode". 'sort' handles huge files quite nicely, but it seems to create large (amounts of) files under /tmp, possibly under $TMPDIR if you set that (not sure). You need to make sure you've got the space, but if you're copying a multi-GB/TB pool, you probably have. My guess is that the necessary amount of space roughly equals the size of the file I'm sorting. 3.) Walk the sorted file, line by line, creating directories and copying files (with File::Copy::cp, but I plan to change that to PoolWrite, so I can add (part of) one pool to an existing second pool, or something that communicates over TCP/IP, so I can copy to a different machine) and linking files (with Perl function link()). In theory, a pool could also be compressed or uncompressed on the fly (uncompressed for copying to zfs, for instance). Once again, because people seem to be determined to miss the point: it's *not* processing by sorted inode numbers in order to save disk seeks that is the point, it's the fact that the 'link' system call takes two paths link $source_path, $dest_path; # to use Perl notation while the 'stat' system call gives you only an inode number. To link a filename to a previously copied inode, you need to know the name you copied it to. A general purpose tool can't know when it will need the information, so it needs to keep information on all inodes with link count > 1 it has encountered. You can keep a mapping of inode_number->file_name in memory for a few thousand files, but not for hundreds of millions. By sorting the list by inode number, I can be sure that I'll never need the info for one inode again once I've reached the next inode, so I only have to keep info for one file in memory, regardless of how many I'm copying. The difficult part is now the 'sort', but, again, the 'sort' command is good at handling huge files - probably without limit to the file size. So, what's the probl
Re: [BackupPC-users] Advice on creating duplicate backup server
Holger Parplies wrote at about 04:10:17 +0100 on Tuesday, December 9, 2008: > Hi, > > Jeffrey J. Kosowsky wrote on 2008-12-08 09:37:16 -0500 [Re: [BackupPC-users] > Advice on creating duplicate backup server]: > > > > It just hit me that given the known architecture of the pool and cpool > > directories shouldn't it be possible to come up with a scheme that > > works better than either rsync (which can choke on too many hard > > links) and 'dd' (which has no notion of incremental and requires you > > to resize the filesystem etc.). > > yes, that hit someone on the list several years ago (I don't remember the > name, sorry). I implemented the idea he sketched (well, more or less, there's > some work left to make it really useful). > > > My thought is as follows: > > 1. First, recurse through the pc directory to create a list of > >files/paths and the corresponding pool links. > >Note that finding the pool links can be done in one of several > >ways: > >- Method 1: Create a sorted list of pool files (which should be > > significantly shorter than the list of all files due to the > > nature of pooling and therefore require less memory than rsyn) > > and then look up the links. > > Wrong. You need one entry per inode that points to an arbitrary path (the > first one you copy). Every file(*) is in the pool, meaning a list of all pool > files is exactly what you need. A different way to look at it: every file > with > a link count > 1 is a pooled file, and it's these files that cause rsync&co > problems, not single link files. (Well, yes, rsync pre-3 needed a complete > list of all files.) OK. I had assumed (wrongly) that rsync needed to keep track of each file that is hard-linked, not just one copy. Still, there are some savings by knowing that you can find your one copy in the pool and you don't have to look at all through the pc tree. > > (*) Files that are not in the pool: > 1.) 0-byte files. They take up no file system blocks, so pooling them > saves only inodes. Not pooling them makes things simpler. > 2.) log files (they get appended to; that would make pooling somewhat > difficult; besides, what chance is there of a pool hit?), > backups files (including backups.old) > attrib files are pooled, contrary to popular belief, and that makes > sense, because they are often identical with the same attrib file from > the previous backup(s). Yes. I am aware of this from the routines I wrote to check/fix pool consistency and missing links to the pool > > > The algorithm I implemented is somewhat similar: > 1.) Walk pool/, cpool/ and pc/, printing information on the files and > directories to a file (which will be quite large; by default I put it > on the destination pool FS, because there should be large amounts of > space there). > 2.) Sort the file with the 'sort' command. The lines in the file are > designed such that they will be sorted into a meaningful order: > - directories first, so I can create them and subsequently not worry > about whether the place I want to copy/link a file to already exists > or not > - files next, sorted by inode number, with the (c)pool file preceeding > its > pc/ links > The consequence is that I get all references to one inode on adjacent > lines. The first time, I copy the file. For the repetitions, I link to > the first copy. All I need to keep in memory is something like one line > from the file list, one "previous inode number", one "file name of > previous inode". > 'sort' handles huge files quite nicely, but it seems to create large > (amounts of) files under /tmp, possibly under $TMPDIR if you set that > (not > sure). You need to make sure you've got the space, but if you're copying > a > multi-GB/TB pool, you probably have. My guess is that the necessary > amount > of space roughly equals the size of the file I'm sorting. > 3.) Walk the sorted file, line by line, creating directories and copying > files > (with File::Copy::cp, but I plan to change that to PoolWrite, so I can > add > (part of) one pool to an existing second pool, or something that > communicates over TCP/IP, so I can copy to a different machine) and > linking files (with Perl function link()). > In theory, a pool could also be compressed or uncompressed on the fly > (uncompressed for copying to zfs, for instance). Yes... I was thinking very similarly though you have clearly fleshed it out and coded it in more detail. I would love to see the code when it's stable and offer feedback or testing if that is helpful. > > > Once again, because people seem to be determined to miss the point: it's > *not* > processing by sorted inode numbers in order to save disk seeks that is the > point, it's the fact that the 'link' system
