Re: [BackupPC-users] Switching backup methods
Jeffrey J. Kosowsky wrote: > > Maybe this is offtopic, but wouldn't it be helpful to have something > like a Tar+ Xfer incremental method that would make up for tar's > deficiencies. > > For example if you ran a "find" on the share and combined that with > the tar data then presumably you would be able to identify which files > have been moved or deleted (vs. which ones are unchanged). This would > (in general) not be too intensive either computationally or from a > bandwidth perspective and would allow Tar incrementals to be "correct" > (of course it still wouldn't detect files that changed but had an > earlier timestamp but generally that is a less frequent case). > > Of course if you go too far down this path, you might end up rewriting > rsync, but at least there might be some low hanging fruit that would > improve the tar method for incrementals. Gnutar already has this mechanism with the "--listed-incremental filename" option. If the specified file does not exist, a full run is done and the file is created containing a list of the directories traversed. If the file does exist, an incremental run is done based on the timestamp of the file and ctimes of the target files and including the complete contents of any directories that are not listed. The backup uses the 'gnudump' format which contains directory listings of all files present at the time of the backup so you can optionally delete anything that wasn't there during a restore. Amanda knows how to use this, but there are some downsides. During incremental runs, the file is modified in place assuming you want incremental levels. If you want incrementals based on the last full, you have to copy the file before the incremental to reuse next time. Obviously, this requires client-side code to manage the files. 'Star' also has an enhanced incremental mode that works more like dump, but it requires runs to be based on filesystem boundaries. -- Les Mikesell lesmikes...@gmail.com -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Switching backup methods
Holger Parplies wrote at about 13:45:34 +0200 on Tuesday, September 29, 2009: > Hi, > > Jeffrey J. Kosowsky wrote on 2009-09-29 02:49:11 -0400 [Re: [BackupPC-users] > Switching backup methods]: > > Holger Parplies wrote at about 15:54:25 +0200 on Saturday, September 26, > > 2009: > > > [...] > > > 3. I believe I recall reading that *restoring* a backup *made with > > rsync* > > >with the tar XferMethod produces warnings (or errors?) - probably > > >because tar doesn't understand the "deleted file" entries in the > > >attrib files. [...] > > > > If I am understanding you correctly, then presumably this problem > > would only occur for incremental backups and not for full backups > > since the concept of "deleted file" shouldn't exist in a full backup. > > I agree on that, but I have never experienced the problem myself, just read > about it one or two times over the last years. At the time, I wasn't familiar > with the attrib file contents. Now, it makes sense that this problem could > occur. But I might be mistaking things. The important part is that if you run > into problems when restoring after changing the XferMethod, changing it back > for the restore might help. > > > Second, how do tar incrementals signal deleted files if they don't use > > the "deleted file type" (10) attribute? > > They don't. tar can't detect deleted files. Deleting a file doesn't change > its > modification time, it deletes it ;-). > > > I have never used tar as a XferMethod before so I don't understand how it > > works differently from rsync here... > > Well, you don't have fileLists ... a full tar backup is just a tar stream of > all the files on the sender side, which is then interpreted by BackupPC and > integrated into the pool. An incremental tar backup is a tar stream of all > the files on the sender side that have changed since a timestamp. Files not > present are either unchanged or deleted (or moved to a new location). There's > no way to tell which. > > Regards, > Holger > Maybe this is offtopic, but wouldn't it be helpful to have something like a Tar+ Xfer incremental method that would make up for tar's deficiencies. For example if you ran a "find" on the share and combined that with the tar data then presumably you would be able to identify which files have been moved or deleted (vs. which ones are unchanged). This would (in general) not be too intensive either computationally or from a bandwidth perspective and would allow Tar incrementals to be "correct" (of course it still wouldn't detect files that changed but had an earlier timestamp but generally that is a less frequent case). Of course if you go too far down this path, you might end up rewriting rsync, but at least there might be some low hanging fruit that would improve the tar method for incrementals. -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Switching backup methods
Hi, Jeffrey J. Kosowsky wrote on 2009-09-29 02:49:11 -0400 [Re: [BackupPC-users] Switching backup methods]: > Holger Parplies wrote at about 15:54:25 +0200 on Saturday, September 26, 2009: > > [...] > > 3. I believe I recall reading that *restoring* a backup *made with rsync* > >with the tar XferMethod produces warnings (or errors?) - probably > >because tar doesn't understand the "deleted file" entries in the > >attrib files. [...] > > If I am understanding you correctly, then presumably this problem > would only occur for incremental backups and not for full backups > since the concept of "deleted file" shouldn't exist in a full backup. I agree on that, but I have never experienced the problem myself, just read about it one or two times over the last years. At the time, I wasn't familiar with the attrib file contents. Now, it makes sense that this problem could occur. But I might be mistaking things. The important part is that if you run into problems when restoring after changing the XferMethod, changing it back for the restore might help. > Second, how do tar incrementals signal deleted files if they don't use > the "deleted file type" (10) attribute? They don't. tar can't detect deleted files. Deleting a file doesn't change its modification time, it deletes it ;-). > I have never used tar as a XferMethod before so I don't understand how it > works differently from rsync here... Well, you don't have fileLists ... a full tar backup is just a tar stream of all the files on the sender side, which is then interpreted by BackupPC and integrated into the pool. An incremental tar backup is a tar stream of all the files on the sender side that have changed since a timestamp. Files not present are either unchanged or deleted (or moved to a new location). There's no way to tell which. Regards, Holger -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Switching backup methods
Holger Parplies wrote at about 15:54:25 +0200 on Saturday, September 26, 2009: > Hi, > > Tino Schwarze wrote on 2009-09-26 13:32:58 +0200 [Re: [BackupPC-users] > Switching backup methods]: > > On Fri, Sep 25, 2009 at 05:04:24PM -0400, jingai wrote: > > > Is it OK to switch between backup methods at any time, or do I need to > > > do something beforehand? > > > > You might have to adjust your excludes. Have a look at the Wiki for more > > information. Apart from that, there's nothing special to consider. > > that's almost true. > > 3. I believe I recall reading that *restoring* a backup *made with rsync* > with >the tar XferMethod produces warnings (or errors?) - probably because tar >doesn't understand the "deleted file" entries in the attrib files. While I >can't see that giving incorrect results, it probably won't give you a good >feeling to read that your restore completed with 579 errors, which you'll >have to individually check. So if you need to restore an old backup after >having switched the XferMethod, it would probably be a good idea to switch >it back for the restore. Recent versions of BackupPC (>= 3.0.0, I believe) >record the XferMethod a backup was done with in the backups file (but I >don't think they automatically use that information for doing a restore). > If I am understanding you correctly, then presumably this problem would only occur for incremental backups and not for full backups since the concept of "deleted file" shouldn't exist in a full backup. Second, how do tar incrementals signal deleted files if they don't use the "deleted file type" (10) attribute? I have never used tar as a XferMethod before so I don't understand how it works differently from rsync here... -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Switching backup methods
Hi, Tino Schwarze wrote on 2009-09-26 13:32:58 +0200 [Re: [BackupPC-users] Switching backup methods]: > On Fri, Sep 25, 2009 at 05:04:24PM -0400, jingai wrote: > > Is it OK to switch between backup methods at any time, or do I need to > > do something beforehand? > > You might have to adjust your excludes. Have a look at the Wiki for more > information. Apart from that, there's nothing special to consider. that's almost true. 1. rsync will copy over *everything* on the first backup (actually upto the first *full* backup), which might not be expected. In your case (you were doing tar backups anyway), that will not be a problem, but in cases where the switch to rsync is done because of bandwidth limitations, it should be noted. Actually, it also matters in your case, because if you only run one rsync incremental after switching, you'll probably observe that it is much slower than a tar incremental was before, so you should know why that is :). 2. You should do a full backup after switching transfer methods. There are subtle differences (see above), and a full backup will make sure everything is as the new transfer method expects. 3. I believe I recall reading that *restoring* a backup *made with rsync* with the tar XferMethod produces warnings (or errors?) - probably because tar doesn't understand the "deleted file" entries in the attrib files. While I can't see that giving incorrect results, it probably won't give you a good feeling to read that your restore completed with 579 errors, which you'll have to individually check. So if you need to restore an old backup after having switched the XferMethod, it would probably be a good idea to switch it back for the restore. Recent versions of BackupPC (>= 3.0.0, I believe) record the XferMethod a backup was done with in the backups file (but I don't think they automatically use that information for doing a restore). Aside from that, though I also like to point at the Wiki, I believe it is still sort of non-functional. At least I can't find anything except the main page ... Regards, Holger -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Switching backup methods
On Fri, Sep 25, 2009 at 05:04:24PM -0400, jingai wrote: > Is it OK to switch between backup methods at any time, or do I need to > do something beforehand? You might have to adjust your excludes. Have a look at the Wiki for more information. Apart from that, there's nothing special to consider. HTH, Tino. -- "What we nourish flourishes." - "Was wir nähren erblüht." www.lichtkreis-chemnitz.de www.craniosacralzentrum.de -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Switching backup methods
On Sep 26, 2009, at 9:54 AM, Holger Parplies wrote: > Hi, > > Tino Schwarze wrote on 2009-09-26 13:32:58 +0200 [Re: [BackupPC- > users] Switching backup methods]: >> On Fri, Sep 25, 2009 at 05:04:24PM -0400, jingai wrote: >>> Is it OK to switch between backup methods at any time, or do I >>> need to >>> do something beforehand? >> >> You might have to adjust your excludes. Have a look at the Wiki for >> more >> information. Apart from that, there's nothing special to consider. > > that's almost true. > > 2. You should do a full backup after switching transfer methods. > There are > subtle differences (see above), and a full backup will make sure > everything > is as the new transfer method expects. I figured that, and that's what I am doing. > 3. I believe I recall reading that *restoring* a backup *made with > rsync* with > the tar XferMethod produces warnings (or errors?) - probably > because tar > doesn't understand the "deleted file" entries in the attrib files. > While I > can't see that giving incorrect results, it probably won't give > you a good > feeling to read that your restore completed with 579 errors, which > you'll > have to individually check. So if you need to restore an old > backup after > having switched the XferMethod, it would probably be a good idea > to switch > it back for the restore. Recent versions of BackupPC (>= 3.0.0, I > believe) > record the XferMethod a backup was done with in the backups file > (but I > don't think they automatically use that information for doing a > restore). I'm not too concerned about the old backups for the machines in question, so this is fine. > Aside from that, though I also like to point at the Wiki, I believe > it is > still sort of non-functional. At least I can't find anything except > the main > page ... I thought I was the only one. Are we just missing something...? Thanks for the help, Jonathan -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
