Re: [BackupPC-users] failed partial backups deleting files from previous partials
On 09/28 11:34 , Rob Owens wrote: > Very interesting discussion. Here's a link to an article on backups > that you've probably all seen, but it's very interesting: > http://www.mikerubel.org/computers/rsync_snapshots/ I haven't seen that, and it was very clever. Learned about some features of rsync that didn't exist when I started learning about it. Thanks for posting that. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
Craig Barratt wrote: > Les writes: > >> But will you still be able to restore the file state as of each separate >> run? Sometimes the reason you are restoring is that the current version >> is a mess. The issue could probably be avoided with a complete tree >> link copy before starting followed by in in-place update, but that is >> probably even less efficient. > > Yes, the exact state of each older backup can be re-created. > > As each backup occurs, two trees are operated on: > > - the in-place update of the newest complete backup > > - a new tree is created with the deltas to get to the > original, now older, backup. Eg: if a new file replaces > an old one, the old file is moved to this tree. > > This new tree becomes the second-most-recent backup, stored > as a set of deltas (changes) relative to the newest complete > backup. > > Each older backup in turn would be represented on disk as a set of > deltas from the more recent one. The only "filled" backup is the > newest one. To view or restore a particular backup, you start with > the most recent (complete) backup, and successively apply the deltas > to get to the older backup. > > It's just the time-reverse of what BackupPC does already: to view > or restore an incremental, it starts with the next older full and > applies the deltas (forward in time) for each incremental level. > > The new, proposed, layout would completely decouple the storage > from the backup type (ie: whether it was an incremental or full > the storage method would be the same). > > As I mentioned earlier, you most often view and restore the > most recent backup. So the penalty doing the delta merges > for old backups is minor. > > You can delete the oldest backup at any time, since nothing > depends on it. No longer will there be the issue of keeping > a full around because incrementals depend on it. There would > be no difference between expiry of incrementals and fulls - they > would be treated the same since they are stored the same way. > > As discussed earlier, the only tricky part is deleting a backup > that isn't the oldest (as needed for exponential expiry). That > requires you to merge two deltas into one. > > Craig > Very interesting discussion. Here's a link to an article on backups that you've probably all seen, but it's very interesting: http://www.mikerubel.org/computers/rsync_snapshots/ -Rob - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
On 09/26 04:31 , Craig Barratt wrote: > Rsync didn't get added to BackupPC until 2.0.0, so the > architecture didn't consider how rsync typically works. ah, ok. I've only used it since 2.0.1 I think. I always guessed that rsync was the original transport mechanism, and the others were added later. the architecture makes more sense now tho. > Right. Tar and smbclient would also be changed to do in-place > updating too. nifty. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
Les writes: > But will you still be able to restore the file state as of each separate > run? Sometimes the reason you are restoring is that the current version > is a mess. The issue could probably be avoided with a complete tree > link copy before starting followed by in in-place update, but that is > probably even less efficient. Yes, the exact state of each older backup can be re-created. As each backup occurs, two trees are operated on: - the in-place update of the newest complete backup - a new tree is created with the deltas to get to the original, now older, backup. Eg: if a new file replaces an old one, the old file is moved to this tree. This new tree becomes the second-most-recent backup, stored as a set of deltas (changes) relative to the newest complete backup. Each older backup in turn would be represented on disk as a set of deltas from the more recent one. The only "filled" backup is the newest one. To view or restore a particular backup, you start with the most recent (complete) backup, and successively apply the deltas to get to the older backup. It's just the time-reverse of what BackupPC does already: to view or restore an incremental, it starts with the next older full and applies the deltas (forward in time) for each incremental level. The new, proposed, layout would completely decouple the storage from the backup type (ie: whether it was an incremental or full the storage method would be the same). As I mentioned earlier, you most often view and restore the most recent backup. So the penalty doing the delta merges for old backups is minor. You can delete the oldest backup at any time, since nothing depends on it. No longer will there be the issue of keeping a full around because incrementals depend on it. There would be no difference between expiry of incrementals and fulls - they would be treated the same since they are stored the same way. As discussed earlier, the only tricky part is deleting a backup that isn't the oldest (as needed for exponential expiry). That requires you to merge two deltas into one. Craig - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
Carl writes: > ah, I see. > I think I see some of the arguments for doing things this way, but what was > *your* reasoning when you first designed this architecture? It's rather > unusual compared to the other rsync-backup programs I've seen (or built). Rsync didn't get added to BackupPC until 2.0.0, so the architecture didn't consider how rsync typically works. Also, I didn't discover the various rsync backup tools until after I had implemented it in BackupPC. > > In 3.1.0 I've added a check that a new partial won't replace an > > old partial unless it contains more files, so that should avoid > > the annoying problem of a new partial potentially being smaller > > than the last. > > Much appreciated. Thanks. :) > Ideally, the new partial would be merged with the old partial tho. I guess > we'll have to wait for a newer version for that feature. (alternatively, how > much would it cost for someone to pay you to add that feature?) Yes, it will have to wait until 4.0. I appreciate the offer, but bribes won't change the timing. > > The issue is that the proposed new style of storage would be most > > efficient with in-place updating of the last (complete) backup. > > ah, so you're planning on abandoning the creation of a 'new' tree with every > backup, and going to in-place updating, which means we get normal rsync > behavior. (Thus obviating my comment above). Right. Tar and smbclient would also be changed to do in-place updating too. Craig - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
Carl Wilhelm Soderstrom wrote: >>> My understanding of how BackupPC works in this regard is imperfect. Why >>> can't the old partial backup be updated, in the way that conventional rsync >>> updates an old copy? >> BackupPC doesn't do anything in place on the server. It always >> creates a new directory tree for each backup. In contrast, with >> typical usage rsync does things in place. > > ah, I see. > I think I see some of the arguments for doing things this way, but what was > *your* reasoning when you first designed this architecture? It's rather > unusual compared to the other rsync-backup programs I've seen (or built). If you are going to keep snapshot-looking backups around at various points in time you can't modify in place. That is, if I restore from a certain day's backup run, I expect it to still have only the files that existed at that time. >> In 3.1.0 I've added a check that a new partial won't replace an >> old partial unless it contains more files, so that should avoid >> the annoying problem of a new partial potentially being smaller >> than the last. > > Much appreciated. Thanks. :) > Ideally, the new partial would be merged with the old partial tho. I guess > we'll have to wait for a newer version for that feature. (alternatively, how > much would it cost for someone to pay you to add that feature?) > >> The issue is that the proposed new style of storage would be most >> efficient with in-place updating of the last (complete) backup. > > ah, so you're planning on abandoning the creation of a 'new' tree with every > backup, and going to in-place updating, which means we get normal rsync > behavior. (Thus obviating my comment above). But will you still be able to restore the file state as of each separate run? Sometimes the reason you are restoring is that the current version is a mess. The issue could probably be avoided with a complete tree link copy before starting followed by in in-place update, but that is probably even less efficient. -- Les Mikesell [EMAIL PROTECTED] - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
On 09/24 12:38 , Craig Barratt wrote: > > My understanding of how BackupPC works in this regard is imperfect. Why > > can't the old partial backup be updated, in the way that conventional rsync > > updates an old copy? > > BackupPC doesn't do anything in place on the server. It always > creates a new directory tree for each backup. In contrast, with > typical usage rsync does things in place. ah, I see. I think I see some of the arguments for doing things this way, but what was *your* reasoning when you first designed this architecture? It's rather unusual compared to the other rsync-backup programs I've seen (or built). > In 3.1.0 I've added a check that a new partial won't replace an > old partial unless it contains more files, so that should avoid > the annoying problem of a new partial potentially being smaller > than the last. Much appreciated. Thanks. :) Ideally, the new partial would be merged with the old partial tho. I guess we'll have to wait for a newer version for that feature. (alternatively, how much would it cost for someone to pay you to add that feature?) > The issue is that the proposed new style of storage would be most > efficient with in-place updating of the last (complete) backup. ah, so you're planning on abandoning the creation of a 'new' tree with every backup, and going to in-place updating, which means we get normal rsync behavior. (Thus obviating my comment above). I appreciate your exposition. :) -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
Carl writes: > > It depends on XferMethod. For rsync it should not delete files from a > > partial. For smb and tar, it starts a new transfer, and if any files > > get transfered (even if less than the prior partial), the new partial > > replaces the old. I should change that so it doesn't save the new > > partial (and therefore delete the old one) unless it has more files > > than the older partial. > > My understanding of how BackupPC works in this regard is imperfect. Why > can't the old partial backup be updated, in the way that conventional rsync > updates an old copy? BackupPC doesn't do anything in place on the server. It always creates a new directory tree for each backup. In contrast, with typical usage rsync does things in place. In 3.1.0 I've added a check that a new partial won't replace an old partial unless it contains more files, so that should avoid the annoying problem of a new partial potentially being smaller than the last. > > Longer term (main feature in 4.0?) I plan to completely change > > the way BackupPC stores backups. The most recent backup should > > always be complete (filled), and the older ones will be stored > > only as reverse-time deltas, independent of whether they were > > made with a full or incremental. Doing a backup involves > > in-place updating of the last backup, and concurrently generating > > a new tree of reverse deltas. A partial would simply mean a > > partial update of the most recent complete backup, and it would > > behave better than the current one. > > > > This makes expiry easy: you can delete the oldest reverse delta whenever > > you want. The exponential expiry (ie: deleting a backup in the middle) > > is slightly trickier - you basically have to merge to deltas to delete > > a backup. > > How much of a performance hit would it be to do this delta merging? Or could > it be optimized to be part of some other nightly cleanup operation? I'm not sure about the penalty merging backups together. It is only needed in the case when you expire a backup in the middle of the backup set. > > The one complication is whether I should go to the trouble of "unwinding" > > the in-place changes when a backup is aborted (ie: re-merging the > > partial first reverse delta). Essentially that is equivalent to > > not saving partials. It would be easier to just mark the first > > delta as a partial, and always keeping a partial. > > I'll leave that for you to experiment with and decide. :) > Are you meaning that in "always keeping a partial" the "partial" would > actually be fully-complete in the common case (where the backup completed > successfully)? I understand your question, but I do not understand your last > sentence in the above paragraph. The issue is that the proposed new style of storage would be most efficient with in-place updating of the last (complete) backup. If the backup failed part way, the most recent (complete) backup is now really a partial, and the delta prior to it (that was being generated during the backup) reflects the last full backup. My point was that it is easier to leave it that way rather than "delete the partial" which requires unwinding all the changes that were made during the failed backup, so as to put the last (complete) backup in its original state. Hope that helps. Craig - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
Apologies for the slow response on this. I've been preoccupied with a Very
Large Project for the past 3 weeks.
On 09/06 05:57 , Craig Barratt wrote:
> Carl writes:
> > BackupPC does not pick up where a 'partial' transfer left off in many cases.
> > I don't fully understand the mechanism (tho it is somewhat explained here:
> > http://osdir.com/ml/sysutils.backup.backuppc.general/2004-08/msg00013.html
> > ); but the result is that perfectly good files are deleted from the partial
> > backup.
>
> It depends on XferMethod. For rsync it should not delete files from a
> partial. For smb and tar, it starts a new transfer, and if any files
> get transfered (even if less than the prior partial), the new partial
> replaces the old. I should change that so it doesn't save the new
> partial (and therefore delete the old one) unless it has more files
> than the older partial.
My understanding of how BackupPC works in this regard is imperfect. Why
can't the old partial backup be updated, in the way that conventional rsync
updates an old copy?
> Also, if the partial is too old, it will be ignored and even rsync will
> start a new backup, which if aborted could result in a partial with fewer
> files. That's configurable with $Conf{PartialAgeMax}.
>
> Which XferMethod are you using when you see this problem, and what
> is $Conf{PartialAgeMax}?
This was done with rsync, and
$Conf{PartialAgeMax} = 3;
but backups were done one day after another. (I don't *think* they had
enough time for the beginning of one to be more than 3 days before the
beginning of the next).
> Longer term (main feature in 4.0?) I plan to completely change
> the way BackupPC stores backups. The most recent backup should
> always be complete (filled), and the older ones will be stored
> only as reverse-time deltas, independent of whether they were
> made with a full or incremental. Doing a backup involves
> in-place updating of the last backup, and concurrently generating
> a new tree of reverse deltas. A partial would simply mean a
> partial update of the most recent complete backup, and it would
> behave better than the current one.
>
> This makes expiry easy: you can delete the oldest reverse delta whenever
> you want. The exponential expiry (ie: deleting a backup in the middle)
> is slightly trickier - you basically have to merge to deltas to delete
> a backup.
How much of a performance hit would it be to do this delta merging? Or could
it be optimized to be part of some other nightly cleanup operation?
> This also has other advantages too:
>
> - it is easy to support "incremental only" rsync backups too, since the
>most recent backup is always filled,
>
> - you don't need to merge any prior backups to have the starting point
>for rsync.
>
> - since most restores are off the most recent version, you only need
>the performance impact of merging multiple backups when browsing
>or restoring older backups.
>
> The one complication is whether I should go to the trouble of "unwinding"
> the in-place changes when a backup is aborted (ie: re-merging the
> partial first reverse delta). Essentially that is equivalent to
> not saving partials. It would be easier to just mark the first
> delta as a partial, and always keeping a partial.
I'll leave that for you to experiment with and decide. :)
Are you meaning that in "always keeping a partial" the "partial" would
actually be fully-complete in the common case (where the backup completed
successfully)? I understand your question, but I do not understand your last
sentence in the above paragraph.
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
Carl writes:
> BackupPC does not pick up where a 'partial' transfer left off in many cases.
> I don't fully understand the mechanism (tho it is somewhat explained here:
> http://osdir.com/ml/sysutils.backup.backuppc.general/2004-08/msg00013.html
> ); but the result is that perfectly good files are deleted from the partial
> backup.
It depends on XferMethod. For rsync it should not delete files from a
partial. For smb and tar, it starts a new transfer, and if any files
get transfered (even if less than the prior partial), the new partial
replaces the old. I should change that so it doesn't save the new
partial (and therefore delete the old one) unless it has more files
than the older partial.
Also, if the partial is too old, it will be ignored and even rsync will
start a new backup, which if aborted could result in a partial with fewer
files. That's configurable with $Conf{PartialAgeMax}.
Which XferMethod are you using when you see this problem, and what
is $Conf{PartialAgeMax}?
Longer term (main feature in 4.0?) I plan to completely change
the way BackupPC stores backups. The most recent backup should
always be complete (filled), and the older ones will be stored
only as reverse-time deltas, independent of whether they were
made with a full or incremental. Doing a backup involves
in-place updating of the last backup, and concurrently generating
a new tree of reverse deltas. A partial would simply mean a
partial update of the most recent complete backup, and it would
behave better than the current one.
This makes expiry easy: you can delete the oldest reverse delta whenever
you want. The exponential expiry (ie: deleting a backup in the middle)
is slightly trickier - you basically have to merge to deltas to delete
a backup.
This also has other advantages too:
- it is easy to support "incremental only" rsync backups too, since the
most recent backup is always filled,
- you don't need to merge any prior backups to have the starting point
for rsync.
- since most restores are off the most recent version, you only need
the performance impact of merging multiple backups when browsing
or restoring older backups.
The one complication is whether I should go to the trouble of "unwinding"
the in-place changes when a backup is aborted (ie: re-merging the
partial first reverse delta). Essentially that is equivalent to
not saving partials. It would be easier to just mark the first
delta as a partial, and always keeping a partial.
The reverse delta concept has been suggested by several people including
Jason Hughes, and I had thought about it too. But I wish I did it this
way at the inception of BackupPC. I just didn't think clearly about
separating how the backup data arrives (full and increment) and how it
is should best be stored.
Craig
-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
This happened to me again last night. I lost 2 GB of partial transfer (which took a long time to get -- uploaded from a home internet connection). One way to fix this would be to do a straight rsync of the files to a temporary location, then let BackupPC do its thing, integrating the new files into the pool. But this would require a very large temporary location. So maybe BackupPC could be programed to do its thing after 'x' number of MB have been transferred to the temporary location, thus reducing the space requirement. Meanwhile rsync would continue transferring files and BackupPC would "do its thing" again once the temporary location had a new set of 'x' MB worth of files. I have no idea if this is feasible or not, but I figured I'd throw it out there for the experts on the list to think about. -Rob Carl Wilhelm Soderstrom wrote: > BackupPC does not pick up where a 'partial' transfer left off in many cases. > I don't fully understand the mechanism (tho it is somewhat explained here: > http://osdir.com/ml/sysutils.backup.backuppc.general/2004-08/msg00013.html > ); but the result is that perfectly good files are deleted from the partial > backup. > > This is completely non-intuitive, if one is familiar with the way rsync > works. One would expect subsequent partial backups to keep transferring new > files, instead of deleting perfectly good files. > > I've also just tried this with tar under BackupPC 3.0.0, and the behavior is > still there. > > I know this is an old issue, but it's bitten me a few times, and recently > caused me to lose an entire weekend's worth of backup data transfer; so I'd > like to re-iterate a desire to have this bug fixed. > > Can we at least put in a minor code change to not trash files from a partial > backup? > > - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] failed partial backups deleting files from previous partials
I'm also going do be depending on this feature shortly (not discarding partial transfers), so I second Carl's proposal. -Rob Carl Wilhelm Soderstrom wrote: > BackupPC does not pick up where a 'partial' transfer left off in many cases. > I don't fully understand the mechanism (tho it is somewhat explained here: > http://osdir.com/ml/sysutils.backup.backuppc.general/2004-08/msg00013.html > ); but the result is that perfectly good files are deleted from the partial > backup. > > This is completely non-intuitive, if one is familiar with the way rsync > works. One would expect subsequent partial backups to keep transferring new > files, instead of deleting perfectly good files. > > I've also just tried this with tar under BackupPC 3.0.0, and the behavior is > still there. > > I know this is an old issue, but it's bitten me a few times, and recently > caused me to lose an entire weekend's worth of backup data transfer; so I'd > like to re-iterate a desire to have this bug fixed. > > Can we at least put in a minor code change to not trash files from a partial > backup? > > - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] failed partial backups deleting files from previous partials
BackupPC does not pick up where a 'partial' transfer left off in many cases. I don't fully understand the mechanism (tho it is somewhat explained here: http://osdir.com/ml/sysutils.backup.backuppc.general/2004-08/msg00013.html ); but the result is that perfectly good files are deleted from the partial backup. This is completely non-intuitive, if one is familiar with the way rsync works. One would expect subsequent partial backups to keep transferring new files, instead of deleting perfectly good files. I've also just tried this with tar under BackupPC 3.0.0, and the behavior is still there. I know this is an old issue, but it's bitten me a few times, and recently caused me to lose an entire weekend's worth of backup data transfer; so I'd like to re-iterate a desire to have this bug fixed. Can we at least put in a minor code change to not trash files from a partial backup? -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
