Re: [BackupPC-users] Tar method - deleted files workaround
Hi, Craig Barratt wrote on 2013-10-06 17:08:36 -0700 [Re: [BackupPC-users] Tar method - deleted files workaround]: Chris, I've never looked into the --listed-incremental option for GNU tar. This might do something similar to what you want. http://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html from what I read there, surprisingly, tar files seem to be able to contain file deletions (i.e. extracting the archive will *delete* a file in the file system). This would mean it could actually work, at least in theory. On second reading, the documentation somewhat contradicts itself, so it's not really clear whether this is true. *Without* deletions being represented in the *tar file*, the whole exercise is somewhat pointless. (The contradiction I see is that the documentation clearly states that the snapshot file is not needed for restoration, yet GNU tar attempts to restore the exact state the file system had when the archive was created. In particular, it will delete those files in the file system that did not exist in their directories when the archive was created. - which it can't do without the snapshot file; it could only delete those files that the incremental run had detected had been removed since the baseline backup, provided this information is present in the incremental tar file.) Let's assume (and verify) they are, else I can delete what I've already written: ;-) I also don't know what is required to support it in BackupPC. The one problem I see is that you have a file with metadata (snapshot file) in addition to the tar stream. While you *could* just keep that file at the remote end (on the backup client), there would need to be some preprocessing, i.e. copying the file for independent incrementals. This would also mean that BackupPC would be keeping part of its state on the client machine, which would be new (and probably undesired). Alternatively, the file could be copied between BackupPC server and client, perhaps in DumpPre/PostUserCmd. All of this means that the administrator of BackupPC needs to know much more about the backup process and the client machines (where may we put the snapshot file?). Currently, we have default configuration values for tar backups over ssh that should mostly work. I doubt that would remain possible if this were to become default mode of operation. That doesn't mean it can't be done. It just means part of the process would need to be implemented by the (expert) BackupPC administrator. And for *local* backups (where BackupPC server == client), native support would be possible. Aside from that, we'd probably need support for file deletions in the BackupPC code. The rsync XferMethod already has that capability, so it shouldn't be too hard, I suppose. Providing this capability should be transparent for anyone not wanting to use --listed-incremental. Some new variables might also be needed both in the *Pre/PostUserCmds and, perhaps, TarClientCmd and/or TarFullArgs/TarIncrArgs, for instance the number of the baseline backup and the incremental level. Hmm. How do we *store* the snapshot file(s) in our pool FS? If the UserCmds need to access them, we'd either need some kind of hook, or they could just access $TopDir/pc/... directly (which is sort of ugly). Is anyone actually interested in experimenting with this option? Regards, Holger -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134071iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Tar method - deleted files workaround
On Sun, Oct 6, 2013 at 7:08 PM, Craig Barratt cbarr...@users.sourceforge.net wrote: I've never looked into the --listed-incremental option for GNU tar. This might do something similar to what you want. http://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html I also don't know what is required to support it in BackupPC. Amanda has used that for many years - it just needs a file maintained on the target hosts to track things.. If the file you specify with --listed-incremental does not exist, you get a full backup and the file is created listing the directories traversed with their timestamps. If the file does exist you get an incremental based on the timestamp in that file in 'gnudump' format taking everything (even old files) under new directories. The file is then modified in place for subsequent higher incremental levels. If you want to make additional incrementals based on the previous full, you have to copy the file before doing the backup so you will have an unmodified version later. The gnudump format includes the contents of directories so that restores can optionally delete files that weren't present at the time of the backup. Or at least that's the way it worked the last time I used amanda, which was several years ago. Now that I think about it, it would be kind of neat to have the gnudump format available from BackupPC_tarCreate and the archive host wrapper so you could make a reasonable series of incrementals for offsite or long term storage. -- Les Mikesell lesmikes...@gmail.com -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134071iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Tar method - deleted files workaround
On Mon, Oct 7, 2013 at 2:30 PM, Holger Parplies wb...@parplies.de wrote: http://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html from what I read there, surprisingly, tar files seem to be able to contain file deletions (i.e. extracting the archive will *delete* a file in the file system). This would mean it could actually work, at least in theory. On second reading, the documentation somewhat contradicts itself, so it's not really clear whether this is true. *Without* deletions being represented in the *tar file*, the whole exercise is somewhat pointless. It doesn't 'represent deletions', it stores the current full directory listing for each directory, even in an incremental run with each entry marked as included in this archive or not. During the restore, you have the option to delete anything that was not present when the backup was taken. The one problem I see is that you have a file with metadata (snapshot file) in addition to the tar stream. While you *could* just keep that file at the remote end (on the backup client), there would need to be some preprocessing, i.e. copying the file for independent incrementals. This would also mean that BackupPC would be keeping part of its state on the client machine, which would be new (and probably undesired). Alternatively, the file could be copied between BackupPC server and client, perhaps in DumpPre/PostUserCmd. All of this means that the administrator of BackupPC needs to know much more about the backup process and the client machines (where may we put the snapshot file?). Amanda has done this more or less forever and the admin doesn't need to know anything about it. It does use a client agent to do some of the grunge work, but root-ssh can do anything a local agent can do. The main job of that independent agent is to let all of the targets send size estimates (done with dump or the trick with gnutar where if it's output device is /dev/null it doesn't bother doing the work of writing the archive so getting --totals is pretty cheap) so amanda can schedule the right mix of fulls and incrementals to fill your tape - something we don't need to worry much about. That doesn't mean it can't be done. It just means part of the process would need to be implemented by the (expert) BackupPC administrator. And for *local* backups (where BackupPC server == client), native support would be possible. No, you'd just need a writable space to hold the files. Aside from that, we'd probably need support for file deletions in the BackupPC code. The rsync XferMethod already has that capability, so it shouldn't be too hard, I suppose. Providing this capability should be transparent for anyone not wanting to use --listed-incremental. That part becomes a little more complicated - unless there is already a perl module that understands gnudump format - and even then it would have to be modified to process deletions in the archive instead of a filesystem. Some new variables might also be needed both in the *Pre/PostUserCmds and, perhaps, TarClientCmd and/or TarFullArgs/TarIncrArgs, for instance the number of the baseline backup and the incremental level. I don't think those concepts would change. Hmm. How do we *store* the snapshot file(s) in our pool FS? If the UserCmds need to access them, we'd either need some kind of hook, or they could just access $TopDir/pc/... directly (which is sort of ugly). I've forgotten the exact details about how amanda handles these files (there will be overlapping sets so you can chose the level of your next run). It's not technically necessary to have the listing files on the central server at all, but it might make management easier. Is anyone actually interested in experimenting with this option? I don't currently have anything where gnutar would be a better option than rsync - but the handling did seem sensible back when people actually put stuff on media other than hard drives. -- Les Mikesell lesmikes...@gmail.com -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134071iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Tar method - deleted files workaround
Chris, I've never looked into the --listed-incremental option for GNU tar. This might do something similar to what you want. http://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html I also don't know what is required to support it in BackupPC. Craig -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134791iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Tar method - deleted files workaround
You could use a combination of
$Conf{DumpPostUserCmd}http://backuppc.sourceforge.net/faq/BackupPC.html#_conf_dumppreusercmd_
and
BackupPC_deleteFilehttp://sourceforge.net/apps/mediawiki/backuppc/index.php?title=BackupPC_DeleteFile
to
accomplish this.
On Tue, Sep 24, 2013 at 2:27 PM, Chris Adamson chris.adam...@mcri.edu.auwrote:
As a workaround for the tar method not detecting deletions on localhost
backups, would it be possible to write a script that is run after each
incremental to perform a file listing on the source directory and compare
that to the list of backed up files and delete those in the backup that are
no longer in the source directory. I realize this is a workaround and would
only be appropriate for localhost backups but it wouldn’t be a bad solution
since incrementals are quick, run when no one is using the system and it
would make the full backups much faster than rsyncd.
** **
*Dr **Chris Adamson
*Research Officer, Developmental Imaging, *Murdoch Childrens Research
Institute
*Murdoch Childrens Research Institute
Royal Children’s Hospital
Flemington Road, Parkville, Victoria 3052, Australia
www.mcri.edu.au
E chris.adam...@mcri.edu.au
T 03 9936 6780
** **
__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
__
--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from
the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Tar method - deleted files workaround
I realized this wont fix the problem of renames. My second thought on this
would be to develop a new transfer method that uses rsync initially to get a
file listing and then construct a list of files that need to be transferred.
Then you pass this to the tar method and use this rather than the -newer option
of tar. This would solve both the deletion and rename problem of incremental
tar.
From: Till Hofmann [mailto:hofmannt...@gmail.com]
Sent: Wednesday, 25 September 2013 3:07 AM
To: General list for user discussion, questions and support
Subject: Re: [BackupPC-users] Tar method - deleted files workaround
You could use a combination of
$Conf{DumpPostUserCmd}http://backuppc.sourceforge.net/faq/BackupPC.html#_conf_dumppreusercmd_
and
BackupPC_deleteFilehttp://sourceforge.net/apps/mediawiki/backuppc/index.php?title=BackupPC_DeleteFile
to accomplish this.
On Tue, Sep 24, 2013 at 2:27 PM, Chris Adamson
chris.adam...@mcri.edu.aumailto:chris.adam...@mcri.edu.au wrote:
As a workaround for the tar method not detecting deletions on localhost
backups, would it be possible to write a script that is run after each
incremental to perform a file listing on the source directory and compare that
to the list of backed up files and delete those in the backup that are no
longer in the source directory. I realize this is a workaround and would only
be appropriate for localhost backups but it wouldn't be a bad solution since
incrementals are quick, run when no one is using the system and it would make
the full backups much faster than rsyncd.
Dr Chris Adamson
Research Officer, Developmental Imaging, Murdoch Childrens Research Institute
Murdoch Childrens Research Institute
Royal Children's Hospital
Flemington Road, Parkville, Victoria 3052, Australia
www.mcri.edu.auhttp://www.mcri.edu.au/
E chris.adam...@mcri.edu.aumailto:chris.adam...@mcri.edu.au
T 03 9936 6780tel:03%209936%206780
__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
__
--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.netmailto:BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
If you have any question, please contact MCRI IT Helpdesk for further
assistance.
__
__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
__--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
