Re: Making a bootable backup (hard)disk... how?
On Sunday 10 June 2012 23:14:57 Ronald F. Guilmette wrote: Well, nevermind about that. I get the general idea, i.e. that dumping at level N causes dumping of everything that has changed since the last dump at level N-1. A point to be aware of is that if you restore from a full backup followed by one or more incrementals then you will restore ALL files which were present when each dump was made - including any files which have been intentionally deleted since the dump was created. This isn't normally a problem but there might be some obscure situations where it could be. -- Mike Clarke ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Monday 11 June 2012 00:03:59 Daniel Feenberg wrote: It does occur to me that /etc is not a felicitous place to keep this information, but given the desirability of dumping filesystems in read only state, placing the dump dates in the filesystem itself isn't feasible. Dumping with the -L option creates a temporary read only snapshot which is used as the source for the backup. This enables you to safely backup a live filesystem, More background on snapshots at http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/snapshots.html -- Mike Clarke ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
Hello. 2012/06/09 19:30:53 -0700 Ronald F. Guilmette r...@tristatelogic.com = To Arthur Chance : RFG Thank you Arthur, and yes, trying to back up a partition that's currently RFG mounted r/w using dd will almost certainly not produce the desired results. You can make snapshot to back up rw-mounted volume with dd. dump(8)ing rw-mounted ufs makes a snapshot behind the scenes. -- Peter Vereshagin pe...@vereshagin.org (http://vereshagin.org) pgp: A0E26627 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Sunday 10 June 2012 03:30:53 Ronald F. Guilmette wrote: I don't care to take own my system to make backups... and don't believe that I should have to do so, and thus, this is one of the reasons why I would prefer to use something like cpio. Also, I don't like backups taking longer than absolutely necessary, and this is why I am specifically _not_ attracted to either the dd solution or to dump/restore, Not an immediate solution but have you considered switching from UFS to ZFS ? If you have sufficient memory and CPU power then this might be worth the effort. Creating ZFS snapshots and backing them up incrementally with zfs send | zfs receive should be very quick. -- Mike Clarke ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
From owner-freebsd-questi...@freebsd.org Sat Jun 9 21:33:57 2012 To: Arthur Chance free...@qeng-ho.org Date: Sat, 09 Jun 2012 19:30:53 -0700 From: Ronald F. Guilmette r...@tristatelogic.com Cc: freebsd-questions@freebsd.org Subject: Re: Making a bootable backup (hard)disk... how? In message 4fd38b9a.4010...@qeng-ho.org, Arthur Chance free...@qeng-ho.org wrote: There's a BFI (brute force and ignorance) way of doing it in the base system - dd. Provided your system disk is quiescent (ideally when running from a live CD or all partitions mounted read-only, otherwise pray to the deity of your choice) and the backup disk is a) at least as large as the system disk, and b) has the same sector size, then a simple dd from the system disk to the backup should work. Thank you Arthur, and yes, trying to back up a partition that's currently mounted r/w using dd will almost certainly not produce the desired results. And of course, as you correctly note also, the target partition had best be at least as big as the source (and perhaps even identical in size). I don't care to take own my system to make backups... and don't believe that I should have to do so, and thus, this is one of the reasons why I would prefer to use something like cpio. Also, I don't like backups taking longer than absolutely necessary, and this is why I am specifically _not_ attracted to either the dd solution or to dump/restore, because as I understand it, with either of these methods you end up copying perhaps a metric buttload worth of unallocated free disk space. (I would prefer not to do that. It just seems wasteful... of time, if nothing else.) Dump does *NOT* copy 'unallocated free disk space'. It does a directory- based traversal. And copies only the contents of the files encountered. does not copy 'deleted but still open' files, nor does it include deleted directory entries in the copy of the directory. dump _can_ be used on a r/w-mounted filesystem. It is smart enough to make multiple passes, note files that have changed since the prior pass, and re-copy those changed files into the archive. As long as changes are relatvely 'infrequent', and to 'small' files, dump will generally manage to outrun the changes. Since dump works through the O/S, it also catches data and/or meta-data changes that are buffered/cached in O/S memory that have not actually made it to disk, yet. Using dump on a mounted, active, writable filesystem is not recommended, but it -does- work adequately, assuming the write activity on the filesystem is low. *IF* the filesystem is fairly full, with source and dest arranged such that they do not saturate I/O controller capabilities, 'dd' (or, preferably 'ddd' which is a double-buffered drop-in replacement) with large buffers (several times the size of the 'on disk' cache'), _is_ the fastest way to make a backup of the data. Also, in one case, one of my partitions has one directory that contains a really massive amount of stuff, and I specifically _don't_ need any of that particular stuff (in that one directory) backed up. So again, I'm looking at tar or cpio or perhaps pax. (Of course cpio is more full-featured than tar, and I don't really know anything about pax, so that leaves me with cpio.) Pax (the portable archiver) is the new, improved, with miracle whitener cpio -- does everthing tradtional cpio does, and more. it will read/write four different (and mutually incompatable) 'cpio' formats, and two different 'tar' formats. One can replace pretty much any use of 'cpio' or 'tar' (or their cousins) with pax. It _is_ worth investigating. when. (Sigh. The good old daze when men were men, and the bits ran scared!) Moral of the story is that sometimes it actually does pay to be a smartass. Regards, rfg P.S. It really is a Damn Shame[tm] that nobody ever hacked FreeBSD cpio to make it be able to copy (a) the extra file flag/mode bits and/or (b) file ACLs and/or (c) file attributes. If pax doesn't already support extended flags, it would be a relatively trivial hack to get it to do so. Ditto for ACLs. pax uses libarchive (man 3 archive) which has the capabilities for handling those things in the description of an archive entry. One would probably want to declare it as a separate archive 'type' -- to support auto-detect of the inclusion of that 'extended' metadata. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
In message alpine.bsf.2.00.1206092039260.71...@wonkity.com, you wrote: On Sat, 9 Jun 2012, Ronald F. Guilmette wrote: Also, I don't like backups taking longer than absolutely necessary, and this is why I am specifically _not_ attracted to either the dd solution or to dump/restore, because as I understand it, with either of these methods you end up copying perhaps a metric buttload worth of unallocated free disk space. No, that is one of the biggest advantages of dump over dd. dump knows UFS, and only copies occupied sectors and needed information. OK. I obviously got the Wrong Impression about dump/restore. Thanks for setting me straight. (I got the Wrong Impression, I think, because I have read assertions like ...dump backs up at the filesystem block level What does that mean exactly? Use of the term block level in this context makes me think of something operating along the lines of dd.) Also, in one case, one of my partitions has one directory that contains a really massive amount of stuff, and I specifically _don't_ need any of that... backed up. Directories and files can be skipped with the nodump flag. I am in your debt again. I really will start looking at dump/restore. P.S. It really is a Damn Shame[tm] that nobody ever hacked FreeBSD cpio to make it be able to copy (a) the extra file flag/mode bits and/or (b) file ACLs and/or (c) file attributes. A quick search shows dump should support ACLs. The other stuff is also there. Try it. http://www.wonkity.com/~wblock/docs/html/backup.html THANK YOU! I am finding the above document very helpful. It really never even occured to me that one could pipe the output of dump directly to restore. (Obviously, that pairing creates something very much along the lines of cpio's -p option... which is one thing that I _am_ already familiar with.) Now that I've read your helpful document, I do have three short follow-up questions... 1) In your example under the heading Copying Filesystems, the second shell command line shown is: dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt restore -ruf -) Is that correct? Shouldn't it actually be this instead? dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt/usr restore -ruf -) I mean if the goal is to create a whole backup under /mnt that looks just like the whole original system, then shouldn't ``restoration'' of the /usr part of the original system go into /mnt/usr? 2) Towards the end of your document you mention rsync. Assuming that I have made a backup of my entire /usr partition (using dumprestore) into /mnt/usr and that at midnight every night from now on I want to simply refresh that and bring it up to date with the current contents of my actual /usr partition, what is the most proper way to do this? Should I use rsync for that? Or should I use dumprestore again? If that latter, then how exactly does that work? I mean if I do the pipeline from dump to restore as you have shown in your examples in your Copying Filesystems section, then what must I do in order prevent dump from dumping files that haven't changed? (And likewise, how do I prevent restore from trying to restore files under /mnt/usr that have not changed? Or is that answer to that question that I simply have to do the first thing, i.e. force dump not to dump any of the unchanged files?) 3) Assuming that I have a second disk to which I plan (now) to use dumprestore to make periodic copies of all of my ``normal'' (non-backup) filesystems onto. Assuming also that I've already installed the FreeBSD boot loader into the boot block on this second disk, and that I've already run fdisk bsdlabel on it so as to set up all of the partitioning to be essentially identical to my ``main'' system disk. Assuming that all of this is the case, by using dumprestore as you have shown in your document under the heading Copying Filesystems, will I have succeded at doing what I was first asking about in this thread, i.e. will I have successfully created a _bootable_ mirror of my main system disk... something that I can just plug in and go with, when and if disaster strikes and my main system disk suffers a horrifying head crash, you know, the day after the manufacturer's warranty period is up? Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Sun, 10 Jun 2012, Ronald F. Guilmette wrote: (I got the Wrong Impression, I think, because I have read assertions like ...dump backs up at the filesystem block level What does that mean exactly? Use of the term block level in this context makes me think of something operating along the lines of dd.) Rather than reading just the contents of files, dump operates at a lower level, backing up all of the blocks used by a filesystem. 1) In your example under the heading Copying Filesystems, the second shell command line shown is: dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt restore -ruf -) Is that correct? Shouldn't it actually be this instead? dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt/usr restore -ruf -) I mean if the goal is to create a whole backup under /mnt that looks just like the whole original system, then shouldn't ``restoration'' of the /usr part of the original system go into /mnt/usr? The mount command above that is mount /dev/da0s1f /mnt The f indicates it's a split filesystem layout, where /, /var, /tmp, and /usr are separate filesystems in different partitions. That's a traditional layout in FreeBSD. bsdinstall in FreeBSD 9 creates a single filesystem, / . Restoring a dump of a /usr filesystem would go to /mnt/usr in that case. (That would mean you are converting from the split layout to a single filesystem, because with a combined filesystem you can't easily back up just /usr with dump. dump does not cross filesystems when making a backup.) 2) Towards the end of your document you mention rsync. Assuming that I have made a backup of my entire /usr partition (using dumprestore) into /mnt/usr and that at midnight every night from now on I want to simply refresh that and bring it up to date with the current contents of my actual /usr partition, what is the most proper way to do this? Should I use rsync for that? Or should I use dumprestore again? If that latter, then how exactly does that work? I mean if I do the pipeline from dump to restore as you have shown in your examples in your Copying Filesystems section, then what must I do in order prevent dump from dumping files that haven't changed? (And likewise, how do I prevent restore from trying to restore files under /mnt/usr that have not changed? Or is that answer to that question that I simply have to do the first thing, i.e. force dump not to dump any of the unchanged files?) See the dump man page about dump levels. I have not tried a dump level higher than zero with a dump|restore copy, but expect it to work. rsync will do it, and would be my choice for that; start with -aH for options, and --exclude might be used for some directories that don't need to be copied. Also see the sysutils/rsnapshot port for an interesting snapshot use. Combined with a bootable FreeBSD like mfsBSD (http://mfsbsd.vx.sk/), that might be a workable alternative to creating a bootable backup. 3) Assuming that I have a second disk to which I plan (now) to use dumprestore to make periodic copies of all of my ``normal'' (non-backup) filesystems onto. Assuming also that I've already installed the FreeBSD boot loader into the boot block on this second disk, and that I've already run fdisk bsdlabel on it so as to set up all of the partitioning to be essentially identical to my ``main'' system disk. Assuming that all of this is the case, by using dumprestore as you have shown in your document under the heading Copying Filesystems, will I have succeded at doing what I was first asking about in this thread, i.e. will I have successfully created a _bootable_ mirror of my main system disk... something that I can just plug in and go with, when and if disaster strikes and my main system disk suffers a horrifying head crash, you know, the day after the manufacturer's warranty period is up? With some tuning, yes. /etc/fstab mountpoints often won't match when such a drive is connected to a different port or new system. Use gpart(8) GPT labels or tunefs(8) filesystem labels. Likewise with the Ethernet board, so see rc.conf(5) about ifconfig_DEFAULT. gpart and GPT are easier than fdisk and bsdlabel. I really need to flip that disk setup article upside down, with gpart first as the new goodness. See also: http://www.wonkity.com/~wblock/docs/html/labels.html http://www.wonkity.com/~wblock/docs/html/disksetup.html ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
In message alpine.bsf.2.00.1206100543280.75...@wonkity.com, Warren Block wbl...@wonkity.com wrote: On Sun, 10 Jun 2012, Ronald F. Guilmette wrote: 1) In your example under the heading Copying Filesystems, the second shell command line shown is: dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt restore -ruf -) Is that correct? Shouldn't it actually be this instead? dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt/usr restore -ruf -) ... The mount command above that is mount /dev/da0s1f /mnt Ah! OK. I failed to notice that. Yes, that clarifies things entirely. ... I mean if I do the pipeline from dump to restore as you have shown in your examples in your Copying Filesystems section, then what must I do in order prevent dump from dumping files that haven't changed? (And likewise, how do I prevent restore from trying to restore files under /mnt/usr that have not changed? Or is that answer to that question that I simply have to do the first thing, i.e. force dump not to dump any of the unchanged files?) See the dump man page about dump levels. OK, I'm looking at it, and to be honest, the subject of dump levels is not covered at all well there. (Neither does it seem to be covered well in other online documents, found via Google, that purport to describe dump restore.) The reference to modified Tower of Hanoi algorithm and the subsequent suggested dumplevel sequence are especially opaque and perplexing. Well, nevermind about that. I get the general idea, i.e. that dumping at level N causes dumping of everything that has changed since the last dump at level N-1. What I don't understand (and what I wish someone would enlighten me about) is just this: It would seem that in order to implement these dump levels, dump must be keeping a record somewhere, for each file in the filesystem, of the level at which that file was last dumped. But where is this infor- mation stored, exactly?? I won't be able to sleep until I know. (Aside: One of the reasons I hate Windoze is that everything is hidden. One of the reasons I love UNIX is that everything is out in the open, which is to say documented. But even after reading the dump(1) man page, it ain't in the least bit obvious to me where dump is recording the last dump level for each node in the filesystem. This info must be stored SOMEWHERE. But where?) I have not tried a dump level higher than zero with a dump|restore copy, but expect it to work. rsync will do it, and would be my choice for that; start with -aH for options, and --exclude might be used for some directories that don't need to be copied. Humm... well, I am reminded of all of the reasons why I have been convinced (now) to use dump/restore, specifically the security of knowing that _all_ info will be copied across, in particular (a) extra mode bits/flags and (b) ACLs and (c) file attributes. (This is not to even mention correct handling of other obscure filesystem things like, e.g. sparse files.) So the obvious question is: If I am going to use rsync to keep my backup disk up-to-date, does rsync handle all of these additional small but important filesystem details properly too? If not, then I had best just stick with dump/restore, and use the dumplevels. Also see the sysutils/rsnapshot port for an interesting snapshot use. Combined with a bootable FreeBSD like mfsBSD (http://mfsbsd.vx.sk/), that might be a workable alternative to creating a bootable backup. Thanks. I've installed that port and I am looking at the man page, but I believe that I do still want to reach my goal of just having a (recovery) hard disk that I can just boot directly from. 3) Assuming that I have a second disk to which I plan (now) to use dumprestore to make periodic copies of all of my ``normal'' (non-backup) filesystems onto. Assuming also that I've already installed the FreeBSD boot loader into the boot block on this second disk, and that I've already run fdisk bsdlabel on it so as to set up all of the partitioning to be essentially identical to my ``main'' system disk. Assuming that all of this is the case, by using dumprestore as you have shown in your document under the heading Copying Filesystems, will I have succeded at doing what I was first asking about in this thread, i.e. will I have successfully created a _bootable_ mirror of my main system disk... With some tuning, yes. /etc/fstab mountpoints often won't match when such a drive is connected to a different port or new system. Right. There's that. So I will need to diddle the /etc/fstab in the backup. Use gpart(8) GPT labels... Ummm... I don't know pooh about this, but I am reading about it now, and the idea make me _very_ nervous. Apparently, `MBR' is traditional, and I am very much of a traditionalist. (Mostly I find that I only get into trouble when I try to buck pre-existing conventions.) or tunefs(8) filesystem labels. I guess you must mean the -L volname option. I'm not at all sure
Re: Making a bootable backup (hard)disk... how?
Warren? Just a couple more quick questions. You recommend: dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt restore -ruf -) I'm real curious about you suggestions for the -C and -b values. I have what amounts to a personal workstation. Yea, OK, it is running mail, web, and FTP servers also, but fundementailly, it is not that busy most of the time. And it's got 4GB of main installed. On average, I suspect that I ain't even using half of that. Given all that, why shouldn't I specify (e.g.): -C512 -b1024 ? Wouldn't that all make the dump go faster? Remember, I'm just gonna do like in your examples and pipe the output of dump to restore, so whereas if I was going to tape (or even to a .dump disk file), -b1024 would be rather wasteful, in my case all I am wasting is a bit of kernel time (for a bit more stuff being passed through the pipe). Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Sun, 10 Jun 2012, Ronald F. Guilmette wrote: In message alpine.bsf.2.00.1206100543280.75...@wonkity.com, Warren Block wbl...@wonkity.com wrote: On Sun, 10 Jun 2012, Ronald F. Guilmette wrote: 1) In your example under the heading Copying Filesystems, the second shell command line shown is: dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt restore -ruf -) Is that correct? Shouldn't it actually be this instead? dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt/usr restore -ruf -) ... The mount command above that is mount /dev/da0s1f /mnt Ah! OK. I failed to notice that. Yes, that clarifies things entirely. ... I mean if I do the pipeline from dump to restore as you have shown in your examples in your Copying Filesystems section, then what must I do in order prevent dump from dumping files that haven't changed? (And likewise, how do I prevent restore from trying to restore files under /mnt/usr that have not changed? Or is that answer to that question that I simply have to do the first thing, i.e. force dump not to dump any of the unchanged files?) See the dump man page about dump levels. OK, I'm looking at it, and to be honest, the subject of dump levels is not covered at all well there. (Neither does it seem to be covered well in other online documents, found via Google, that purport to describe dump restore.) The reference to modified Tower of Hanoi algorithm and the subsequent suggested dumplevel sequence are especially opaque and perplexing. It's diffs. One complete copy to start, then a partial dump, just the files that have changed since the first. And then another partial, just the files that have changed since the second. Start over with a full dump every so often, because that reduces the amount of backups that have to be restored until you get to the final version of the file. As far as I can recall, I've always used level zero. What I don't understand (and what I wish someone would enlighten me about) is just this: It would seem that in order to implement these dump levels, dump must be keeping a record somewhere, for each file in the filesystem, of the level at which that file was last dumped. But where is this infor- mation stored, exactly?? I won't be able to sleep until I know. (Aside: One of the reasons I hate Windoze is that everything is hidden. One of the reasons I love UNIX is that everything is out in the open, which is to say documented. But even after reading the dump(1) man page, it ain't in the least bit obvious to me where dump is recording the last dump level for each node in the filesystem. This info must be stored SOMEWHERE. But where?) Sorry, I've never checked. File dates, maybe. The source for dump would be the place to look. rsync will do it, and would be my choice for that; start with -aH for options, and --exclude might be used for some directories that don't need to be copied. Humm... well, I am reminded of all of the reasons why I have been convinced (now) to use dump/restore, specifically the security of knowing that _all_ info will be copied across, in particular (a) extra mode bits/flags and (b) ACLs and (c) file attributes. (This is not to even mention correct handling of other obscure filesystem things like, e.g. sparse files.) So the obvious question is: If I am going to use rsync to keep my backup disk up-to-date, does rsync handle all of these additional small but important filesystem details properly too? If not, then I had best just stick with dump/restore, and use the dumplevels. It does many of those things, enough that many people use it for backup. Right. There's that. So I will need to diddle the /etc/fstab in the backup. Use gpart(8) GPT labels... Ummm... I don't know pooh about this, but I am reading about it now, and the idea make me _very_ nervous. Apparently, `MBR' is traditional, and I am very much of a traditionalist. (Mostly I find that I only get into trouble when I try to buck pre-existing conventions.) or tunefs(8) filesystem labels. I guess you must mean the -L volname option. I'm not at all sure that I see how that would help anything. When a mountpoint is /dev/gpt/myrootfs, it doesn't matter whether the drive shows up as ada0 or ad4 or da2, the mount still works. Likewise with the Ethernet board, so see rc.conf(5) about ifconfig_DEFAULT. Something else (the Handbook?) mentioned something about the ethernet card(s) in the context also. I just don't get it. Why would I even need to think about that in this context? My current system disk is alive well and already has all of the proper ifconfig stuff in /etc/rc.conf to make the local ethernet card work properly on my network. I am engaged in making a _verbatim_ backup of my main system disk to another hard disk that, in an emergency, would be plugged in to the exact same system (with the exact same ethernet card) and if I have made the verbatim backup properly, then it also will have the proper
Re: Making a bootable backup (hard)disk... how?
On Sun, 10 Jun 2012, Ronald F. Guilmette wrote: What I don't understand (and what I wish someone would enlighten me about) is just this: It would seem that in order to implement these dump levels, dump must be keeping a record somewhere, for each file in the filesystem, of the level at which that file was last dumped. But where is this infor- mation stored, exactly?? I won't be able to sleep until I know. Only the dates of the levels of backup are stored, in /etc/dumpdates. Then the fact that a file has been dumped is inferred by comparing the file's last mod date with the dates in /etc/dumpdates. See the -T and -u options of the dump man page where this is implied but perhaps not actually stated. It does occur to me that /etc is not a felicitous place to keep this information, but given the desirability of dumping filesystems in read only state, placing the dump dates in the filesystem itself isn't feasible. daniel feenberg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
Ronald F. Guilmette r...@tristatelogic.com wrote: Warren Block wbl...@wonkity.com wrote: On Sun, 10 Jun 2012, Ronald F. Guilmette wrote: ... I mean if I do the pipeline from dump to restore as you have shown in your examples in your Copying Filesystems section, then what must I do in order prevent dump from dumping files that haven't changed? (And likewise, how do I prevent restore from trying to restore files under /mnt/usr that have not changed? Or is that answer to that question that I simply have to do the first thing, i.e. force dump not to dump any of the unchanged files?) See the dump man page about dump levels. OK, I'm looking at it, and to be honest, the subject of dump levels is not covered at all well there. (Neither does it seem to be covered well in other online documents, found via Google, that purport to describe dump restore.) The reference to modified Tower of Hanoi algorithm and the subsequent suggested dumplevel sequence are especially opaque and perplexing. Well, nevermind about that. I get the general idea, i.e. that dumping at level N causes dumping of everything that has changed since the last dump at level N-1. That's about all -really- need to know. The sequencing stuff is just a recommendation on a way to do multiple levels of incremental backups. What I don't understand (and what I wish someone would enlighten me about) is just this: It would seem that in order to implement these dump levels, dump must be keeping a record somewhere, for each file in the filesystem, of the level at which that file was last dumped. But where is this infor- mation stored, exactly?? I won't be able to sleep until I know. First, your assumption is incorrect. *grin* All that dump stores is the timestamp of the completion of the last dump at each level. As for where that information is stored. RTFM applies. the 'files' section. By default '/etc/dumpdates', but modifiable by the -D option. (Aside: One of the reasons I hate Windoze is that everything is hidden. One of the reasons I love UNIX is that everything is out in the open, which is to say documented. But even after reading the dump(1) man page, it ain't in the least bit obvious to me where dump is recording the last dump level for each node in the filesystem. This info must be stored SOMEWHERE. But where?) Wrong assumptions lead to looking for the wrong things. grin Why store the date for every node, when storing the date of the dump sufficies? (assuming you're consisently using the same dump command for dumps recorded to the same 'dumpfile', that is, and using different dumpfiles for different 'selective' dump commands) _Anything_ modified after the last dump at the next lower level needs to be saved as part of -this- dump, unless it is 'excluded' from this dump. A level 0 dump backs up 'everything', except things which are expliclitly excluded. A level 1 dump backs up 'everything that has changed since the last level 0 dump' (whether or not it was inclued in the level 0 dump), except for things which are eplicitly excluded. A subsequent level 1 dump catches everything since the last level _0_ dump. A level 2 dump backs up 'everything that has changed since the last level 1 dump' (whether or not it was inclued in the level 1 dump, or level 0 dump), except for things which are eplicitly excluded. A level 3 dump backs up 'everything that has changed since the last level 2 dump' (whether or not it was inclued in the level 2, 1, or 0 dump), except for things which are eplicitly excluded. A subsequent level 2 dump catches everything since the last level _1_ dump. Another level 1 dump -here- catches everything singe the prior level 0 dump. and kills the record of any level 2 (or higher) dump. A level 3 dump here -- with dump anything since the level 1 dump (since there was no 'current' level 2 dump). So the obvious question is: If I am going to use rsync to keep my backup disk up-to-date, does rsync handle all of these additional small but important filesystem details properly too? If not, then I had best just stick with dump/restore, and use the dumplevels. It doesn't matter what anybody tells you. what matters is what the machine does. Thus the authoritative response is try it and find out. grin I was first asking about in this thread, i.e. will I have successfully created a _bootable_ mirror of my main system disk... With some tuning, yes. /etc/fstab mountpoints often won't match when such a drive is connected to a different port or new system. Right. There's that. So I will need to diddle the /etc/fstab in the backup. 'dump'/'restore' doesn't do -everything- necessary. You have to: set up slices/partitions as needed. 'label' as appropriate INSTALL BOOT BLOCKS 'newfs' the appropriate filesystems then dump/restore the filesystems on the disk. Likewise with the Ethernet board, so see rc.conf(5) about
Re: Making a bootable backup (hard)disk... how?
Ronald F. Guilmette writes: Warren? Just a couple more quick questions. You recommend: dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt restore -ruf -) I'm real curious about you suggestions for the -C and -b values. I have what amounts to a personal workstation. Yea, OK, it is running mail, web, and FTP servers also, but fundementailly, it is not that busy most of the time. And it's got 4GB of main installed. On average, I suspect that I ain't even using half of that. Given all that, why shouldn't I specify (e.g.): -C512 -b1024 ? Wouldn't that all make the dump go faster? There are many possible obstacles to faster dump speed; enumerating them is left as an exercise for the reader. As it happens, I have a set-up very similar to what you describe ... except with 8g of memory. A few years ago I did some testing with various cache sizes (as part of diagnosing other problems) and ended up with C=32. (I should probably run the tests again, given some hardware changes since.) That machine's level 0 runs tonight; I will try to remember to (retain and) post the results. Robert Huff ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Sun, 10 Jun 2012, Robert Huff wrote: Ronald F. Guilmette writes: Warren? Just a couple more quick questions. You recommend: dump -C16 -b64 -0uanL -h0 -f - /usr | (cd /mnt restore -ruf -) I'm real curious about you suggestions for the -C and -b values. I have what amounts to a personal workstation. Yea, OK, it is running mail, web, and FTP servers also, but fundementailly, it is not that busy most of the time. And it's got 4GB of main installed. On average, I suspect that I ain't even using half of that. Given all that, why shouldn't I specify (e.g.): -C512 -b1024 ? Wouldn't that all make the dump go faster? There are many possible obstacles to faster dump speed; enumerating them is left as an exercise for the reader. As it happens, I have a set-up very similar to what you describe ... except with 8g of memory. A few years ago I did some testing with various cache sizes (as part of diagnosing other problems) and ended up with C=32. (I should probably run the tests again, given some hardware changes since.) That machine's level 0 runs tonight; I will try to remember to (retain and) post the results. -C16 is a little small because it's hard to tell how much memory someone running the example will have. I usually use -C32 on machines with plenty of memory. Remember that dump forks multiple times, so it's not 32M, but N*32M. -b64 is the largest safe buffer size to use. People have experienced problems with larger amounts. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On 06/09/12 00:58, Warren Block wrote: On Fri, 8 Jun 2012, Robert Huff wrote: Ronald F. Guilmette writes: I got a lot of disks here, so that part is not a problem. I just need to make sure that I'm gonna do this the Right Way[tm]. (I've already been making my own ham-fisted disk-to-disk backups in the past, but I'm sure that the way I have been doing that is sub-optimal, so I'm here seeking knowledge of how to do this the Right Way.) The bottom line is this... I know how to use cpio, and would like to use it to create a complete and _bootable_ backup of my main system disk. (My main system disk has only one BIOS partition, and that is sub-divided into the usual set of FreeBSD partitions, you know, /, /dev, /tmp, /usr, /var, /usr/compat/linux/proc, and /var/named/dev.) As far as I know, the only way guaranteed to preserve metadata is dump/restore. See previous (not necessarily recent) discussion (on this list, and possibly in the Handbook) for more information. The rsync port has a flags option. I haven't tried it for a full backup. Even if it can copy all filesystem attributes like dump, there are still non-filesystem things needed for booting that neither can copy, like partition tables and boot blocks. There's a BFI (brute force and ignorance) way of doing it in the base system - dd. Provided your system disk is quiescent (ideally when running from a live CD or all partitions mounted read-only, otherwise pray to the deity of your choice) and the backup disk is a) at least as large as the system disk, and b) has the same sector size, then a simple dd from the system disk to the backup should work. [Greybeard war story:] Going back about 25-30 years, a friend of mine was responsible for running the Unix systems in an EE department of a UK university. He used to back up the disks (probably around 10 MB in those days) to 1/2 tape every night. Eventually he got sick of undergraduates asking him to restore files they'd accidentally deleted, and hit upon the idea of dd'ing the disk to tape, and then if a student wanted a restore, he'd mount the tape as a r/o file system (these were the days of tape block devices and trusting all users) and tell them to use restore to get the files themselves. It took forever, but it was the student's time that was wasted, not his. Most of them learned not to delete wanted files after one or two times of doing this. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
In message 4fd38b9a.4010...@qeng-ho.org, Arthur Chance free...@qeng-ho.org wrote: There's a BFI (brute force and ignorance) way of doing it in the base system - dd. Provided your system disk is quiescent (ideally when running from a live CD or all partitions mounted read-only, otherwise pray to the deity of your choice) and the backup disk is a) at least as large as the system disk, and b) has the same sector size, then a simple dd from the system disk to the backup should work. Thank you Arthur, and yes, trying to back up a partition that's currently mounted r/w using dd will almost certainly not produce the desired results. And of course, as you correctly note also, the target partition had best be at least as big as the source (and perhaps even identical in size). I don't care to take own my system to make backups... and don't believe that I should have to do so, and thus, this is one of the reasons why I would prefer to use something like cpio. Also, I don't like backups taking longer than absolutely necessary, and this is why I am specifically _not_ attracted to either the dd solution or to dump/restore, because as I understand it, with either of these methods you end up copying perhaps a metric buttload worth of unallocated free disk space. (I would prefer not to do that. It just seems wasteful... of time, if nothing else.) Also, in one case, one of my partitions has one directory that contains a really massive amount of stuff, and I specifically _don't_ need any of that particular stuff (in that one directory) backed up. So again, I'm looking at tar or cpio or perhaps pax. (Of course cpio is more full-featured than tar, and I don't really know anything about pax, so that leaves me with cpio.) [Greybeard war story:] Going back about 25-30 years, a friend of mine was responsible for running the Unix systems in an EE department of a UK university. He used to back up the disks (probably around 10 MB in those days) to 1/2 tape every night. Eventually he got sick of undergraduates asking him to restore files they'd accidentally deleted, and hit upon the idea of dd'ing the disk to tape, and then if a student wanted a restore, he'd mount the tape as a r/o file system (these were the days of tape block devices and trusting all users) and tell them to use restore to get the files themselves. It took forever, but it was the student's time that was wasted, not his. Most of them learned not to delete wanted files after one or two times of doing this. My own greybeard story: Back in the late 70s I was in college and had just switched major to CS. My college had a mainframe, but also a PDP-11 to which were connected a number of ASR-33s (Who could ever forget THOSE!) Students who asked could get an account on the PDP-11/RSTS-E system. I did, and started experimenting. I remember one day I really pissed off the specific Computer Center staff member who was assigned to the care feeding of the RSTS-E system when her backup tape started whizzing back and forth for no apparent reason. At least it seemed that way to her, since she had only just mounted the tape and _she_ had not issued any console commands yet which would have caused the tape drive to do anything. :-) Needless to say RSTS-E was, apparently, not quite as sophisticated at that time as, say, UNIX, in terms of protecting physical I/O devices from general Joe-Schmo users. I never became friends with that specific Computer Center staff member, but I did end up as the only CS undergraduate who got a job in the campus Computer Center. Sorry. You talking about tapes just reminded of that incident, way back when. (Sigh. The good old daze when men were men, and the bits ran scared!) Moral of the story is that sometimes it actually does pay to be a smartass. Regards, rfg P.S. It really is a Damn Shame[tm] that nobody ever hacked FreeBSD cpio to make it be able to copy (a) the extra file flag/mode bits and/or (b) file ACLs and/or (c) file attributes. I guess I have found a new project for myself... when/if I ever find some free time. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Sat, 9 Jun 2012, Ronald F. Guilmette wrote: Also, I don't like backups taking longer than absolutely necessary, and this is why I am specifically _not_ attracted to either the dd solution or to dump/restore, because as I understand it, with either of these methods you end up copying perhaps a metric buttload worth of unallocated free disk space. No, that is one of the biggest advantages of dump over dd. dump knows UFS, and only copies occupied sectors and needed information. That's why it can be restored file-by-file or to a partition of a different size. Also, in one case, one of my partitions has one directory that contains a really massive amount of stuff, and I specifically _don't_ need any of that particular stuff (in that one directory) backed up. So again, I'm looking at tar or cpio or perhaps pax. (Of course cpio is more full-featured than tar, and I don't really know anything about pax, so that leaves me with cpio.) Directories and files can be skipped with the nodump flag. P.S. It really is a Damn Shame[tm] that nobody ever hacked FreeBSD cpio to make it be able to copy (a) the extra file flag/mode bits and/or (b) file ACLs and/or (c) file attributes. A quick search shows dump should support ACLs. The other stuff is also there. Try it. http://www.wonkity.com/~wblock/docs/html/backup.html ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Fri, 8 Jun 2012, Robert Huff wrote: Ronald F. Guilmette writes: I got a lot of disks here, so that part is not a problem. I just need to make sure that I'm gonna do this the Right Way[tm]. (I've already been making my own ham-fisted disk-to-disk backups in the past, but I'm sure that the way I have been doing that is sub-optimal, so I'm here seeking knowledge of how to do this the Right Way.) The bottom line is this... I know how to use cpio, and would like to use it to create a complete and _bootable_ backup of my main system disk. (My main system disk has only one BIOS partition, and that is sub-divided into the usual set of FreeBSD partitions, you know, /, /dev, /tmp, /usr, /var, /usr/compat/linux/proc, and /var/named/dev.) As far as I know, the only way guaranteed to preserve metadata is dump/restore. See previous (not necessarily recent) discussion (on this list, and possibly in the Handbook) for more information. The rsync port has a flags option. I haven't tried it for a full backup. Even if it can copy all filesystem attributes like dump, there are still non-filesystem things needed for booting that neither can copy, like partition tables and boot blocks. There might be something for a bootable backup in ports, or it could be done with a script using gpart. Set up the disk with GPT labels to make it relocatable, use dump to copy everything the first time, fix up a few things like fstab and ifconfig_DEFAULT in the same script. Afterwards, rsync may be enough for fast updates. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Making a bootable backup (hard)disk... how?
On Fri, Jun 8, 2012 at 4:58 PM, Warren Block wbl...@wonkity.com wrote: On Fri, 8 Jun 2012, Robert Huff wrote: Ronald F. Guilmette writes: I got a lot of disks here, so that part is not a problem. I just need to make sure that I'm gonna do this the Right Way[tm]. (I've already been making my own ham-fisted disk-to-disk backups in the past, but I'm sure that the way I have been doing that is sub-optimal, so I'm here seeking knowledge of how to do this the Right Way.) The bottom line is this... I know how to use cpio, and would like to use it to create a complete and _bootable_ backup of my main system disk. (My main system disk has only one BIOS partition, and that is sub-divided into the usual set of FreeBSD partitions, you know, /, /dev, /tmp, /usr, /var, /usr/compat/linux/proc, and /var/named/dev.) As far as I know, the only way guaranteed to preserve metadata is dump/restore. See previous (not necessarily recent) discussion (on this list, and possibly in the Handbook) for more information. The rsync port has a flags option. I haven't tried it for a full backup. Even if it can copy all filesystem attributes like dump, there are still non-filesystem things needed for booting that neither can copy, like partition tables and boot blocks. There might be something for a bootable backup in ports, or it could be done with a script using gpart. Set up the disk with GPT labels to make it relocatable, use dump to copy everything the first time, fix up a few things like fstab and ifconfig_DEFAULT in the same script. Afterwards, rsync may be enough for fast updates. I am applying the following technique in Linux ( because I am not using FreeBSD ) : On the same computer : (1) I am disconnecting power cable of the existing HDD . (2) I am connecting a new disk ( by attaching Channel Zero SATA Cable to it ) . (3) I am installing a new operating system on HDD (4) I am installing mc ( Midnight Commander ) . (5) After obtaining a working new HDD , I am connecting the power cable of the previous disk . (6) I am booting the computer . The new operating system is seeing previous HDD . (7) I am mounting previous HDD . (8) By using mc , I am copying required files/directories into new HDD . Mostly I am using KDE4 . Dolphin in KDE4 has a memory leak which is causing disaster for very large list of files because for each file , it is consuming irrecoverable memory loss from 600 bytes to 1000 bytes during copying . (9) I am keeping previous HDD as bootable by disconnecting its power cable because it is not necessary to keep it powered . You can do the following : (1) Install your operating system on back-up HDD . (2) By using a script , copy required files /directories onto back-up HDD . With this technique , you can maintain a revolving back-up HDD units : Create back-up HDD 1 . Create back-up HDD 2 . . . . Create back-up HDD n . When n-th HDD is created , and a new back-up is required switch to HDD 1 . Using rsync you may update only required files / directories . Full copy is not required to update subsequent HDD units because it is very likely that many files are the same . In that form , EACH back-up HDD is bootable when it is required . You may consider HAST for a single HDD . You may use external USB HDD by selecting a robust model . Disconnecting power cable of existing HDD during a new operating system install on another HDD is preventing an accidental loss of data because naming the recognized HDD units is NOT standard and very unreliable . Thank you very much . Mehmet Erol Sanliturk ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
