Re: Cycle of send/receive for backup/restore is incomplete...

2014-04-24 Thread Robert White

I am, then, quite confused...

On 04/23/2014 09:28 PM, Brendan Hide wrote:

Replied inline:
btrfs doesn't differentiate snapshots and subvolumes. They're the same
first-class citizen. A snapshot is a subvolume that just happens to have
some data (automagically/naturally) deduplicated with another subvolume.


What then is the -s for in

btrfs subvolume list -s mount point

Clearly some information of bias and status exists or the option and its 
functional behavior wouldn't exist.


I understand the use of -p for doing diffs, but here's the thing...

mount /dev/sda /drivea
mount /dev/sdb /driveb

btrfs subvolume create /drivea/Base
btrfs subvolume snapshot -r /drivea/Base /drivea/Base_BACKUP

btrfs subvolume list -a /drivea
[blah blah blah] Base
[blah blah blah] Base_BACKUP

* skipping the network and delta-T nonsense as irrelevant
btrfs send /drivea/Base_BACKUP | btrfs receive /driveb/

*No way to make /driveb/Base_BACKUP _not_ end up read only
*So make a writeable snapshot

btrfs subvolume snapshot /driveb/Base_BACKUP /driveb/Base

btrfs subvolume list -a /driveb
[blah blah blah] Base_BACKUP
[blah blah blah] Base

 Confusing and/or problematic bit:
btrfs subvolume list -s /drivea
[blah blah blah] Base_BACKUP

btrfs subvolume list -s /driveb
[blah blah blah] Base

So if I want to, say, write a backup script that rotates through the 
subvolumes to rotor backups, the restored drive (driveb in this example) 
automatically fails. There is no apparent way to coerce the relationship 
such that both drives end up with Base being the writable base and 
Base_BACKUP be the read-only snapshot returned when doing list -s.


So the systems are the same but they aren't really the same according 
to this clearly visible symptom.


As such various automations that one might right for an original system 
could then choke and die after restoration from this backup. Either that 
or you have to use /bin/cp (or similar) and lose the backup history when 
you restore.


Its a surprise waiting to happen. It surprised me.

It's _impossible_ to strip Base of it's subvolume status on /driveb. If 
you delete the Base_BACKUP element so that Base is the only thing on the 
drive, it's still a shapshot according to -s. What does this status even 
mean if it's as meaningless as it seems.


That seems like a second surprise.

---

Is this a common case? It could easily be if you use NAS or movable USB 
to do your backups and restore-or-media-migration operations.


Are we sure it doesn't matter? I find it problematic but fixable in 
concept. I've got no information whether the internal parentage 
relationship could be reversed so that the before and after of the 
subvolume list -s result are the same.


No I'm not looking for byte-level identical status, I know that's 
ridiculous.


I want semantically identical status.

My experience with list -s says I'm not getting semantically identical 
status after the fact and I have no clear way to coerce it.




Why it matters...

If I am doing monthly and weekly archiving I don't want to interrupt the 
rolling archive(s) if I end up having to do a restore. I don't want to 
create a catastrophe point or an interrupting epoch in the archive history.


It sounds like it doesn't matter once you know not to use the -s status 
for anything...

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Cycle of send/receive for backup/restore is incomplete...

2014-04-23 Thread Brendan Hide

Replied inline:

On 2014/04/24 12:30 AM, Robert White wrote:
So the backup/restore system described using snapshots is incomplete 
because the final restore is a copy operation. As such, the act of 
restoring from the backup will require restarting the entire backup 
cycle because the copy operation will scramble the metadata 
consanguinity.


The real choice is to restore by sending the snapshot back via send 
and receive so that all the UIDs and metadata continue to match up.


But there's no way to promote the final snapshot to a non-snapshot 
subvolume identical to the one made by the original btrfs subvolume 
create operation.


btrfs doesn't differentiate snapshots and subvolumes. They're the same 
first-class citizen. A snapshot is a subvolume that just happens to have 
some data (automagically/naturally) deduplicated with another subvolume.


Consider a file system with __System as the default mount (e.g. btrfs 
subvolume create /__System). You make a snapshot (btrfs sub snap -r 
/__System /__System_BACKUP). Then you send the backup to another file 
system with send receive. Nothing new here.


The thing is, if you want to restore from that backup, you'd 
send/receive /__System_BACKUP to the new/restore drive. But that 
snapshot is _forced_ to be read only. So then your only choice is to 
make a writable snapshot called /__System. At this point you have a 
tiny problem, the three drives aren't really the same.


The __System and __System_BACKUP on the final drive are subvolumes of 
/, while on the original system / and /__System were full subvolumes.


There's no such thing as a full subvolume. Again, they're all 
first-class citizens. The real root of a btrfs is always treated as a 
subvolume, as are the subvolumes inside it too. Just because other 
subvolumes are contained therein it doesn't mean they're diminished 
somehow. You cannot have multiple subvolumes *without* having them be a 
sub volume of the real root subvolume.


It's dumb, it's a tiny difference, but it's annoying. There needs to 
be a way to promote /__System to a non-snapshot status.


If you look at the output of btrfs subvolume list -s / on the 
various drives it's not possible to end up with the exact same system 
as the original.


From a user application perspective, the system *is* identical to the 
original. That's the important part.


If you want the disk to be identical bit for bit then you want a 
different backup system entirely, one that backs up the hard disk, not 
the files/content.


On the other hand if you just want to have all your snapshots restored 
as well, that's not too difficult. Its pointless from most perspectives 
- but not difficult.


There needs to be either an option to btrfs subvolume create that 
takes a snapshot as an argument to base the new device on, or an 
option to receive that will make a read-write non-snapshot subvolume.


This feature already exists. This is a very important aspect of how 
snapshots work with send / receive and why it makes things very 
efficient. They work just as well for a restore as they do for a backup. 
The flag you are looking for is -p for parent, which you should 
already be using for the backups in the first place:


From backup host:
$ btrfs send -p /backup/path/yesterday /backup/path/last_backup | 
netcat or whatever you choose


From restored host:
$ netcat or whatever you choose | btrfs receive /tmp/btrfs_root/

Then you make the non-read-only snapshot of the restored subvolume.


[snip]




--
__
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html