On Tuesday 27 May 2014 11:28:17 Rich Freeman wrote: > On Tue, May 27, 2014 at 11:12 AM, J. Roeleveld <jo...@antarean.org> wrote: > > On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote: > >> btrfs wouldn't have any issues with this at all. You'd have an > >> advantage in that you wouldn't have to unmount the filesystem to > >> cleanly create the snapshot (which you have to do with lvm). > > > > That, or a "sync" prior to creating the snapshot. :) > > If the filesystem is still mounted, I'm not sure that a sync is > guaranteed to give you a clean remount. It only flushes the > caches/etc. You need to remount read-only or unmount before doing the > sync (and the sync probably isn't actually necessary as I'd think LVM > would snapshot the contents of the cache as well).
I do this for the OS-partitions of my VMs. In the VM, run "sync", then on the host, take an LVM snapshot and then mount the snapshot on the host. I have not had any errors from this. > > I have a yearly (full), monthly, weekly and daily. Each incremental is > > against the most recent one of itself or longer period. > > That means having to keep multiple snapshots active, which I prefer to > > avoid. > You only need to store snapshots for use with incremental backups. > So, if all your backups are full, then you don't need to retain any > snapshots (and you wouldn't use btrfs send anyway). If your yearly is > full and your monthlies are incremental against the yearly then you > need to keep your yearly snapshot for a year. If your yearly is full > and your monthlies are incremental against the last month, then you > only need to keep the yearly until the next monthly. If your > monthlies are full then you only need to keep the current monthly > assuming your dailies are incremental against those, but if they're > incremental from the last daily then you never need to keep anything > for more than a day. That makes for an interesting option. Not sure if I would implement it that way. > > But, it is a good idea for backing up desktops and laptops. > > It is really intended more for something like datacenter replication. > Snapshot every 5 min, send the data to the backup datacenter, delete > the snapshots upon confirmation of successful receipt. In such a > scenario you wouldn't retain the sent files but just keep playing them > against the replicate filesystem. > > They'd be fine for backups as well, as long as you can store the > snapshots online until no longer needed for incrementals. "app-backup/dar" uses catalogues for the incrementals. I think I will stick to that for the foreseeable future. > >> But, you can always just create a snapshot, write it to backup with > >> your favorite tool (it is just a directory tree), and then remove it > >> as soon as you're done with it. Creating a snapshot is atomic at the > >> filesystem level, though again if you want application level > >> consistency you need to deal with that until somebody comes up with a > >> transactional way to store files on Linux that is more elegant that > >> fsyncing on every write. > > > > That would require a method to keep database and filesystem perfectly in > > sync when they are not necessarily on the same machine. > > Well, right now we can't even guarantee consistency when everything is > written by a single process on the same machine. The best we have is > a clunky fsync operation which kills the write cache and destroys > performance, and even that doesn't do anything if you have more than > one file that must be consistent. Yep, and that's why those filesystems are actually umounted prior to creating the LVM snapshot. Umounting forces the filesystem to be put into a consistent state. > The result is journals on top of journals as nobody can trust the next > layer down to do its job correctly. > > Going across machines does complicate things further as there are more > failure modes that take out one part of the overall system but not > another. However, I'd like to think that an OS that natively supports > transactions could at least standardize things so that every layer > along the path isn't storing its own journal. > > In fact, many of the optimizations possible with zfs and btrfs are due > to the fact that you eliminate all those layers. Either of those 2, probably btrfs as I prefer native support in the kernel, will be implemented when I get the opportunity to get the NAS on bare metal and remove the virtualization for that component. I need to find a different host for the other services first. That might take a while. -- Joost
