Re: [linux-lvm] faster snapshot creation?
On Sat, 22 Feb 2020, Eric Toombs wrote: Snapshot creation is already pretty fast: $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template Logical volume "snap" created. 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps That's about half a second in real time. But I have a scenario that would benefit from it being even faster. I'm doing many small unit tests So, is there a sort of "dumber" way of making these snapshots, maybe by changing the allocation algorithm or something? How about using a filesystem that supports snapshot, e.g. nilfs, or (I think) btrfs? That would be much faster than doing it at the LVM level, which has to sync metadata and stuff. a) load your template into work directory b) tag snapshot c) run test (possibly in container) d) restore tagged snapshot e) goto c ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] faster snapshot creation?
Dne 22. 02. 20 v 12:58 Eric Toombs napsal(a): Snapshot creation is already pretty fast: $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template Logical volume "snap" created. 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata 28916maxresident)k 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps That's about half a second in real time. But I have a scenario that would benefit from it being even faster. I'm doing many small unit tests starting from a template filesystem. I do the snapshot, run the unit test on the snapshot, then delete the snapshot afterwards using lvremove. Each unit test, though, takes much less than a second to run (often on the order of 10ms), so most of the time is being spent making these snapshots. So, is there a sort of "dumber" way of making these snapshots, maybe by changing the allocation algorithm or something? Hi IMHO - what takes most of the time are these couple things: Each command has 'non-trivial' time overhead on its startup (scanning you system with devices and validating everything) For old snapshots - COW are needs to be 'created' & 'zeroed' as separate LV. Then you need to 'flush' all existing IO on origin device (so it's in the consistent states - i.e. the filesystem synchronizes all it's content in its metadata) - this all takes some measurable amount of time. You can 'prepare' empty zeroed LV ahead of time and then use 'lvconvert' to attach snapshot (with -Zn) - this should speed-up attachment of snapshot. For the 'second' point you could likely issue 'sync' ahead of time so most buffers will be flushed (if there is no big IO traffic). Saying all this - why are you using old snapshot when you are targeting for performance ?? You really should consider usage of thin-pool - where you could chain a long series of snapshot without having a dramatic performance degradation of the whole IO throughput - old snapshot are really meant to be used only if you want to take i.e. backup of a filesystem and you need some 'consistent' point in time - for everything else you should be using thin-pools nowdays... Regards Zdenek ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] faster snapshot creation?
Il 2020-02-22 12:58 Eric Toombs ha scritto: So, is there a sort of "dumber" way of making these snapshots, maybe by changing the allocation algorithm or something? Hi, I think that total snapshot creation time is dominated by LVM flushing its (meta)data to the physical disks. Two things to try: - use thin lvm snapshot (see man lvmthin); - create a pv/vg/lv from a memory-backed loop device (ie: truncate --size=1G /dev/shm/test.img; losetup -f /dev/shm/test.img; pvcreate /dev/loop0 and so on...) -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: g.da...@assyoma.it - i...@assyoma.it GPG public key ID: FF5F32A8 ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
[linux-lvm] faster snapshot creation?
Snapshot creation is already pretty fast: > $ time sudo lvcreate --size 512M --snapshot --name snap /dev/testdbs/template > Logical volume "snap" created. > 0.03user 0.05system 0:00.46elapsed 18%CPU (0avgtext+0avgdata > 28916maxresident)k > 768inputs+9828outputs (0major+6315minor)pagefaults 0swaps That's about half a second in real time. But I have a scenario that would benefit from it being even faster. I'm doing many small unit tests starting from a template filesystem. I do the snapshot, run the unit test on the snapshot, then delete the snapshot afterwards using lvremove. Each unit test, though, takes much less than a second to run (often on the order of 10ms), so most of the time is being spent making these snapshots. So, is there a sort of "dumber" way of making these snapshots, maybe by changing the allocation algorithm or something? ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
