On 02/25/2016 02:49 AM, Stefan Priebe - Profihost AG wrote: > > Am 22.02.2016 um 23:08 schrieb John Snow: >> >> >> On 02/22/2016 03:21 PM, Stefan Priebe wrote: >>> Hello, >>> >>> is there any chance or hack to work with a bigger cluster size for the >>> drive backup job? >>> >>> See: >>> http://git.qemu.org/?p=qemu.git;a=blob;f=block/backup.c;h=16105d40b193be9bb40346027bdf58e62b956a96;hb=98d2c6f2cd80afaa2dc10091f5e35a97c181e4f5 >>> >>> >>> This is very slow with ceph - may be due to the 64k block size. I would >>> like to check whether this is faster with cephs native block size of 4mb. >>> >>> Greets, >>> Stefan >>> >> >> It's hardcoded to 64K at the moment, but I am checking in a patch to >> round up the cluster size to be the bigger of (64k, >> $target_cluster_size) in order to make sure that incremental backups in >> particular never copy a fraction of a cluster. As a side-effect, the >> same round-up will happen for all modes (sync=top,none,full). >> >> If QEMU is aware of the target cluster size of 4MB, this would >> immediately jump the copy-size up to 4MB clusters for you. >> >> See: https://lists.nongnu.org/archive/html/qemu-devel/2016-02/msg02839.html > > Thanks for your patches and thanks for your great answer. But our > problem is not the target but the source ;-) The target has a local > cache and don't care about the cluster size but the source does not. > > But it works fine if we change the default cluster size to 4MB. So it > has point us to the right direction. > > Stefan >
Ah, sorry, I misunderstood. It's easy to change anyway! I am in favor of adding a configurable parameter, as long as it keeps the other constraints I mentioned in mind. --js >> >> Otherwise, after my trivial fix, you should find cluster_size to be a >> mutable concept and perhaps one that you could introduce a runtime >> parameter for if you could convince the necessary parties that it's >> needed in the API. >> >> You'd have to be careful in the case of incremental that all the various >> cluster sizes work well together: >> >> - Granularity of bitmap (Defaults to cluster size of source, or 64K if >> unknown. can be configured to be arbitrarily larger.) >> - Cluster size of source file (For qcow2, defaults to 64k) >> - Cluster size of target file >> - Cluster size of backup routine (Currently always 64K) >> >> I think that LCM(source_cluster_size, target_cluster_size, >> backup_cluster_size) = MAX(A, B, C) will always be a safe minimum. >> >> Bitmap granularity is more flexible, and it is more efficient when the >> bitmap granularity matches the backup cluster size, but it can cope with >> mismatches. If the bitmap is smaller or larger than the backup cluster >> size, it generally means that more data that is clean is going to be >> transferred across the pipe. >> >> (Hmm, I wonder if it's worth checking in code to adjust a bitmap >> granularity after it has been created so people can easily experiment >> with performance tweaking here.) >> >> >> Let me know if any of my ramble sounds interesting :) >> --John >> -- —js