Re: [pve-devel] Improve container backup speed dramatically (factor 100-1000)

2020-11-20 Thread Dominik Csapak
arg, of course i forgot the reference to the rolling hash info... here it is: 0: https://en.wikipedia.org/wiki/Rolling_hash ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Improve container backup speed dramatically (factor 100-1000)

2020-11-20 Thread Dominik Csapak
hi, it seems there are some misunderstandings as how the backup actually works, i'll try to clear that up On 11/20/20 8:18 AM, Carsten Härle wrote: Yes, that is how the current variable sized chunking algorithm works. ... "zfs diff" does not provide the information needed for our deduplicati

Re: [pve-devel] Improve container backup speed dramatically (factor 100-1000)

2020-11-19 Thread Carsten Härle
>>Yes, that is how the current variable sized chunking algorithm works. ... "zfs diff" does not provide the information needed for our deduplication algorithm, so we cannot use that. << 1) Can you please outline the algorithm? 2) Why you think, it is not possible to use the changed information

Re: [pve-devel] Improve container backup speed dramatically (factor 100-1000)

2020-11-19 Thread Dietmar Maurer
> Container backup is very slow compared to VM backup. I have a 500 GB > container (sftp server) with minimal changing files, but even the incremental > bakcups take 2 hours with heavy disk activity. Almost nothing is transfered > to the backup server. It seems that it it reads the whole conta

[pve-devel] Improve container backup speed dramatically (factor 100-1000)

2020-11-19 Thread Carsten Härle
Container backup is very slow compared to VM backup. I have a 500 GB container (sftp server) with minimal changing files, but even the incremental bakcups take 2 hours with heavy disk activity. Almost nothing is transfered to the backup server. It seems that it it reads the whole container every