[rdiff-backup-users] Re: Remote encrypted backup with slow connection.
On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller wrote: > On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: >> local rdiff-backup dir with remote server but how? If I will use for >> example rsync it still need to check whole files for changes (read, >> download it) and upload only new. I hope you will understand what I >> need and help me. > > rsync won't check whole files unless you give the -c flag. Otherwise, it > just compares metadata. I don't know if that's also the case with > rdiff-backup, but I assume so.So So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still it will download changed files to generate diff. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd wrote: > Piotr Karbowski wrote: >> >> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller wrote: >> >>> >>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: >>> local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. >>> >>> rsync won't check whole files unless you give the -c flag. Otherwise, it >>> just compares metadata. I don't know if that's also the case with >>> rdiff-backup, but I assume so. >>> >> >> So I need to know how rdiff default compares data, if by size and >> mod-time, it will not be so painful but still itefficient will download >> changed >> files to generate diff. > > Rdiff-backup is designed to be ultra-efficient at this activity. It only > sends the changes in a file over the wire, not the whole file. To do this it > uses the librsync library which is effectively the same as rsync. You can > read more about the technique at http://en.wikipedia.org/wiki/Rsync. > rdiff-backup does not use file times to determine whether to do backups. It > can backup very large files with small changes very quickly. > > Dominic > You dont understand me, rdiff-backup is efficient, but to make diff it must read WHOLE file, on remote nfs or sshfs it is SLW and painful. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd wrote: > Piotr Karbowski wrote: >> >> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd >> wrote: >> >>> >>> Piotr Karbowski wrote: >>> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller wrote: > > On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: > > >> >> local rdiff-backup dir with remote server but how? If I will use for >> example rsync it still need to check whole files for changes (read, >> download it) and upload only new. I hope you will understand what I >> need and help me. >> >> > > rsync won't check whole files unless you give the -c flag. Otherwise, > it > just compares metadata. I don't know if that's also the case with > rdiff-backup, but I assume so. > > So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still itefficient will download changed files to generate diff. >>> >>> Rdiff-backup is designed to be ultra-efficient at this activity. It only >>> sends the changes in a file over the wire, not the whole file. To do this >>> it >>> uses the librsync library which is effectively the same as rsync. You can >>> read more about the technique at http://en.wikipedia.org/wiki/Rsync. >>> rdiff-backup does not use file times to determine whether to do backups. >>> It >>> can backup very large files with small changes very quickly. >>> >>> Dominic >>> >>> >> >> You dont understand me, rdiff-backup is efficient, but to make diff it >> must read WHOLE file, on remote nfs or sshfs it is SLW and >> painful > > Sorry I get it now. But I think rdiff-backup and rsync require a separate > computer at the remote end in order to optimise transfers, so if you are > just accessing a remote share using sshfs or similar then they can still > work of course but as you realise they will be slow. I guess it is not > possible for you to run rdiff-backup (or rsync) at the remote end as well? > > You could run rdiff-backup locally to create a backup store and then mirror > this store to the remote share using rcp. Still it will be slow because > rdiff-backup always stores the latest copy of each file in full and so if > this changes even slightly then the whole file will must be transferred by > rcp. > > Duplicity http://duplicity.nongnu.org/ might work better for you, because it > uses forward diffs. Also its archives are secure. > > Although not directly relevant I found a page here > http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to > greatly speed up OpenSSH in some situations. Duplicity is interesing project. What you think about using rdiff-backup to create local backup, for example in /backups and then send this /backups to remote server by duplicity? As far as I know duplicity is encrypted so I DONT need using encfs, dmcrypt or other - only ssh access is needed (realy I dont need duplicity on remote server?). I just want be able to send _ENCRYPTED_ backups to remote server where I have only ssh access (sftp/scp work). ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
On Tue, Sep 22, 2009 at 4:36 PM, Dominic Raferd wrote: > Piotr Karbowski wrote: >> >> On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd >> wrote: >> >>> >>> Piotr Karbowski wrote: >>> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd wrote: > > Piotr Karbowski wrote: > > >> >> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller >> wrote: >> >> >> >>> >>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: >>> >>> >>> local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. >>> >>> rsync won't check whole files unless you give the -c flag. Otherwise, >>> it >>> just compares metadata. I don't know if that's also the case with >>> rdiff-backup, but I assume so. >>> >>> >>> >> >> So I need to know how rdiff default compares data, if by size and >> mod-time, it will not be so painful but still itefficient will >> download >> changed >> files to generate diff. >> >> > > Rdiff-backup is designed to be ultra-efficient at this activity. It > only > sends the changes in a file over the wire, not the whole file. To do > this > it > uses the librsync library which is effectively the same as rsync. You > can > read more about the technique at http://en.wikipedia.org/wiki/Rsync. > rdiff-backup does not use file times to determine whether to do > backups. > It > can backup very large files with small changes very quickly. > > Dominic > > > You dont understand me, rdiff-backup is efficient, but to make diff it must read WHOLE file, on remote nfs or sshfs it is SLW and painful >>> >>> Sorry I get it now. But I think rdiff-backup and rsync require a separate >>> computer at the remote end in order to optimise transfers, so if you are >>> just accessing a remote share using sshfs or similar then they can still >>> work of course but as you realise they will be slow. I guess it is not >>> possible for you to run rdiff-backup (or rsync) at the remote end as >>> well? >>> >>> You could run rdiff-backup locally to create a backup store and then >>> mirror >>> this store to the remote share using rcp. Still it will be slow because >>> rdiff-backup always stores the latest copy of each file in full and so if >>> this changes even slightly then the whole file will must be transferred >>> by >>> rcp. >>> >>> Duplicity http://duplicity.nongnu.org/ might work better for you, because >>> it >>> uses forward diffs. Also its archives are secure. >>> >>> Although not directly relevant I found a page here >>> http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to >>> greatly speed up OpenSSH in some situations. >>> >> >> Duplicity is interesing project. What you think about using >> rdiff-backup to create local backup, for example in /backups and then >> send this /backups to remote server by duplicity? As far as I know >> duplicity is encrypted so I DONT need using encfs, dmcrypt or other - >> only ssh access is needed (realy I dont need duplicity on remote >> server?). >> >> I just want be able to send _ENCRYPTED_ backups to remote server where >> I have only ssh access (sftp/scp work). > > I have not used duplicity myself, I use rdiff-backup. But I am not sure you > need to run rdiff-backup first, I think duplicity may make its own local > copies of backup increments so that it can send future increments without > having to access the earlier increments from the remote share. And yes > duplicity sends encrypted files so you don't need other encryption. > > Because duplicity uses forward diffs you have to keep all backups forever, > and if there is any corruption of a file you lose all backups that occurred > *after* the date of this file. Rdiff-backup uses reverse diffs so > corruption, if it occurs, affects backups *before* the date of the corrupted > backup. With rdiff-backup you can delete backups before a certain date > (though in my experience the storage is so efficient it is not usually worth > bothering), which is not possible with duplicity. > > Still it sounds like duplicity would better suit your needs. > So best will be using duplicity to make local backup and run rsync to send new files (diffs) to remote server. and if backup will be TOO big just mv backup backup_old and start new backup (every 4 weeks for example) ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
On Tue, Sep 22, 2009 at 5:45 PM, Dominic Raferd wrote: > Piotr Karbowski wrote: >> >> So best will be using duplicity to make local backup and run rsync to >> send new files (diffs) to remote server. and if backup will be TOO big >> just mv backup backup_old and start new backup (every 4 weeks for >> example) >> > > I think so. There is a duplicity mailing list where you could probably get > more help: http://lists.nongnu.org/mailman/listinfo/duplicity-talk > Thanks for you all! ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller wrote: On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. rsync won't check whole files unless you give the -c flag. Otherwise, it just compares metadata. I don't know if that's also the case with rdiff-backup, but I assume so. So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still it will download changed files to generate diff. Rdiff-backup is designed to be ultra-efficient at this activity. It only sends the changes in a file over the wire, not the whole file. To do this it uses the librsync library which is effectively the same as rsync. You can read more about the technique at http://en.wikipedia.org/wiki/Rsync. rdiff-backup does not use file times to determine whether to do backups. It can backup very large files with small changes very quickly. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd wrote: Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller wrote: On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. rsync won't check whole files unless you give the -c flag. Otherwise, it just compares metadata. I don't know if that's also the case with rdiff-backup, but I assume so. So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still itefficient will download changed files to generate diff. Rdiff-backup is designed to be ultra-efficient at this activity. It only sends the changes in a file over the wire, not the whole file. To do this it uses the librsync library which is effectively the same as rsync. You can read more about the technique at http://en.wikipedia.org/wiki/Rsync. rdiff-backup does not use file times to determine whether to do backups. It can backup very large files with small changes very quickly. Dominic You dont understand me, rdiff-backup is efficient, but to make diff it must read WHOLE file, on remote nfs or sshfs it is SLW and painful Sorry I get it now. But I think rdiff-backup and rsync require a separate computer at the remote end in order to optimise transfers, so if you are just accessing a remote share using sshfs or similar then they can still work of course but as you realise they will be slow. I guess it is not possible for you to run rdiff-backup (or rsync) at the remote end as well? You could run rdiff-backup locally to create a backup store and then mirror this store to the remote share using rcp. Still it will be slow because rdiff-backup always stores the latest copy of each file in full and so if this changes even slightly then the whole file will must be transferred by rcp. Duplicity http://duplicity.nongnu.org/ might work better for you, because it uses forward diffs. Also its archives are secure. Although not directly relevant I found a page here http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to greatly speed up OpenSSH in some situations. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd wrote: Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd wrote: Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller wrote: On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. rsync won't check whole files unless you give the -c flag. Otherwise, it just compares metadata. I don't know if that's also the case with rdiff-backup, but I assume so. So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still itefficient will download changed files to generate diff. Rdiff-backup is designed to be ultra-efficient at this activity. It only sends the changes in a file over the wire, not the whole file. To do this it uses the librsync library which is effectively the same as rsync. You can read more about the technique at http://en.wikipedia.org/wiki/Rsync. rdiff-backup does not use file times to determine whether to do backups. It can backup very large files with small changes very quickly. Dominic You dont understand me, rdiff-backup is efficient, but to make diff it must read WHOLE file, on remote nfs or sshfs it is SLW and painful Sorry I get it now. But I think rdiff-backup and rsync require a separate computer at the remote end in order to optimise transfers, so if you are just accessing a remote share using sshfs or similar then they can still work of course but as you realise they will be slow. I guess it is not possible for you to run rdiff-backup (or rsync) at the remote end as well? You could run rdiff-backup locally to create a backup store and then mirror this store to the remote share using rcp. Still it will be slow because rdiff-backup always stores the latest copy of each file in full and so if this changes even slightly then the whole file will must be transferred by rcp. Duplicity http://duplicity.nongnu.org/ might work better for you, because it uses forward diffs. Also its archives are secure. Although not directly relevant I found a page here http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to greatly speed up OpenSSH in some situations. Duplicity is interesing project. What you think about using rdiff-backup to create local backup, for example in /backups and then send this /backups to remote server by duplicity? As far as I know duplicity is encrypted so I DONT need using encfs, dmcrypt or other - only ssh access is needed (realy I dont need duplicity on remote server?). I just want be able to send _ENCRYPTED_ backups to remote server where I have only ssh access (sftp/scp work). I have not used duplicity myself, I use rdiff-backup. But I am not sure you need to run rdiff-backup first, I think duplicity may make its own local copies of backup increments so that it can send future increments without having to access the earlier increments from the remote share. And yes duplicity sends encrypted files so you don't need other encryption. Because duplicity uses forward diffs you have to keep all backups forever, and if there is any corruption of a file you lose all backups that occurred *after* the date of this file. Rdiff-backup uses reverse diffs so corruption, if it occurs, affects backups *before* the date of the corrupted backup. With rdiff-backup you can delete backups before a certain date (though in my experience the storage is so efficient it is not usually worth bothering), which is not possible with duplicity. Still it sounds like duplicity would better suit your needs. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: So best will be using duplicity to make local backup and run rsync to send new files (diffs) to remote server. and if backup will be TOO big just mv backup backup_old and start new backup (every 4 weeks for example) I think so. There is a duplicity mailing list where you could probably get more help: http://lists.nongnu.org/mailman/listinfo/duplicity-talk ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
