Re: [rdiff-backup-users] Feature requests questions/discussion
I think the hashing of the files is worth the overhead. At work we currently keep 3 months of backups on our backup server ~ 1.4TBytes of data. I would like the hash values to be available for reporting against, so that I can also do the equ of tripwire on the servers. ie these files changed, these files were added and these files were deleted. At the moment I find a lot of false positives. ie a file is backed up when it has not changed at all. When I was first reading about rdiff-backup, and that it used rsync-lib, it seemed to be a side affect of the rsync library. It had to do with the method that rsync figured out what had changed. F. On Thu, 2005-10-27 at 13:44 +0200, Wiebe Cazemier wrote: Ben Escoto wrote: Another problem is that we would have to rewrite the mirror_metadata file each time. If there was a bug in rdiff-backup or a disk corruption, and the main file was lost, we would lose all the metadata information. I don't have any specific worry in mind here, it just makes me uneasy. This is a genuine concern. Remember I corrupted my external HD severly with USB errors? That sort of thing can happen. Just two days ago, I had a similair situation. My computer suddenly became very unstable, probably because I did rrmod -f on a module which I couldn't get un-used. At first I suspected my HD, so I made a backup as soon as I could. It crashed as well. Later, when I diagnosed my disc and restarted my system, I ran a new backup. I was very glad the failed one got rolled back. I know you can save a temporary mirror_metadata as a working version, but if rdiff-backup had crashed writing the temp version over the orignal one, you are bust. Or, power can fail suddenly. Anything can happen... Anyway, it's good to guard against the unexpected and unprobable. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki signature.asc Description: This is a digitally signed message part ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Feature requests questions/discussion
On Fri, 4 Nov 2005, Fran Firman wrote: At the moment I find a lot of false positives. ie a file is backed up when it has not changed at all. When I was first reading about rdiff-backup, and that it used rsync-lib, it seemed to be a side affect of the rsync library. It had to do with the method that rsync figured out what had changed. there was a point in the past, with some specific version of librsync i think, that i was having troubles with lots of increments for files which hadn't changed at all... an upgrade made the problem disappear and i didn't bother tracking it further. -dean ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Feature requests questions/discussion
Wiebe Cazemier [EMAIL PROTECTED] wrote the following on Fri, 21 Oct 2005 17:48:47 +0200 First, I would really like an option like --store-checksums so that rdiff-backup calculates md5 hashes when doing a backup, and that that checksum it used for integrity checks upon restoration. But at restoration time, the check should not be an option, it should always be done, if the file being restored has hash info. This to prevent user-mistake. I once severly corrupted a partition on an external HD because of USB2 transfer errors (which I haven't been able to solve BTW). A lot was damaged, but of every file that resided in compressed files I was told about the corruption, beceause of the hashing usually done in zips/gzips/etc. The rdiff-backup repository became useless, because it had no idea the files were damaged. Such an option would of course be annoying to most people, because it's quite slow, but most of my backups are done through cron, so it doesn't matter for me. I think as an option, this could be very valuable. It doesn't even have to be that slow, if it's cleverly integrated with the copy routine. It's a good idea, and one that someone else has suggested before. The checksums would be stored in the mirror-metadata file. I don't even think it would be hard to implement. And there could be a --verify switch to go through the repository and make sure everything checksums correctly. Summarized, --store-checksums would calculate checksum info for integrity checks, and --checksum-diffs would use checksums for change-detections, instead of mtime+size. Another good suggestion I think, which has come up before. You mentioned ctime before, I was going to add in ctime checking but there was some complication (I forget what) and it never got in. Does anyone else think they would use Wiebe's --checksum-diff option? -- Ben Escoto pgpMNV7oncYxd.pgp Description: PGP signature ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Feature requests questions/discussion
Ben Escoto wrote: It's a good idea, and one that someone else has suggested before. The checksums would be stored in the mirror-metadata file. I don't even think it would be hard to implement. Do you think it's possible to combine it with the copy syscall/API-call rdiff-backup probably uses so that the data which is read by copying is checksummed at the same time? If it is, it wouldn't even take more time to backup a new file than it would without checksums. And there could be a --verify switch to go through the repository and make sure everything checksums correctly. Indeed, I even forgot this. You mentioned ctime before, I was going to add in ctime checking but there was some complication (I forget what) and it never got in. If --checksum-diffs would make it, I wouldn't need ctime checking. But I guess there is value for such an option (speed being a definite advantage), as a third change-detection method. But it won't be widely useable I guess. I wouldn't be surprised if NFS or SAMBA don't supply correct ctimes. And what about FAT32? Doing ls -lc does reveil different times than a normal ls -l, but does that FS, and the windows driver which controlles it, properly set ctimes? ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Feature requests questions/discussion
On Tue, 25 Oct 2005, Ben Escoto wrote: It's a good idea, and one that someone else has suggested before. The checksums would be stored in the mirror-metadata file. I don't even think it would be hard to implement. And there could be a --verify switch to go through the repository and make sure everything checksums correctly. that would be cool... i'd use that. Summarized, --store-checksums would calculate checksum info for integrity checks, and --checksum-diffs would use checksums for change-detections, instead of mtime+size. Another good suggestion I think, which has come up before. You mentioned ctime before, I was going to add in ctime checking but there was some complication (I forget what) and it never got in. Does anyone else think they would use Wiebe's --checksum-diff option? in many cases for me this would cost too much on the machine being backed up (lots of disk seeks and cpu time) ... but maybe i could use it periodically on weekends... hey, did you know there's actually nanosecond resolution to [acm]time on linux 2.6? (and on several BSDs i think) i don't know if the interfaces show up in python -- but the C structure elements are st_atimensec/st_ctimensec/st_mtimensec, and the utimes(2) syscall can set nanosecond resolution timestamps. -dean ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
