Hi, Les Mikesell wrote on 2009-06-02 17:32:24 -0500 [Re: [BackupPC-users] Backing up a BackupPC server]: > Jeffrey J. Kosowsky wrote: > > [...] > > Once we are talking about redoing things, I would prefer to use a > > full md5sum hash for the name of the pool file. [...] > > With this approach then you would automatically have "a common hashed > > filename that is ['statistically'] unique across all instances for > > every piece of content." > > Somehow the number of possible different file contents and the number > possible md5sums don't seem quite statistically equivalent to me. And > then there's: > > http://www.mscs.dal.ca/~selinger/md5collision/
first of all, if you are *not* using rsync, you *don't* get a *full* md5sum hash for free or even cheap. You (Jeffrey) know the code well enough to realize that BackupPC goes to great pains to avoid writing to the pool disk unless necessary. If you need to transfer the whole file (of arbitrary size) before you can look up the pool entry, you *have to* write a temporary copy (probably compressed, too, giving up the benefits you gain from only compressing once and decompressing when matching). You have to handle collisions just the same (meaning re-reading your temporary copy and comparing to the pool file). Yuck. Yes, you can special-case small files that fit into memory, but yuck just the same. If you use a *partial* md5sum, there's no gain from rsync, and you trivially get collisions just like you do now. That is not to say, if we end up using a database, that it would not be a good idea to store the full md5sum in the database. In fact, with a database, file names would be somewhat arbitrary, and I'd propose keeping them *short* for the sake of rsync et al. and file lists. Regards, Holger ------------------------------------------------------------------------------ OpenSolaris 2009.06 is a cutting edge operating system for enterprises looking to deploy the next generation of Solaris that includes the latest innovations from Sun and the OpenSource community. Download a copy and enjoy capabilities such as Networking, Storage and Virtualization. Go to: http://p.sf.net/sfu/opensolaris-get _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/