Interesting thread! Jeffrey writes:
> That being said, I agree that using a database to store both the > hardlinks along with the metadata stored in the attrib files would be > a more elegant, extensible, and platform-independent solution though > presumably it would require a major re-write of BackupPC. > > I certainly understand why BackupPC uses hardlinks since it allows for > an easy way to do the pooling and in a sense as you suggest uses the > filesystem as a rudimentary database. > > On the other hand as I and others have mentioned before moving to a > database would add the following advantages: I agree on all the points. If I was writing BackupPC today I would seriously consider this approach. As Les points out, the most important open question (other than reliability) is whether the performance is adequate as the store expands to millions of files (and 10^8, 10^9 or more file blocks). Of course, BackupPC is relatively slow, so maybe the baseline expectation is already sufficiently low. I recently heard about lessfs, which runs on top of FUSE to provide a file system that does block-level de-duplication. See: http://www.lessfs.com https://sourceforge.net/project/showfiles.php?group_id=257120 http://tokyocabinet.sourceforge.net/index.html The actual storage is several very large (sparse?) files on any file system(s) of your choice. It should provide all the benefits you expect: no issues of local limitations on hardlink counts, meta-data etc, and the database files can be copied or rsynced. I'm corresponding with the author to see if some additional useful features could be added. Yes, taking this approach would require a very substantial rewrite. BackupPC would become a lot simpler. But it also creates a significant issue of backward compatibility. The only solution would be to provide tools that import the old BackupPC store into a new one. That is possible, but would likely be very slow. Craig ------------------------------------------------------------------------------ OpenSolaris 2009.06 is a cutting edge operating system for enterprises looking to deploy the next generation of Solaris that includes the latest innovations from Sun and the OpenSource community. Download a copy and enjoy capabilities such as Networking, Storage and Virtualization. Go to: http://p.sf.net/sfu/opensolaris-get _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/