Les Mikesell wrote at about 12:35:49 -0500 on Monday, August 31, 2009: > Jeffrey J. Kosowsky wrote: > > > > > It's almost as if you guys haven't heard of filesystem-specific dump > > > utilities. For such utils (vxdump, ufsdump, zfs send/receive, etc.) > > the > > > number of hardlinks isn't a problem. You can do both full and > > > incremental dumps, even across separate machines. This isn't a problem > > > that needs solving. > > > > I think you are missing some key points. > > First, why should a program require it's own separate filesystem? This > > seems to me like an unnatural and kludgey type of requirement. > > It doesn't, unless you hit some limit on the filesystem you use. The > usual limit is how many times and how far you have to move the disk head > since that's the slow operation. > > > I see lots of advantage in keeping the database portion relatively > > small, fast, replicable, and moveable. Then you can keep and > > distribute the files themselves wherever you want them spread across > > one or more separate filesystems. Then the database portion is > > optimized for what a database does best and the file-storing > > portion can be optimized for what a filesystem stores best. And both > > parts are easily moveable, replicable and not dependent or limited by > > hardlinks or other filesystem-dependent functionality. > > But the parts aren't independent. How do you propose keeping them in > sync or fixing them when they inevitably differ? > > > Don't get me wrong - Backuppc is great and hardlinks are a great > > kludge to at first glance get something for nothing. I'm just saying > > that hardlinks while "easy" bring some longer-term limitations and > > that there comes the time when it may be worth investing in going > > beyond them. > > > > Personally I would like to see Backuppc evolve to combine the pooling > > functionality, leveraging of rsync, and relative simplicity of the > > existing Backuppc with the expandability, portability, and flexibility > > of the database-based systems like Bacula. I believe that the > > combination of a database to store the file attributes and metadata > > together with a filesystem to store the pool would be an ideal hybrid. > > But someone has yet to establish that this would be faster if you don't > add the requirement of putting the sql tables on different drives - and > weren't you just saying that applications shouldn't have requirements > like that? > > > > > > > For anyone thinking that working with giant multi-gigabyte BLOBs in a > > > database is the right way to go, I suggest you actually attempt it > > > yourself and see what happens. I'm backing up my HD video production > > > rig with BackupPC, and although such a machine (Windows, 16T of > > storage, > > > most video files are at least 50G in size) is outside of the intent of > > > BackupPC, it actually works. If BackupPC were to rely on an SQL > > > database, it would greatly shrink the potential userbase. > > > > You are attacking a straw man. No one has ever suggested > > "multi-gigabyte BLOBS in a database." The database would only consist > > of the filenames, links, attrib data, and other backup-related metadata. I > > would imagine in most cases this would be at most a couple of > > gigabytes, assuming you have millions of files in your pool. > > If you don't put them in the database you can't enforce atomic > operations - something you get in the filesystem for the price of a seek > over to the inode to bump the link count with the entry locked. >
I don't see why everything needs to worship at the alter of atomic operations. There are other ways to ensure that things don't go wrong. Again, it would be helpful to know some specific use cases that you are concerned about then we can think about potential "locking" mechanisms or checks to make sure that nothing breaks. ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/