On Sun, Apr 20, 2008 at 4:39 PM, Bob Friesenhahn <[EMAIL PROTECTED]> wrote: > On Sun, 20 Apr 2008, Peter Tribble wrote: > > > > My experience so far is that anything past a terabyte and 10 million > files, > > and any backup software struggles. > > > > What is the cause of the "struggling"? Does the backup host run short of > RAM or CPU? If backups are incremental, is a large portion of time spent > determining the changes to be backed up? What is the relative cost of many > small files vs large files?
It's just the fact that, while the backup completes, it can take over 24 hours. Clearly this takes you well over any backup window. It's not so much that the backup software is defective; it's an indication that traditional notions of backup need to be rethought. I have one small (200G) filesystem that takes an hour to do an incremental with no changes. (After a while, it was obvious we don't need to do that every night.) The real killer, I think, is sheer number of files. For us, 10 million files isn't excessive. I have one filesystem that's likely to have getting on for 200 million files by the time the project finishes. (Gulp!) > How does 'zfs send' performance compare with a traditional incremental > backup system? I haven't done that particular comparison. (zfs send isn't useful for backup - doesn't span tapes, doesn't hold an index of the files.) But I have compared it against various varieties of tar for moving data between machines, and the performance of 'zfs send' wasn't particularly good - I ended up using tar instead. (Maybe lots of smallish files again.) For incrementals, it may be useful. But that presumes a replicated configuration (preferably with the other node at a DR site), rather than use in backups. -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/ _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
