Re: [BackupPC-users] filesystem benchmark results

On Fri, 2006-03-10 at 20:18, Matt wrote: > >Are you comparing uncompressed native rsync runs to the perl version > >that handles backuppc's compressed files? > > > More or less yes. Dirvish boils down to a perl wrapper around "rsync > --link-dest=DIR ..." > > I haven't tested backuppc without co

Re: [BackupPC-users] filesystem benchmark results

Les Mikesell wrote: >Are you comparing uncompressed native rsync runs to the perl version >that handles backuppc's compressed files? > More or less yes. Dirvish boils down to a perl wrapper around "rsync --link-dest=DIR ..." I haven't tested backuppc without compression -- after all this was one

Re: [BackupPC-users] filesystem benchmark results

On Fri, 2006-03-10 at 14:41, Matt wrote: > I'm not convinced that there > >is much you can do to maintain any kind of structured order > >over a long term when you are adding files from multiple > >sources simultaneously and expiring them more or less randomly. > > > It's not really random! Th

Re: [BackupPC-users] filesystem benchmark results

On Fri, 2006-03-10 at 15:37, David Brown wrote: > describes the state of my > understanding of incremental backup (as of about 5 years ago) when I wrote > Adump. Adump grew out of my frustration of no existing solutions being > able to correctly rest

Re: [BackupPC-users] filesystem benchmark results

On Fri, Mar 10, 2006 at 02:21:41PM -0600, Les Mikesell wrote: > Sort-of... Consider what happens if you rename a directory > containing old files. An incremental based on timestamps won't > take the files in their new locations. Dump can deal with the > renamed directory but you have to back up

Re: [BackupPC-users] filesystem benchmark results

Les Mikesell wrote: >Of course there are other ways to do things, but they aren't >necessarily going to be better. I'm not convinced that there >is much you can do to maintain any kind of structured order >over a long term when you are adding files from multiple >sources simultaneously and expiri

Re: [BackupPC-users] filesystem benchmark results

On Fri, 2006-03-10 at 13:55, David Brown wrote: > Also, as far as I know, every backup program that does incrementals looks > at just the timestamp field. On unix, the ctime and mtime field together > will always tell you of a file change, as long as the system clock is > monotonically increasing

Re: [BackupPC-users] filesystem benchmark results

On Fri, Mar 10, 2006 at 01:39:24PM -0600, Les Mikesell wrote: > Rsync incrementals turn on the option to skip files where the length > and timestamp match, making them considerably faster but more likely > to accidentally miss a changed file. However, since they always work > against the last full

Re: [BackupPC-users] filesystem benchmark results

On Fri, 2006-03-10 at 12:57, Matt wrote: > >I'm not going to discuss this proposal any further. The way that backuppc > >(and other programs) work is too ingrained in people's thinking to realize > >that there are other ways to do things. > > > > > I generally dislike such statements. They sto

Re: [BackupPC-users] filesystem benchmark results

David Brown wrote: >When files are no longer used, they are simply deleted, and those numbered >slots are never used. > >There is _NO_ linking in my proposal, that is it's point. Also, if you >hash the entire file, instead of just the beginning, and you use a strong >enough hash function, there

Re: [BackupPC-users] filesystem benchmark results

On Fri, Mar 10, 2006 at 07:53:20AM -0600, Les Mikesell wrote: > On Fri, 2006-03-10 at 01:29, David Brown wrote: > > > By storing the files in the same order as the traversal, they will likely > > stay near files that will be retrieved at a similar time. > > How would you maintain any kind of orde

Re: [BackupPC-users] filesystem benchmark results

On Fri, 2006-03-10 at 01:29, David Brown wrote: > By storing the files in the same order as the traversal, they will likely > stay near files that will be retrieved at a similar time. How would you maintain any kind of order as backups are expired and replaced? You also need to check new files f

Re: [BackupPC-users] filesystem benchmark results

On Thu, Mar 09, 2006 at 06:31:51PM -0800, Matt wrote: > Wouldn't it be better to keep the directory structure of the > (compressed) files and keep hash and attributes in the DB? After all, > this is how the data are received and how they will be accessed during > a restore. Even that isn't all

Re: [BackupPC-users] filesystem benchmark results

On Thu, 2006-09-03 at 18:31 -0800, Matt wrote: > Speedwise, backuppc really sucks compared to dirvish. With dirvish I has > able to backup 1.4 TB in 30 minutes. Most of the time was spend by > rsync collecting and transmitting the file lists. Now a full backups > take almost a day, even if the a

Re: [BackupPC-users] filesystem benchmark results

David Brown wrote: > I think the solution is to change backuppc to not > >create multiple trees, but to store the filesystem tree in some kind of >database, and just store the files themselves in the pool. Database >engines are optimized to be able to handle multiple indexing into the data, >wher

Re: [BackupPC-users] filesystem benchmark results

IMHO, BackupPC could use a combination of index data in the database and files on FS. The database can be anything - sqlite3, mysql, even ODBC-compliant.. That would speed up some checks, I beleive2006/3/8, David Brown <[EMAIL PROTECTED]>: On Tue, Mar 07, 2006 at 09:23:36AM -0600, Carl Wilhelm Sode

Re: [BackupPC-users] filesystem benchmark results

On Tue, Mar 07, 2006 at 09:23:36AM -0600, Carl Wilhelm Soderstrom wrote: > I'm experimenting with an external firewire drive enclosure, and I formatted > it with 3 different filesystems, then used bonnie++ to generate 10GB of > sequential data, and 1,024,000 small files between 1000 and 100 bytes

Re: [BackupPC-users] filesystem benchmark results

On Tue, 2006-03-07 at 12:46, David Brown wrote: > There are no links. Each file has one entry, under the pool directory. > All that has to be managed is creation and deletion of these files. It is > not difficult to be able to easily recover from a crash in either of these > scenarios, as long a

RE: [BackupPC-users] filesystem benchmark results

[BackupPC-users] filesystem benchmark results > > okay, right? it's only when you want to preserve or copy your > > pool that there's an issue? (or am i neglecting something? i > > might well be.) > > Even just the normal process of looking at the pool,

Re: [BackupPC-users] filesystem benchmark results

On Tue, Mar 07, 2006 at 12:15:50PM -0600, Les Mikesell wrote: > The piece that has to be atomic has to do with the actual pool > file, so unless you move the data into the database as well > you can't atomically manage the links or ever be sure that > they are actually correct. And if you move th

Re: [BackupPC-users] filesystem benchmark results

> > okay, right? it's only when you want to preserve or copy your > > pool that there's an issue? (or am i neglecting something? i > > might well be.) > > Even just the normal process of looking at the pool, either to see if a > file is present, or as part of the cleanup scan is much slow

Re: [BackupPC-users] filesystem benchmark results

On Tue, 2006-03-07 at 11:55, David Brown wrote: > > > > All you'll do by trying is lose the atomic nature of the hardlinks. > > You aren't ever going have the data at the same time you know all > > of it's names so you can store them close together. Just throw in > > lots of ram and let caching d

Re: [BackupPC-users] filesystem benchmark results

On Tue, Mar 07, 2006 at 11:49:40AM -0600, Les Mikesell wrote: > > I still say it is going to be a lot easier to change how backuppc works > > than it is going to be to find a filesystem that will deal with this very > > unusual use case well. > > All you'll do by trying is lose the atomic nature

Re: [BackupPC-users] filesystem benchmark results

On Tue, 2006-03-07 at 11:10, David Brown wrote: > The depth isn't really the issue. It is that they are created under one > tree, and hardlinked to another tree. The normal FS optimization of > putting the inodes of files in a given directory near each other breaks > down, and the directories in

Re: [BackupPC-users] filesystem benchmark results

On Tue, Mar 07, 2006 at 12:20:04PM -0500, Paul Fox wrote: > to clarify -- in the "normal" case, where the backup data is > usually not read, but only written, the current filesystems are > okay, right? it's only when you want to preserve or copy your > pool that there's an issue? (or am i neglec

Re: [BackupPC-users] filesystem benchmark results

> The depth isn't really the issue. It is that they are created under one > tree, and hardlinked to another tree. The normal FS optimization of > putting the inodes of files in a given directory near each other breaks > down, and the directories in the pool end up with files of very diverse

Re: [BackupPC-users] filesystem benchmark results

On Tue, Mar 07, 2006 at 10:21:15AM -0600, Carl Wilhelm Soderstrom wrote: > ok. point taken. > Bonnie does create a very shallow tree for these files, but it's only a > directory or two deep. The depth isn't really the issue. It is that they are created under one tree, and hardlinked to another t

Re: [BackupPC-users] filesystem benchmark results

On 03/07 08:14 , David Brown wrote: > Unfortunately, the resultant filesystem has very little resemblance to the > file tree that backuppc writes. I'm not sure if there is any utility that > creates this kind of tree, and I would argue that backuppc shouldn't be > either, since it is so hard on th

Re: [BackupPC-users] filesystem benchmark results

On Tue, Mar 07, 2006 at 09:23:36AM -0600, Carl Wilhelm Soderstrom wrote: > I'm experimenting with an external firewire drive enclosure, and I formatted > it with 3 different filesystems, then used bonnie++ to generate 10GB of > sequential data, and 1,024,000 small files between 1000 and 100 bytes

Re: [BackupPC-users] filesystem benchmark results

On 03/07 09:54 , Les Mikesell wrote: > See if you can find a benchmark program called 'postmark'. This > used to be available from NetApp but I haven't been able to find > a copy recently. It specifically tests creation and deletion > of lots of small files. When I used it years ago it showed >

Re: [BackupPC-users] filesystem benchmark results

On Tue, 2006-03-07 at 09:23, Carl Wilhelm Soderstrom wrote: > Am I missing something here? Am I mis-interpreting the data? Is there anyone > else out there with more bonnie experience than I, who can suggest other > things to try to gain more surety about this? See if you can find a benchmark pro

Re: [BackupPC-users] filesystem benchmark results

On 03/07 04:43 , Guus Houtzager wrote: > I think you're right. I have 2 suggestions for additional testing. It's my > experience that backuppc became really really slow after a few weeks when > more data began to accumulate. Could you test ext3 again, but with a few > million more files? I'm als

Re: [BackupPC-users] filesystem benchmark results

Hi, On Tuesday 07 March 2006 16:23, Carl Wilhelm Soderstrom wrote: > I'm experimenting with an external firewire drive enclosure, and I > formatted it with 3 different filesystems, then used bonnie++ to generate > 10GB of sequential data, and 1,024,000 small files between 1000 and 100 > bytes in s

[BackupPC-users] filesystem benchmark results

I'm experimenting with an external firewire drive enclosure, and I formatted it with 3 different filesystems, then used bonnie++ to generate 10GB of sequential data, and 1,024,000 small files between 1000 and 100 bytes in size. I tried it with xfs, reiserfs, and ext3; and contrary to a lot of hype