Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread backuppc
MtK - SmartMtK wrote at about 21:15:01 +0300 on Friday, August 30, 2013: > > It's a lot of files that aren't organized in any particular way. So > > assume the head is going to move for most directory/inode/data > > accesses and multiply by the time the disk takes to seek. > > How is this d

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
I have the same behavior with and without --checksum-seed=32761 (which I added recently) and no most of the files are small, the large files are either the same of completely new. -- Learn the latest--Visual Studio 2012,

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
> Do you still have your old filesystem so you can compare the times to walk with something like 'find . -ctime -1 >/dev/null' (make it read the inode contents as well as the directories)? the old filesystem is still in /var/lib/backuppc.orig, so yes, but again, it'll take a few days (!) > Are

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Michael Stowe
> Hey, > > after a few years using backuppc on a local storage (internal HDDs) I've > moved the pool into an external location connected through NFS. I'm sure this has been mentioned before, but connecting the pool in this way isn't such a great idea, due to potential slowness and reliability iss

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Les Mikesell
On Fri, Aug 30, 2013 at 9:47 AM, MtK - SmartMtK wrote: > I had the exact same behavior on the local 2-drive-mirror, with the > different that I could do only a single backup at a time, since the IO would > make the system unresponsive (now I can easily do 3-4 at the same time). > > Also, just to p

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Les Mikesell
On Fri, Aug 30, 2013 at 1:15 PM, MtK - SmartMtK wrote: > >> It's a lot of files that aren't organized in any particular way. So >> assume the head is going to move for most directory/inode/data >> accesses and multiply by the time the disk takes to seek. > > How is this different then having loc

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread backuppc
MtK - SmartMtK wrote at about 19:08:38 +0300 on Friday, August 30, 2013: > > >> Also, just to point out: > >> df/ls/du/find/etc on the external machine (where the NFS is) take > >> hours (!), so maybe it's not the NFS itself but the directory/file > >> structure. > > >This seems particul

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
>> Also, just to point out: >> df/ls/du/find/etc on the external machine (where the NFS is) take >> hours (!), so maybe it's not the NFS itself but the directory/file >> structure. >This seems particularly important! again, same behavior as when the pool was local, and that's why I couldn't cop

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
> It's a lot of files that aren't organized in any particular way. So > assume the head is going to move for most directory/inode/data > accesses and multiply by the time the disk takes to seek. How is this different then having local disk? ---

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Les Mikesell
On Fri, Aug 30, 2013 at 11:08 AM, MtK - SmartMtK wrote: > >>> Also, just to point out: >>> df/ls/du/find/etc on the external machine (where the NFS is) take >>> hours (!), so maybe it's not the NFS itself but the directory/file >>> structure. > >>This seems particularly important! > again, same be

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Michael Stowe
>> You don't say how much data this is, what transport you're using, or >> what > kind of connection there is between the systems. This *could* be > impressively fast for what you're working with. For now, all I can say > > is, don't expect miracles out of nfs. > Pool is 270.29GB comprising 2451

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Les Mikesell
On Fri, Aug 30, 2013 at 12:27 PM, MtK - SmartMtK wrote: > I guess it's possible I just rather not do this until: > 1. I see that the pool works OK (so that du/find/etc work faster than now) > 2. I completely pin-point NFS as the bottleneck of this. Do you still have your old filesystem so you can

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Les Mikesell
On Fri, Aug 30, 2013 at 12:46 PM, MtK - SmartMtK wrote: > > > Just to be clear again, both ZFS and the ext3 I had locally before behave > the same way. It's a lot of files that aren't organized in any particular way. So assume the head is going to move for most directory/inode/data accesses and

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
I guess it's possible I just rather not do this until: 1. I see that the pool works OK (so that du/find/etc work faster than now) 2. I completely pin-point NFS as the bottleneck of this. -- Learn the latest--Visual Stud

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread Michael Stowe
> I had the exact same behavior on the local 2-drive-mirror, with the > different that I could do only a single backup at a time, since the IO > would make the system unresponsive (now I can easily do 3-4 at the same > time). Having the system do less work would naturally make it more responsive,

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
I had the exact same behavior on the local 2-drive-mirror, with the different that I could do only a single backup at a time, since the IO would make the system unresponsive (now I can easily do 3-4 at the same time). Also, just to point out: df/ls/du/find/etc on the external machine (where the NF

Re: [BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
> You don't say how much data this is, what transport you're using, or what kind of connection there is between the systems. This *could* be impressively fast for what you're working with. For now, all I can say > is, don't expect miracles out of nfs. Pool is 270.29GB comprising 2451613 files and

[BackupPC-users] New Pool same issues

2013-08-30 Thread MtK - SmartMtK
Hey, after a few years using backuppc on a local storage (internal HDDs) I've moved the pool into an external location connected through NFS. Since moving the entire pool/cpool/pc was about to take a few days (!) I decided to try and use the new location from scratch (without moving the backups h