Re: copying milllions of small files and millions of dirs

2013-10-10 Thread aurfalien
On Aug 15, 2013, at 11:46 PM, Nicolas KOWALSKI wrote: On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Is there a faster way to copy files over NFS? I would use find+cpio. This handles hard links, permissions, and in case of later runs, will not copy files if they already exist

Re: copying milllions of small files and millions of dirs

2013-10-10 Thread Warren Block
On Thu, 10 Oct 2013, aurfalien wrote: On Aug 15, 2013, at 11:46 PM, Nicolas KOWALSKI wrote: On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Is there a faster way to copy files over NFS? I would use find+cpio. This handles hard links, permissions, and in case of later runs, will

Re: copying milllions of small files and millions of dirs

2013-08-20 Thread krad
When i migrated a large mailspool in maildir format from the old nfs server to the new one in a previous job, I 1st generated a list of the top level maildirs. I then generated the rsync commands + plus a few other bits and pieces for each maildir to make a single transaction like function. I then

Re: copying milllions of small files and millions of dirs

2013-08-20 Thread krad
whops that should have been ls /tmp/scripts/| while read f echo sh /tmp/scripts/$f done | xjobs -j 20 On 20 August 2013 08:32, krad kra...@gmail.com wrote: When i migrated a large mailspool in maildir format from the old nfs server to the new one in a previous job, I 1st generated a list of

Re: copying milllions of small files and millions of dirs

2013-08-20 Thread Frank Leonhardt
On 20/08/2013 08:32, krad wrote: When i migrated a large mailspool in maildir format from the old nfs server to the new one in a previous job, I 1st generated a list of the top level maildirs. I then generated the rsync commands + plus a few other bits and pieces for each maildir to make a

Re: copying milllions of small files and millions of dirs

2013-08-20 Thread Warren Block
On Mon, 19 Aug 2013, Mark Felder wrote: On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote: On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Is there a faster way to copy files over NFS? I would use find+cpio. This handles hard links, permissions, and in case of later runs,

Re: copying milllions of small files and millions of dirs

2013-08-19 Thread Mark Felder
On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote: On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Is there a faster way to copy files over NFS? I would use find+cpio. This handles hard links, permissions, and in case of later runs, will not copy files if they already exist

Re: copying milllions of small files and millions of dirs

2013-08-19 Thread aurfalien
On Aug 19, 2013, at 10:41 AM, Mark Felder wrote: On Fri, Aug 16, 2013, at 1:46, Nicolas KOWALSKI wrote: On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Is there a faster way to copy files over NFS? I would use find+cpio. This handles hard links, permissions, and in case of

Re: copying milllions of small files and millions of dirs

2013-08-16 Thread Nicolas KOWALSKI
On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Is there a faster way to copy files over NFS? I would use find+cpio. This handles hard links, permissions, and in case of later runs, will not copy files if they already exist on the destination. # cd /source/dir # find . | cpio -pvdm

copying milllions of small files and millions of dirs

2013-08-15 Thread aurfalien
Hi all, Is there a faster way to copy files over NFS? Currently breaking up a simple rsync over 7 or so scripts which copies 22 dirs having ~500,000 dirs or files each. Obviously reading all the meta data is a PITA. Doin 10Gb/jumbos but in this case it don't make much of a hoot of a diff.

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread aurfalien
On Aug 15, 2013, at 11:26 AM, Charles Swiger wrote: On Aug 15, 2013, at 11:13 AM, aurfalien aurfal...@gmail.com wrote: Is there a faster way to copy files over NFS? Probably. Ok, thanks for the specifics. Currently breaking up a simple rsync over 7 or so scripts which copies 22 dirs

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread aurfalien
On Aug 15, 2013, at 11:52 AM, Charles Swiger wrote: On Aug 15, 2013, at 11:37 AM, aurfalien aurfal...@gmail.com wrote: On Aug 15, 2013, at 11:26 AM, Charles Swiger wrote: On Aug 15, 2013, at 11:13 AM, aurfalien aurfal...@gmail.com wrote: Is there a faster way to copy files over NFS?

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread Charles Swiger
On Aug 15, 2013, at 11:13 AM, aurfalien aurfal...@gmail.com wrote: Is there a faster way to copy files over NFS? Probably. Currently breaking up a simple rsync over 7 or so scripts which copies 22 dirs having ~500,000 dirs or files each. There's a maximum useful concurrency which depends on

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread Adam Vande More
On Thu, Aug 15, 2013 at 1:13 PM, aurfalien aurfal...@gmail.com wrote: Hi all, Is there a faster way to copy files over NFS? Remove NFS from the setup. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread aurfalien
On Aug 15, 2013, at 12:36 PM, Adam Vande More wrote: On Thu, Aug 15, 2013 at 1:13 PM, aurfalien aurfal...@gmail.com wrote: Hi all, Is there a faster way to copy files over NFS? Remove NFS from the setup. Yea, your mouth to gods ears. My BlueArc is an NFS NAS only box. So no way to

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread Charles Swiger
On Aug 15, 2013, at 11:37 AM, aurfalien aurfal...@gmail.com wrote: On Aug 15, 2013, at 11:26 AM, Charles Swiger wrote: On Aug 15, 2013, at 11:13 AM, aurfalien aurfal...@gmail.com wrote: Is there a faster way to copy files over NFS? Probably. Ok, thanks for the specifics. You're most

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread Frank Leonhardt
On 15/08/2013 19:13, aurfalien wrote: Hi all, Is there a faster way to copy files over NFS? Currently breaking up a simple rsync over 7 or so scripts which copies 22 dirs having ~500,000 dirs or files each. I'm reading all this with interest. The first thing I'd have tried would be tar

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread Charles Swiger
[ ...combining replies for brevity... ] On Aug 15, 2013, at 1:02 PM, Frank Leonhardt fra...@fjl.co.uk wrote: I'm reading all this with interest. The first thing I'd have tried would be tar (and probably netcat) but I'm a probably bit of a dinosaur. (If someone wants to buy me some really big

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread Roland Smith
On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Hi all, Is there a faster way to copy files over NFS? Can you log into your NAS with ssh or telnet? It so I would suggest using tar(1) and nc(1). It has been a while since I measured it, but IIRC the combination of tar (without

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread aurfalien
On Aug 15, 2013, at 1:35 PM, Roland Smith wrote: On Thu, Aug 15, 2013 at 11:13:25AM -0700, aurfalien wrote: Hi all, Is there a faster way to copy files over NFS? Can you log into your NAS with ssh or telnet? I can but thats a back channel link of 100Mb link. - aurf

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread aurfalien
On Aug 15, 2013, at 1:22 PM, Charles Swiger wrote: [ ...combining replies for brevity... ] On Aug 15, 2013, at 1:02 PM, Frank Leonhardt fra...@fjl.co.uk wrote: I'm reading all this with interest. The first thing I'd have tried would be tar (and probably netcat) but I'm a probably bit of a

Re: copying milllions of small files and millions of dirs

2013-08-15 Thread iamatt
I would use ndmp. That is how we archive our nas crap isilon stuff but we have the backend accelerators Not sure if there is ndmp for FreeBSD. Like another poster said you are most likely i/o bound anyway. On Thu, Aug 15, 2013 at 2:14 PM, aurfalien aurfal...@gmail.com wrote: On Aug