Re: Bad performance when accessing a lot of small files

2007-12-29 Thread Alexandre Biancalana
On Dec 28, 2007 7:38 PM, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > > * Alexandre Biancalana <[EMAIL PROTECTED]> [071225 16:45] wrote: > > On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > > > > > Have you tried the 'noatime' mount option? That should help. > > > > Yeah, the zfs set at

Re: Bad performance when accessing a lot of small files

2007-12-28 Thread Alfred Perlstein
* Alexandre Biancalana <[EMAIL PROTECTED]> [071225 16:45] wrote: > On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > > > Have you tried the 'noatime' mount option? That should help. > > Yeah, the zfs set atime=off was already done > > > > > Can you provide a histogram of the count

Re: Bad performance when accessing a lot of small files

2007-12-28 Thread Alexandre Biancalana
On 12/25/07, Alexandre Biancalana <[EMAIL PROTECTED]> wrote: > On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > > > Have you tried the 'noatime' mount option? That should help. > > Yeah, the zfs set atime=off was already done > > > > > Can you provide a histogram of the count of file

Re: Bad performance when accessing a lot of small files

2007-12-25 Thread Alexandre Biancalana
On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > Have you tried the 'noatime' mount option? That should help. Yeah, the zfs set atime=off was already done > > Can you provide a histogram of the count of files per directory? Excuse-me, but I don't understand __

Re: Bad performance when accessing a lot of small files

2007-12-21 Thread Alfred Perlstein
* Alexandre Biancalana <[EMAIL PROTECTED]> [071221 15:53] wrote: > On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > > > > What application are you scanning these files with? I know I had > > issues with rsync in particular where I had to have it rsync > > smaller pieces of a collection f

Re: Bad performance when accessing a lot of small files

2007-12-21 Thread Alexandre Biancalana
On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > * Alexandre Biancalana <[EMAIL PROTECTED]> [071221 12:48] wrote: > > On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > > > > Hi Alfred ! > > > > > > > > There is a lot of very good tuning advice in this thread, however > > > one th

Re: Bad performance when accessing a lot of small files

2007-12-21 Thread Alfred Perlstein
* Alexandre Biancalana <[EMAIL PROTECTED]> [071221 12:48] wrote: > On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: > > Hi Alfred ! > > > > > There is a lot of very good tuning advice in this thread, however > > one thing to note is that having ~1 million files in a directory > > is not a

Re: Bad performance when accessing a lot of small files

2007-12-21 Thread Alexandre Biancalana
On 12/21/07, Alfred Perlstein <[EMAIL PROTECTED]> wrote: Hi Alfred ! > > There is a lot of very good tuning advice in this thread, however > one thing to note is that having ~1 million files in a directory > is not a very good thing to do on just about any filesystem. I think I was not clear, I

Re: Bad performance when accessing a lot of small files

2007-12-21 Thread Alfred Perlstein
* Alexandre Biancalana <[EMAIL PROTECTED]> [071219 11:35] wrote: > Hi List, > > I have a backup server running FreeBSD 7-BETA3. The cpu is CPU: > Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz, 3GB Ram, 10x 500GB > SATA, Areca 1231-ML, the filesystem used to backup my other servers > locally is

Re: Re[2]: Bad performance when accessing a lot of small files

2007-12-20 Thread Alexandre Biancalana
On 12/20/07, dima <[EMAIL PROTECTED]> wrote: > >> Increase the dirhash maxmem > >> I have 'sysctl -w vfs.ufs.dirhash_maxmem=268435456' (256Mb)in my > >> rc.local on a (maildir) fileserver here > >> > > This is a zfs filesystem, by the way I raise this limit as suggested. > > > > I let you know > >

Re[2]: Bad performance when accessing a lot of small files

2007-12-20 Thread dima
>> Increase the dirhash maxmem >> I have 'sysctl -w vfs.ufs.dirhash_maxmem=268435456' (256Mb)in my >> rc.local on a (maildir) fileserver here >> > This is a zfs filesystem, by the way I raise this limit as suggested. > > I let you know Can you provide sysctl vfs.zfs output then? ZFS can be tuned

Re: Re[2]: Bad performance when accessing a lot of small files

2007-12-20 Thread Alexandre Biancalana
On 12/20/07, dima <[EMAIL PROTECTED]> wrote: > > Try to run this command while listing your large directory. You'll see that > dirhash_mem will reach dirhash_maxmem. So, the value of dirhash_maxmem should > be increased. > > One more hint. Mounting filesystems with -noatime option greatly improve

Re: Bad performance when accessing a lot of small files

2007-12-20 Thread Alexandre Biancalana
On 12/20/07, Dennis Kjær Jensen <[EMAIL PROTECTED]> wrote: > Alexandre Biancalana wrote: > > > > Increase the dirhash maxmem > I have 'sysctl -w vfs.ufs.dirhash_maxmem=268435456' (256Mb)in my > rc.local on a (maildir) fileserver here > This is a zfs filesystem, by the way I raise this limit as sugg

Re[2]: Bad performance when accessing a lot of small files

2007-12-20 Thread dima
>>>The behavior that I'm observing and that want your help is when the >>>system is accessing some directory with many small files ( directories >>>with ~ 1 million of ~30kb files), the performance is very poor. >> >> Hi, > > I'm using zfs, I think this change the things.. no ? > > > Hav

Re: Bad performance when accessing a lot of small files

2007-12-19 Thread Mike Tancsa
At 03:07 PM 12/19/2007, Alexandre Biancalana wrote: On 12/19/07, Mike Tancsa <[EMAIL PROTECTED]> wrote: > At 02:09 PM 12/19/2007, Alexandre Biancalana wrote: > > >The behavior that I'm observing and that want your help is when the > >system is accessing some directory with many small files ( dire

Re: Bad performance when accessing a lot of small files

2007-12-19 Thread Alexandre Biancalana
On 12/19/07, Mike Tancsa <[EMAIL PROTECTED]> wrote: > At 02:09 PM 12/19/2007, Alexandre Biancalana wrote: > > >The behavior that I'm observing and that want your help is when the > >system is accessing some directory with many small files ( directories > >with ~ 1 million of ~30kb files), the perfo

Re: Bad performance when accessing a lot of small files

2007-12-19 Thread Mike Tancsa
At 02:09 PM 12/19/2007, Alexandre Biancalana wrote: The behavior that I'm observing and that want your help is when the system is accessing some directory with many small files ( directories with ~ 1 million of ~30kb files), the performance is very poor. Hi, Have you adjusted the dirha

Bad performance when accessing a lot of small files

2007-12-19 Thread Alexandre Biancalana
Hi List, I have a backup server running FreeBSD 7-BETA3. The cpu is CPU: Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz, 3GB Ram, 10x 500GB SATA, Areca 1231-ML, the filesystem used to backup my other servers locally is build on top of ARC-1231, 4TB (32k stripe) zfs filesystem with gzip compress