Re: zpool doesn't upgrade - Re: ZFS directory with a large number of files

2011-08-09 Thread Ronald Klop
On Tue, 09 Aug 2011 10:38:01 +0200, seanr...@gmail.com wrote: On Tue, Aug 2, 2011 at 8:59 PM, Ronald Klop wrote: On Tue, 02 Aug 2011 12:55:43 +0200, seanr...@gmail.com wrote: I think this zpool upgrade thing is weird. Can you try 'zpool upgrade -a'? Mine says: zpool get version zroo

Re: ZFS directory with a large number of files

2011-08-09 Thread Damien Fleuriot
On 8/2/11 9:39 AM, seanr...@gmail.com wrote: > Hi there, > > I Googled around and checked the PRs and wasn't successful in finding > any reports of what I'm seeing. I'm hoping someone here can help me > debug what's going on. > > On my FreeBSD 8.2-S machine (built circa 12th June), I created a >

Re: zpool doesn't upgrade - Re: ZFS directory with a large number of files

2011-08-09 Thread seanr...@gmail.com
On Tue, Aug 2, 2011 at 8:59 PM, Ronald Klop wrote: > On Tue, 02 Aug 2011 12:55:43 +0200, seanr...@gmail.com > wrote: > I think this zpool upgrade thing is weird. Can you try 'zpool upgrade -a'? > > Mine says: > zpool get version zroot > NAME   PROPERTY  VALUE    SOURCE > zroot  version   28      

Re: ZFS directory with a large number of files

2011-08-08 Thread Daniel Kalchev
On 06.08.11 09:24, Gary Palmer wrote: Its been quite a while since I worked on the filesystem stuff in any detail but I believe, at least for UFS, it doesn't GC the directory, just truncate it if enough of the entries at the end are deleted to free up at least one fragment or block. This wa

Re: ZFS directory with a large number of files

2011-08-07 Thread seanr...@gmail.com
On Sun, Aug 7, 2011 at 10:20 AM, Sean Rees wrote: > > On Aug 6, 2011, at 07:24, Gary Palmer wrote: > >> On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: >>> On 08/05/2011 20:38, Daniel O'Connor wrote: >>> Ahh, but OP had moved these files away and performance was still poor.. >>>

Re: ZFS directory with a large number of files

2011-08-07 Thread Sean Rees
On Aug 6, 2011, at 07:24, Gary Palmer wrote: > On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: >> On 08/05/2011 20:38, Daniel O'Connor wrote: >> >>> Ahh, but OP had moved these files away and performance was still poor.. >>> _that_ is the bug. >> >> I'm no file system expert, but

Re: ZFS directory with a large number of files

2011-08-05 Thread Gary Palmer
On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: > On 08/05/2011 20:38, Daniel O'Connor wrote: > > > Ahh, but OP had moved these files away and performance was still poor.. > > _that_ is the bug. > > I'm no file system expert, but it seems to me the key questions are; how > long does

Re: ZFS directory with a large number of files

2011-08-05 Thread Hartmann, O.
On 08/05/11 21:47, Patrick M. Hausen wrote: Hi, all, Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: Daniel Kalchev wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance

Re: ZFS directory with a large number of files

2011-08-05 Thread Doug Barton
On 08/05/2011 20:38, Daniel O'Connor wrote: > Ahh, but OP had moved these files away and performance was still poor.. > _that_ is the bug. I'm no file system expert, but it seems to me the key questions are; how long does it take the system to recover from this condition, and if it's more than N

Re: ZFS directory with a large number of files

2011-08-05 Thread Daniel O'Connor
On 06/08/2011, at 5:17, Patrick M. Hausen wrote: > Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: >> Daniel Kalchev wrote: >> >>> On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory i

Re: ZFS directory with a large number of files

2011-08-05 Thread Patrick M. Hausen
Hi, all, Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: > Daniel Kalchev wrote: > >> On 02.08.11 12:46, Daniel O'Connor wrote: >>> I am pretty sure UFS does not have this problem. i.e. once you >>> delete/move the files out of the directory its performance would be >>> good again. >> >

Re: ZFS directory with a large number of files

2011-08-05 Thread Christian Weisgerber
Daniel Kalchev wrote: > On 02.08.11 12:46, Daniel O'Connor wrote: > > I am pretty sure UFS does not have this problem. i.e. once you > > delete/move the files out of the directory its performance would be > > good again. > > UFS would be the classic example of poor performance if you do this.

Re: ZFS directory with a large number of files

2011-08-02 Thread Peter Jeremy
On 2011-Aug-02 08:39:03 +0100, "seanr...@gmail.com" wrote: >On my FreeBSD 8.2-S machine (built circa 12th June), I created a >directory and populated it over the course of 3 weeks with about 2 >million individual files. As you might imagine, a 'ls' of this >directory took quite some time. > >The f

zpool doesn't upgrade - Re: ZFS directory with a large number of files

2011-08-02 Thread Ronald Klop
On Tue, 02 Aug 2011 12:55:43 +0200, seanr...@gmail.com wrote: On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev wrote: If it is a limitation in ZFS it would be nice to know that, perhaps it truly, really is a bug that can be avoided (or it's inherent in the way ZFS handles such things) I

Re: ZFS directory with a large number of files

2011-08-02 Thread Matthew Seaman
On 02/08/2011 11:10, Daniel Kalchev wrote: > Other than that, perhaps in ZFS it would be easier to prune the unused > directory entries, than it is in UFS. It looks like this is not > implemented. Remember that ZFS uses copy-on-write for all filesystem updates. Any change to a directory contents

Re: ZFS directory with a large number of files

2011-08-02 Thread seanr...@gmail.com
On Tue, Aug 2, 2011 at 12:07 PM, Jeremy Chadwick wrote: > On Tue, Aug 02, 2011 at 11:55:43AM +0100, seanr...@gmail.com wrote: >> On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev wrote: >> >> If it is a limitation in ZFS it would be nice to know that, perhaps it >> >> truly, really is a bug that ca

Re: ZFS directory with a large number of files

2011-08-02 Thread Jeremy Chadwick
On Tue, Aug 02, 2011 at 11:55:43AM +0100, seanr...@gmail.com wrote: > On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev wrote: > >> If it is a limitation in ZFS it would be nice to know that, perhaps it > >> truly, really is a bug that can be avoided (or it's inherent in the way ZFS > >> handles suc

Re: ZFS directory with a large number of files

2011-08-02 Thread seanr...@gmail.com
On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev wrote: >> If it is a limitation in ZFS it would be nice to know that, perhaps it >> truly, really is a bug that can be avoided (or it's inherent in the way ZFS >> handles such things) > > It is possible  that there is not enough memory in ARC to cach

Re: ZFS directory with a large number of files

2011-08-02 Thread Daniel Kalchev
On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again. UFS would be the classic example of poor performance if you do this. If it is a limitation in ZFS it woul

Re: ZFS directory with a large number of files

2011-08-02 Thread Daniel O'Connor
On 02/08/2011, at 18:38, Jeremy Chadwick wrote: > On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: >> On my FreeBSD 8.2-S machine (built circa 12th June), I created a >> directory and populated it over the course of 3 weeks with about 2 >> million individual files. > > I'll kee

Re: ZFS directory with a large number of files

2011-08-02 Thread Daniel O'Connor
On 02/08/2011, at 19:12, Jeremy Chadwick wrote: > When I was being taught the ropes of system administration at Oregon > State, the team of crotchety UNIX admins there made it quite clear that > there were things you just Did Not Do(tm) to computer systems. Shoving > thousands of files into a sin

Re: ZFS directory with a large number of files

2011-08-02 Thread Jeremy Chadwick
On Tue, Aug 02, 2011 at 10:16:35AM +0100, seanr...@gmail.com wrote: > On Tue, Aug 2, 2011 at 10:08 AM, Jeremy Chadwick > wrote: > > On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: > >> On my FreeBSD 8.2-S machine (built circa 12th June), I created a > >> directory and populated

Re: ZFS directory with a large number of files

2011-08-02 Thread C. P. Ghost
On Tue, Aug 2, 2011 at 9:39 AM, seanr...@gmail.com wrote: > On my FreeBSD 8.2-S machine (built circa 12th June), I created a > directory and populated it over the course of 3 weeks with about 2 > million individual files. As you might imagine, a 'ls' of this > directory took quite some time. What

Re: ZFS directory with a large number of files

2011-08-02 Thread seanr...@gmail.com
inline On Tue, Aug 2, 2011 at 10:08 AM, Jeremy Chadwick wrote: > On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: >> On my FreeBSD 8.2-S machine (built circa 12th June), I created a >> directory and populated it over the course of 3 weeks with about 2 >> million individual file

Re: ZFS directory with a large number of files

2011-08-02 Thread Jeremy Chadwick
On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: > On my FreeBSD 8.2-S machine (built circa 12th June), I created a > directory and populated it over the course of 3 weeks with about 2 > million individual files. I'll keep this real simple: Why did you do this? I hope this was

Re: ZFS directory with a large number of files

2011-08-02 Thread Ronald Klop
Not an in depth solution for ZFS, but maybe a solution for you. mkdir images2 mv images/* images2 rmdir images Ronald. On Tue, 02 Aug 2011 09:39:03 +0200, seanr...@gmail.com wrote: Hi there, I Googled around and checked the PRs and wasn't successful in finding any reports of what I'm see