Re: zpool doesn't upgrade - Re: ZFS directory with a large number of files

2011-08-09 Thread seanr...@gmail.com
On Tue, Aug 2, 2011 at 8:59 PM, Ronald Klop ronald-freeb...@klop.yi.org wrote: On Tue, 02 Aug 2011 12:55:43 +0200, seanr...@gmail.com seanr...@gmail.com wrote: I think this zpool upgrade thing is weird. Can you try 'zpool upgrade -a'? Mine says: zpool get version zroot NAME   PROPERTY  

Re: ZFS directory with a large number of files

2011-08-09 Thread Damien Fleuriot
On 8/2/11 9:39 AM, seanr...@gmail.com wrote: Hi there, I Googled around and checked the PRs and wasn't successful in finding any reports of what I'm seeing. I'm hoping someone here can help me debug what's going on. On my FreeBSD 8.2-S machine (built circa 12th June), I created a

Re: zpool doesn't upgrade - Re: ZFS directory with a large number of files

2011-08-09 Thread Ronald Klop
On Tue, 09 Aug 2011 10:38:01 +0200, seanr...@gmail.com seanr...@gmail.com wrote: On Tue, Aug 2, 2011 at 8:59 PM, Ronald Klop ronald-freeb...@klop.yi.org wrote: On Tue, 02 Aug 2011 12:55:43 +0200, seanr...@gmail.com seanr...@gmail.com wrote: I think this zpool upgrade thing is weird. Can

Re: ZFS directory with a large number of files

2011-08-08 Thread Daniel Kalchev
On 06.08.11 09:24, Gary Palmer wrote: Its been quite a while since I worked on the filesystem stuff in any detail but I believe, at least for UFS, it doesn't GC the directory, just truncate it if enough of the entries at the end are deleted to free up at least one fragment or block. This

Re: ZFS directory with a large number of files

2011-08-07 Thread Sean Rees
On Aug 6, 2011, at 07:24, Gary Palmer wrote: On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: On 08/05/2011 20:38, Daniel O'Connor wrote: Ahh, but OP had moved these files away and performance was still poor.. _that_ is the bug. I'm no file system expert, but it seems to me

Re: ZFS directory with a large number of files

2011-08-07 Thread seanr...@gmail.com
On Sun, Aug 7, 2011 at 10:20 AM, Sean Rees seanr...@gmail.com wrote: On Aug 6, 2011, at 07:24, Gary Palmer wrote: On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: On 08/05/2011 20:38, Daniel O'Connor wrote: Ahh, but OP had moved these files away and performance was still poor..

Re: ZFS directory with a large number of files

2011-08-06 Thread Gary Palmer
On Fri, Aug 05, 2011 at 08:56:36PM -0700, Doug Barton wrote: On 08/05/2011 20:38, Daniel O'Connor wrote: Ahh, but OP had moved these files away and performance was still poor.. _that_ is the bug. I'm no file system expert, but it seems to me the key questions are; how long does it take

Re: ZFS directory with a large number of files

2011-08-05 Thread Christian Weisgerber
Daniel Kalchev dan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again. UFS would be the classic example of poor performance if you

Re: ZFS directory with a large number of files

2011-08-05 Thread Patrick M. Hausen
Hi, all, Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: Daniel Kalchev dan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again.

Re: ZFS directory with a large number of files

2011-08-05 Thread Daniel O'Connor
On 06/08/2011, at 5:17, Patrick M. Hausen wrote: Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: Daniel Kalchev dan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory

Re: ZFS directory with a large number of files

2011-08-05 Thread Doug Barton
On 08/05/2011 20:38, Daniel O'Connor wrote: Ahh, but OP had moved these files away and performance was still poor.. _that_ is the bug. I'm no file system expert, but it seems to me the key questions are; how long does it take the system to recover from this condition, and if it's more than N

Re: ZFS directory with a large number of files

2011-08-05 Thread Hartmann, O.
On 08/05/11 21:47, Patrick M. Hausen wrote: Hi, all, Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: Daniel Kalchevdan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the

Re: ZFS directory with a large number of files

2011-08-03 Thread Peter Jeremy
On 2011-Aug-02 08:39:03 +0100, seanr...@gmail.com seanr...@gmail.com wrote: On my FreeBSD 8.2-S machine (built circa 12th June), I created a directory and populated it over the course of 3 weeks with about 2 million individual files. As you might imagine, a 'ls' of this directory took quite some

ZFS directory with a large number of files

2011-08-02 Thread seanr...@gmail.com
Hi there, I Googled around and checked the PRs and wasn't successful in finding any reports of what I'm seeing. I'm hoping someone here can help me debug what's going on. On my FreeBSD 8.2-S machine (built circa 12th June), I created a directory and populated it over the course of 3 weeks with

Re: ZFS directory with a large number of files

2011-08-02 Thread Ronald Klop
Not an in depth solution for ZFS, but maybe a solution for you. mkdir images2 mv images/* images2 rmdir images Ronald. On Tue, 02 Aug 2011 09:39:03 +0200, seanr...@gmail.com seanr...@gmail.com wrote: Hi there, I Googled around and checked the PRs and wasn't successful in finding any

Re: ZFS directory with a large number of files

2011-08-02 Thread Jeremy Chadwick
On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: On my FreeBSD 8.2-S machine (built circa 12th June), I created a directory and populated it over the course of 3 weeks with about 2 million individual files. I'll keep this real simple: Why did you do this? I hope this was a

Re: ZFS directory with a large number of files

2011-08-02 Thread seanr...@gmail.com
inline On Tue, Aug 2, 2011 at 10:08 AM, Jeremy Chadwick free...@jdc.parodius.com wrote: On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: On my FreeBSD 8.2-S machine (built circa 12th June), I created a directory and populated it over the course of 3 weeks with about 2

Re: ZFS directory with a large number of files

2011-08-02 Thread C. P. Ghost
On Tue, Aug 2, 2011 at 9:39 AM, seanr...@gmail.com seanr...@gmail.com wrote: On my FreeBSD 8.2-S machine (built circa 12th June), I created a directory and populated it over the course of 3 weeks with about 2 million individual files. As you might imagine, a 'ls' of this directory took quite

Re: ZFS directory with a large number of files

2011-08-02 Thread Jeremy Chadwick
On Tue, Aug 02, 2011 at 10:16:35AM +0100, seanr...@gmail.com wrote: On Tue, Aug 2, 2011 at 10:08 AM, Jeremy Chadwick free...@jdc.parodius.com wrote: On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: On my FreeBSD 8.2-S machine (built circa 12th June), I created a

Re: ZFS directory with a large number of files

2011-08-02 Thread Daniel O'Connor
On 02/08/2011, at 19:12, Jeremy Chadwick wrote: When I was being taught the ropes of system administration at Oregon State, the team of crotchety UNIX admins there made it quite clear that there were things you just Did Not Do(tm) to computer systems. Shoving thousands of files into a single

Re: ZFS directory with a large number of files

2011-08-02 Thread Daniel O'Connor
On 02/08/2011, at 18:38, Jeremy Chadwick wrote: On Tue, Aug 02, 2011 at 08:39:03AM +0100, seanr...@gmail.com wrote: On my FreeBSD 8.2-S machine (built circa 12th June), I created a directory and populated it over the course of 3 weeks with about 2 million individual files. I'll keep this

Re: ZFS directory with a large number of files

2011-08-02 Thread Daniel Kalchev
On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again. UFS would be the classic example of poor performance if you do this. If it is a limitation in ZFS it

Re: ZFS directory with a large number of files

2011-08-02 Thread seanr...@gmail.com
On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev dan...@digsys.bg wrote: If it is a limitation in ZFS it would be nice to know that, perhaps it truly, really is a bug that can be avoided (or it's inherent in the way ZFS handles such things) It is possible  that there is not enough memory in

Re: ZFS directory with a large number of files

2011-08-02 Thread Jeremy Chadwick
On Tue, Aug 02, 2011 at 11:55:43AM +0100, seanr...@gmail.com wrote: On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev dan...@digsys.bg wrote: If it is a limitation in ZFS it would be nice to know that, perhaps it truly, really is a bug that can be avoided (or it's inherent in the way ZFS

Re: ZFS directory with a large number of files

2011-08-02 Thread seanr...@gmail.com
On Tue, Aug 2, 2011 at 12:07 PM, Jeremy Chadwick free...@jdc.parodius.com wrote: On Tue, Aug 02, 2011 at 11:55:43AM +0100, seanr...@gmail.com wrote: On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev dan...@digsys.bg wrote: If it is a limitation in ZFS it would be nice to know that, perhaps it

Re: ZFS directory with a large number of files

2011-08-02 Thread Matthew Seaman
On 02/08/2011 11:10, Daniel Kalchev wrote: Other than that, perhaps in ZFS it would be easier to prune the unused directory entries, than it is in UFS. It looks like this is not implemented. Remember that ZFS uses copy-on-write for all filesystem updates. Any change to a directory contents

zpool doesn't upgrade - Re: ZFS directory with a large number of files

2011-08-02 Thread Ronald Klop
On Tue, 02 Aug 2011 12:55:43 +0200, seanr...@gmail.com seanr...@gmail.com wrote: On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev dan...@digsys.bg wrote: If it is a limitation in ZFS it would be nice to know that, perhaps it truly, really is a bug that can be avoided (or it's inherent in the