Re: ZFS directory with a large number of files
Daniel Kalchev dan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again. UFS would be the classic example of poor performance if you do this. Classic indeed. UFS dirhash has pretty much taken care of this a decade ago. -- Christian naddy Weisgerber na...@mips.inka.de ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS directory with a large number of files
Hi, all, Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: Daniel Kalchev dan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again. UFS would be the classic example of poor performance if you do this. Classic indeed. UFS dirhash has pretty much taken care of this a decade ago. While dirhash is quite an improvement, it is definitely no silver bullet. When I asked Kirk McKusick at last year's EuroBSDCon if having a six-figure number of files in a single directory was a clever idea (I just had a customer who ran into that situation), he just smiled and shook his head. The directory in question was the typo3temp/pics directory that TYPO3 uses to scale images that are part of the website, so they are handed to the browser in exactly the size they are supposed to be rendered. The performance impact was quite heavy, because at some point requests started to pile up, PHP scripts did not finish in time, fcgi slots stayed used ... most of you will know that scenario. At some threshold a machine goes from loaded, maybe a bit slow, but generally responsive to no f*ing way. Best regards, Patrick ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS directory with a large number of files
On 06/08/2011, at 5:17, Patrick M. Hausen wrote: Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: Daniel Kalchev dan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again. UFS would be the classic example of poor performance if you do this. Classic indeed. UFS dirhash has pretty much taken care of this a decade ago. While dirhash is quite an improvement, it is definitely no silver bullet. When I asked Kirk McKusick at last year's EuroBSDCon if having a six-figure number of files in a single directory was a clever idea (I just had a customer who ran into that situation), he just smiled and shook his head. Ahh, but OP had moved these files away and performance was still poor.. _that_ is the bug. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS directory with a large number of files
On 08/05/2011 20:38, Daniel O'Connor wrote: Ahh, but OP had moved these files away and performance was still poor.. _that_ is the bug. I'm no file system expert, but it seems to me the key questions are; how long does it take the system to recover from this condition, and if it's more than N $periods is that a problem? We can't stop users from doing wacky stuff, but the system should be robust in the face of this. -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS directory with a large number of files
On 08/05/11 21:47, Patrick M. Hausen wrote: Hi, all, Am 05.08.2011 um 17:12 schrieb Christian Weisgerber: Daniel Kalchevdan...@digsys.bg wrote: On 02.08.11 12:46, Daniel O'Connor wrote: I am pretty sure UFS does not have this problem. i.e. once you delete/move the files out of the directory its performance would be good again. UFS would be the classic example of poor performance if you do this. Classic indeed. UFS dirhash has pretty much taken care of this a decade ago. While dirhash is quite an improvement, it is definitely no silver bullet. When I asked Kirk McKusick at last year's EuroBSDCon if having a six-figure number of files in a single directory was a clever idea (I just had a customer who ran into that situation), he just smiled and shook his head. The directory in question was the typo3temp/pics directory that TYPO3 uses to scale images that are part of the website, so they are handed to the browser in exactly the size they are supposed to be rendered. The performance impact was quite heavy, because at some point requests started to pile up, PHP scripts did not finish in time, fcgi slots stayed used ... most of you will know that scenario. At some threshold a machine goes from loaded, maybe a bit slow, but generally responsive to no f*ing way. Best regards, Patrick I have similar situation here, but with a numerical simulation software, which drops for each timestep of integration a file of all integrated objects. Since the code is adopted and not very clever written in terms of doinf its I/O, I have to deal with this. While performing dynamical high resolution integrations of several hundreds of thousand objects over a time scale of 1 Ga produces even with a larger dump-delta creates a lot of files. making those files bigger results in a situation where they are hard to analyse, so its a tradeoff situation. ZFS and UFS2 perform bad on this situation, UFS2 even more than ZFS, but also ZFS is still a pain in the ass. Oliver ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
