UFS2 optimization for many small files

2008-03-12 Thread Angelo Turetta
I recently upgraded the disk of my mail server. The server was initially installed with a single 36GB RAID1 volume with FreeBSD 5 (summer 2004). Over the years I upgraded to FreeBSD 6, and some months ago I added another 36GB RAID1 volume and one 72GB RAID1 volume. I then proceeded to copy my

Re: UFS2 optimization for many small files

2008-03-12 Thread Angelo Turetta
Chuck Swiger wrote: On Mar 12, 2008, at 11:44 AM, Angelo Turetta wrote: I tried understanding where the difference was, but I cannot work-out any cause in the file systems: I believe Cyrus will create hard links if the same email message is kept in multiple folders. Do you know if this

Re: UFS2 optimization for many small files

2008-03-12 Thread Chuck Swiger
On Mar 12, 2008, at 11:44 AM, Angelo Turetta wrote: I then proceeded to copy my cyrus-imapd partition from /usr/local/ mail (on /dev/da0s1f) to the new 76GB /mail (/dev/da2s1d). During this copy I noticed the disk usage of the mailboxes (as reported by du(8)) growing about 20% larger in the

Re: UFS2 optimization for many small files

2008-03-12 Thread Chuck Swiger
On Mar 12, 2008, at 12:23 PM, Angelo Turetta wrote: Chuck Swiger wrote: On Mar 12, 2008, at 11:44 AM, Angelo Turetta wrote: I tried understanding where the difference was, but I cannot work- out any cause in the file systems: I believe Cyrus will create hard links if the same email message is

Re: UFS2 optimization for many small files

2008-03-12 Thread Lars Kristiansen
Chuck Swiger skrev: On Mar 12, 2008, at 12:23 PM, Angelo Turetta wrote: Chuck Swiger wrote: On Mar 12, 2008, at 11:44 AM, Angelo Turetta wrote: I tried understanding where the difference was, but I cannot work-out any cause in the file systems: I believe Cyrus will create hard links if the

Re: UFS2 optimization for many small files

2007-07-01 Thread Claus Guttesen
We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if

Re: UFS2 optimization for many small files

2007-07-01 Thread Wojciech Puchar
We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if

Re: UFS2 optimization for many small files

2007-07-01 Thread Wojciech Puchar
approx. 15 partitions ranging from 400 GB to 2 TB in size. If the server for some reason had crashed the webservers were unable to the question is about the reason it crashed... access the nfs-mounted partitions during the period the server did a snapshot of a partition, in order to perform

Re: UFS2 optimization for many small files

2007-07-01 Thread Wojciech Puchar
I have tried using a 4K/0.5K UFS1 filesystem in the past and found the performance was very poor. UFS2 was based on 16K/2K and I would expect it to perform even worse with 4K/0.5K. I would suggest you try 8K/1K. not for small files. you are light with large files but it's not THAT bad as

Re: UFS2 optimization for many small files

2007-07-01 Thread Wojciech Puchar
Thank you very much. Try zfs on amd64 unless your app doesn't work well with zfs or your does zfs have RELIABLE and USABLE software allowing to efficiently backup large filesystems to other media? (DVD's, tapes, other hard discs) ___

Re: UFS2 optimization for many small files

2007-07-01 Thread Claus Guttesen
approx. 15 partitions ranging from 400 GB to 2 TB in size. If the server for some reason had crashed the webservers were unable to the question is about the reason it crashed... access the nfs-mounted partitions during the period the server did a snapshot of a partition, in order to

Re: UFS2 optimization for many small files

2007-07-01 Thread Claus Guttesen
Try zfs on amd64 unless your app doesn't work well with zfs or your does zfs have RELIABLE and USABLE software allowing to efficiently backup large filesystems to other media? (DVD's, tapes, other hard discs) Zfs has send/receive where you can do snapshots and send them to a different host.

Re: UFS2 optimization for many small files

2007-07-01 Thread Wojciech Puchar
snapshot of a partition, in order to perform a background-fsck and thus our website was down. So ufs2 does not scale well. Reasons not related to the nfs-server itself. FreeBSD itself was rock-solid. It was firmware-related on the storage-side. i always use software mirror concat or both in

Re: UFS2 optimization for many small files

2007-07-01 Thread Nikolay Pavlov
On Wednesday, 27 June 2007 at 14:11:19 +0400, Nguyen Tam Chinh wrote: Greetings, We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This

Re: UFS2 optimization for many small files

2007-07-01 Thread Nguyen Tam Chinh
On 7/2/07, Nikolay Pavlov [EMAIL PROTECTED] wrote: On Wednesday, 27 June 2007 at 14:11:19 +0400, Nguyen Tam Chinh wrote: Greetings, We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system

Re: UFS2 optimization for many small files

2007-06-30 Thread Peter Jeremy
On 2007-Jun-27 14:11:19 +0400, Nguyen Tam Chinh [EMAIL PROTECTED] wrote: We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. Short answer: No. Longer answer: FreeBSD and UFS2 have

UFS2 optimization for many small files

2007-06-27 Thread Nguyen Tam Chinh
Greetings, We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd

Re: UFS2 optimization for many small files

2007-06-27 Thread Bill Moran
In response to Nguyen Tam Chinh [EMAIL PROTECTED]: We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k

Re: UFS2 optimization for many small files

2007-06-27 Thread Philipp Ost
Nguyen Tam Chinh wrote: [snipped] Please advice with optimizations or tricks. [...] Did you already looked at 'man 7 tuning'? HTH, Philipp -- www.familie-ost.info/~pj ___ freebsd-questions@freebsd.org mailing list