Re: ffs dirpref speedup
There's a (maybe overly simple) avoidance algorithm for this. It attempts to ensure that it doesn't end up filling up cgs with directories and no files. There are also two tunables int32_t fs_avgfilesize; /* expected average file size */ int32_t fs_avgfpdir; /* expected # of files per directory */ aimed at being able to tune things such as squid caches (and I would guess the ports tree). > We discussed this a while back; it has some interesting (and in some > cases) undesirable side-effects. FFS tries to balance directories across > CGs in order to balance the use of CGs for file allocation. The approach > being advocated here will tend to use CGs one at a time, resulting in > poor distribution of files and corresponding fragmentation problems. > > > I wonder whether FreeBSD has this improvement or not. > > > > With softupdates or async mounted filesystems it seems that the speedup is > > very big... > > > > Thanks, > > -- > > Attila Nagye-mail: [EMAIL PROTECTED] > > Budapest Polytechnic (BMF.HU) @work: +361 210 1415 (194) > > H-1084 Budapest, Tavaszmezo u. 15-17. cell.: +3630 306 6758 > > -- Forwarded message -- > > Date: Sat, 7 Apr 2001 02:02:21 +0400 > > From: Grigoriy Orlov <[EMAIL PROTECTED]> > > To: [EMAIL PROTECTED] > > Subject: Re: CVS: cvs.openbsd.org: src > > > > On Fri, Apr 06, 2001 at 10:27:55PM +0100, Brian Somers wrote: > > > > CVSROOT:/cvs > > > > Module name:src > > > > Changes by: [EMAIL PROTECTED]2001/04/06 14:43:31 > > > > > > > > Modified files: > > > > sys/ufs/ffs: fs.h ffs_alloc.c ffs_vfsops.c > > > > sbin/fsck_ffs : setup.c > > > > sbin/tunefs: tunefs.c > > > > > > > > Log message: > > > > Replace FFS directory preference algorithm(dirpref) by new one. > > > > It allocates directory inode in the same cylinder group as a parent > > > > directory in. This speedup file/directory intensive operations on > > > > a big file systems in times. > > > > > > > > Don't forget to recompile fsck_ffs with updated fs.h or you will get > > > > "VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE" at > > > > the next boot. In any case you can ignore this error safely. > > > > > > > > Requested by deraadt@ > > > > > > Do you have any numbers or statistics ? Just curious as to how > > > big/small the gain is > > > > These results is old and I improve algorithm after these tests was done. > > Nevertheless they show how big may be perfomance speedup. I have done two > > file/directory intensive tests on a two OpenBSD systems with old and new > > dirpref algorithm. The first test is "tar -xzf ports.tar.gz", the second > > is "rm -rf ports". There ports.tar.gz - port collection from OpenBSD 2.8 > > release, it contains 6596 dirs and 13868 files. The test systems are: > > > > 1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for > >test is at wd1. Size of test file system is 8 Gb, number of cg=991, > >size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current > >from Dec 2000 with BUFCACHEPERCENT=35 > > > > 2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system > >at wd0, file system for test is at wd1. Size of test file system is 40 Gb, > >number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k > >OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50 > > > > Test Results > > > > tar -xzf ports.tar.gz rm -rf ports > > mode old dirpref new dirpref speedup old dirpref new dirpref speedup > > First system > > normal 667 472 1.41 477331 1.44 > > async 285 144 1.98 130 14 9.29 > > sync 768 616 1.25 477334 1.43 > > softdep413 252 1.64 241 38 6.34 > > Second system > > normal 329 81 4.06 263.5 93.5 2.81 > > async 302 25.711.75 112 2.26 49.56 > > sync 281 57.0 4.93 263 90.5 2.9 > > softdep341 40.6 8.4284 4.76 59.66 > > > > "old dirpref" and "new dirpref" columns give a test time in seconds. > > speedup - speed increasement in times, ie. old dirpref / new dirpref. > > - > > > > If you want a more detailed algorithm description, please sent mail > > to me directly. > > > > Grigoriy. > > > > > > > > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > > with "unsubscribe freebsd-hackers" in the body of the message > > -- > ... every activity meets with opposition, everyone who acts has his > rivals
Re: ffs dirpref speedup
We discussed this a while back; it has some interesting (and in some cases) undesirable side-effects. FFS tries to balance directories across CGs in order to balance the use of CGs for file allocation. The approach being advocated here will tend to use CGs one at a time, resulting in poor distribution of files and corresponding fragmentation problems. > I wonder whether FreeBSD has this improvement or not. > > With softupdates or async mounted filesystems it seems that the speedup is > very big... > > Thanks, > -- > Attila Nagye-mail: [EMAIL PROTECTED] > Budapest Polytechnic (BMF.HU) @work: +361 210 1415 (194) > H-1084 Budapest, Tavaszmezo u. 15-17. cell.: +3630 306 6758 > -- Forwarded message -- > Date: Sat, 7 Apr 2001 02:02:21 +0400 > From: Grigoriy Orlov <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Subject: Re: CVS: cvs.openbsd.org: src > > On Fri, Apr 06, 2001 at 10:27:55PM +0100, Brian Somers wrote: > > > CVSROOT: /cvs > > > Module name: src > > > Changes by: [EMAIL PROTECTED]2001/04/06 14:43:31 > > > > > > Modified files: > > > sys/ufs/ffs: fs.h ffs_alloc.c ffs_vfsops.c > > > sbin/fsck_ffs : setup.c > > > sbin/tunefs: tunefs.c > > > > > > Log message: > > > Replace FFS directory preference algorithm(dirpref) by new one. > > > It allocates directory inode in the same cylinder group as a parent > > > directory in. This speedup file/directory intensive operations on > > > a big file systems in times. > > > > > > Don't forget to recompile fsck_ffs with updated fs.h or you will get > > > "VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE" at > > > the next boot. In any case you can ignore this error safely. > > > > > > Requested by deraadt@ > > > > Do you have any numbers or statistics ? Just curious as to how > > big/small the gain is > > These results is old and I improve algorithm after these tests was done. > Nevertheless they show how big may be perfomance speedup. I have done two > file/directory intensive tests on a two OpenBSD systems with old and new > dirpref algorithm. The first test is "tar -xzf ports.tar.gz", the second > is "rm -rf ports". There ports.tar.gz - port collection from OpenBSD 2.8 > release, it contains 6596 dirs and 13868 files. The test systems are: > > 1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for >test is at wd1. Size of test file system is 8 Gb, number of cg=991, >size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current >from Dec 2000 with BUFCACHEPERCENT=35 > > 2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system >at wd0, file system for test is at wd1. Size of test file system is 40 Gb, >number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k >OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50 > > Test Results > > tar -xzf ports.tar.gz rm -rf ports > mode old dirpref new dirpref speedup old dirpref new dirpref speedup > First system > normal 667 472 1.41 477331 1.44 > async 285 144 1.98 130 14 9.29 > sync 768 616 1.25 477334 1.43 > softdep413 252 1.64 241 38 6.34 > Second system > normal 329 81 4.06 263.5 93.5 2.81 > async 302 25.711.75 112 2.26 49.56 > sync 281 57.0 4.93 263 90.5 2.9 > softdep341 40.6 8.4284 4.76 59.66 > > "old dirpref" and "new dirpref" columns give a test time in seconds. > speedup - speed increasement in times, ie. old dirpref / new dirpref. > - > > If you want a more detailed algorithm description, please sent mail > to me directly. > > Grigoriy. > > > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
ffs dirpref speedup
Hello, I wonder whether FreeBSD has this improvement or not. With softupdates or async mounted filesystems it seems that the speedup is very big... Thanks, -- Attila Nagye-mail: [EMAIL PROTECTED] Budapest Polytechnic (BMF.HU) @work: +361 210 1415 (194) H-1084 Budapest, Tavaszmezo u. 15-17. cell.: +3630 306 6758 -- Forwarded message -- Date: Sat, 7 Apr 2001 02:02:21 +0400 From: Grigoriy Orlov <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Subject: Re: CVS: cvs.openbsd.org: src On Fri, Apr 06, 2001 at 10:27:55PM +0100, Brian Somers wrote: > > CVSROOT:/cvs > > Module name:src > > Changes by: [EMAIL PROTECTED]2001/04/06 14:43:31 > > > > Modified files: > > sys/ufs/ffs: fs.h ffs_alloc.c ffs_vfsops.c > > sbin/fsck_ffs : setup.c > > sbin/tunefs: tunefs.c > > > > Log message: > > Replace FFS directory preference algorithm(dirpref) by new one. > > It allocates directory inode in the same cylinder group as a parent > > directory in. This speedup file/directory intensive operations on > > a big file systems in times. > > > > Don't forget to recompile fsck_ffs with updated fs.h or you will get > > "VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE" at > > the next boot. In any case you can ignore this error safely. > > > > Requested by deraadt@ > > Do you have any numbers or statistics ? Just curious as to how > big/small the gain is These results is old and I improve algorithm after these tests was done. Nevertheless they show how big may be perfomance speedup. I have done two file/directory intensive tests on a two OpenBSD systems with old and new dirpref algorithm. The first test is "tar -xzf ports.tar.gz", the second is "rm -rf ports". There ports.tar.gz - port collection from OpenBSD 2.8 release, it contains 6596 dirs and 13868 files. The test systems are: 1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for test is at wd1. Size of test file system is 8 Gb, number of cg=991, size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=35 2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system at wd0, file system for test is at wd1. Size of test file system is 40 Gb, number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50 Test Results tar -xzf ports.tar.gz rm -rf ports mode old dirpref new dirpref speedup old dirpref new dirpref speedup First system normal 667 472 1.41 477331 1.44 async 285 144 1.98 130 14 9.29 sync 768 616 1.25 477334 1.43 softdep413 252 1.64 241 38 6.34 Second system normal 329 81 4.06 263.5 93.5 2.81 async 302 25.711.75 112 2.26 49.56 sync 281 57.0 4.93 263 90.5 2.9 softdep341 40.6 8.4284 4.76 59.66 "old dirpref" and "new dirpref" columns give a test time in seconds. speedup - speed increasement in times, ie. old dirpref / new dirpref. - If you want a more detailed algorithm description, please sent mail to me directly. Grigoriy. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
