Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread Marc Espie
On Sun, Apr 10, 2011 at 02:40:09PM +0200, David Vasek wrote: > On Sun, 10 Apr 2011, Marc Espie wrote: > > >This is completely stupid. > > > >What do you trust more ? your file system, or fsck ? > > > >oth have bugs ! I'm sure of it ! > > > >so, if you run fsck, it's likely > >you're going to run i

Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread David Vasek
On Sun, 10 Apr 2011, Marc Espie wrote: This is completely stupid. What do you trust more ? your file system, or fsck ? oth have bugs ! I'm sure of it ! so, if you run fsck, it's likely you're going to run into fsck bugs eventually (and trying fsck on a mounted partition was really, really stu

Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread Marc Espie
On Sun, Apr 10, 2011 at 11:27:41AM +, Miod Vallat wrote: > > Now, consider this: the fs code is very heavily tested. People use it 24 > > hours > > a day, 365 days a year. > > Except on leap years, of course. Those years see even more real-life > testing happening! Good point. Maybe we shoul

Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread Miod Vallat
> Now, consider this: the fs code is very heavily tested. People use it 24 hours > a day, 365 days a year. Except on leap years, of course. Those years see even more real-life testing happening!

Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread Marc Espie
On Sat, Apr 02, 2011 at 01:45:36PM -0500, Amit Kulkarni wrote: > Hi, > > I am replying in a single email. > > I do a fsck once in a while, not regular. In the last 6-8 months I > might have done it about 5 times. And I did it multi-user the few > times I did it, but plan on doing it single user i

Re: horribly slow fsck_ffs pass1 performance

2011-04-05 Thread Andres Perera
On Tue, Apr 5, 2011 at 7:06 AM, Janne Johansson wrote: >> /forcefsck and /fastboot have nothing to do with that >> they are not even administered by the fs >> > I wasn't trying to imply the filesystem is putting the files there, nor > reading them. Rather, those two files show that > "since there

Re: horribly slow fsck_ffs pass1 performance

2011-04-05 Thread Janne Johansson
> /forcefsck and /fastboot have nothing to do with that > > they are not even administered by the fs > > I wasn't trying to imply the filesystem is putting the files there, nor reading them. Rather, those two files show that "since there is no way to mark known brokeness in a ext file system, we wr

Re: horribly slow fsck_ffs pass1 performance

2011-04-04 Thread Tobias Weingartner
On Saturday, April 2, Amit Kulkarni wrote: > > FreeBSD which is the origin of FFS does a > background fsck, and if Kirk McCusick feels so strongly I will do it > too. FreeBSD was not the origin of the FFS code. Background fsck in freebsd is mainly meant to reduce the amount of time it takes to g

Re: horribly slow fsck_ffs pass1 performance

2011-04-03 Thread Andres Perera
On Sun, Apr 3, 2011 at 4:21 AM, Janne Johansson wrote: > 2011/4/2 Benny Lofgren > >> >> I've noticed that some (all?) linux systems do uncalled-for file system >> checks at boot if no check have been made recently, but I've never >> understood this practice. It must mean they don't trust their ow

Re: horribly slow fsck_ffs pass1 performance

2011-04-03 Thread Janne Johansson
2011/4/2 Benny Lofgren > > I've noticed that some (all?) linux systems do uncalled-for file system > checks at boot if no check have been made recently, but I've never > understood this practice. It must mean they don't trust their own file > systems, > I'm quite sure this comes from the fact th

Re: horribly slow fsck_ffs pass1 performance

2011-04-02 Thread Otto Moerbeek
On Sat, Apr 02, 2011 at 01:45:36PM -0500, Amit Kulkarni wrote: > Hi, > > I am replying in a single email. > > I do a fsck once in a while, not regular. In the last 6-8 months I > might have done it about 5 times. And I did it multi-user the few > times I did it, but plan on doing it single user

Re: horribly slow fsck_ffs pass1 performance

2011-04-02 Thread Kevin Chadwick
On Sat, 02 Apr 2011 17:46:51 +0200 Benny Lofgren wrote: > It must mean they don't trust their own file > systems, which frankly I find a bit unsettling... I'd rather use a file > system that's been field proven for decades than use something thats > just come out of the experimenting shop. Hopef

Re: horribly slow fsck_ffs pass1 performance

2011-04-02 Thread Stuart Henderson
On 2011/04/02 13:45, Amit Kulkarni wrote: > > I do a fsck once in a while, not regular. In the last 6-8 months I > might have done it about 5 times. And I did it multi-user the few > times I did it, but plan on doing it single user in future and I do > plan to do it monthly. After seeing the messa

Re: horribly slow fsck_ffs pass1 performance

2011-04-02 Thread Amit Kulkarni
Hi, I am replying in a single email. I do a fsck once in a while, not regular. In the last 6-8 months I might have done it about 5 times. And I did it multi-user the few times I did it, but plan on doing it single user in future and I do plan to do it monthly. After seeing the messages when you f

Re: horribly slow fsck_ffs pass1 performance

2011-04-02 Thread Benny Lofgren
On 2011-04-01 19.03, Amit Kulkarni wrote: > Thank you Arthur and the team for a very fast turnaround! Thank you > for reducing the pain. I will schedule a fsck every month or so, > knowing it won't screw up anything and be done really quick. Why "schedule" fsck runs at all? The file system code is

Re: horribly slow fsck_ffs pass1 performance

2011-04-02 Thread Benny Lofgren
On 2011-04-01 21.48, Amit Kulkarni wrote: >> And jumping up and down after a first successful test is not a sound >> engineering principle either. [...stuff deleted...] > It turns out that I had extracted into the default firefox download > location (/home/amit/downloads I forgot exactly where) all

Re: horribly slow fsck_ffs pass1 performance

2011-04-01 Thread Amit Kulkarni
> What I don't like is that you never have given details (even when > requested) on your extremely slow original fsck which started this > thread. The last couple of years I tested fsck on many different > setups, but I never saw fsck times of 4 hours and not even finished. > So there's something s

Re: horribly slow fsck_ffs pass1 performance

2011-04-01 Thread Otto Moerbeek
fOn Fri, Apr 01, 2011 at 12:03:19PM -0500, Amit Kulkarni wrote: > Hi Otto, > > fsck -p is not possible to do in multi-user because of > > # fsck -p /extra > NO WRITE ACCESS > /dev/rwd0m: UNEXPECTED INCONSISTENCY; RUN fsck_ffs MANUALLY. Of course. What's the point of checking a mounted filesyste

Re: horribly slow fsck_ffs pass1 performance

2011-04-01 Thread Amit Kulkarni
Hi Otto, fsck -p is not possible to do in multi-user because of # fsck -p /extra NO WRITE ACCESS /dev/rwd0m: UNEXPECTED INCONSISTENCY; RUN fsck_ffs MANUALLY. I haven't checked but it probably wants to do it single user when all fs are unmounted. And it would work when fs are unclean shutdown. I

Re: horribly slow fsck_ffs pass1 performance

2011-04-01 Thread Otto Moerbeek
Op 31 mrt. 2011 om 22:25 heeft Otto Moerbeek het volgende geschreven: > On Thu, Mar 31, 2011 at 10:14:46PM +0200, Otto Moerbeek wrote: > >> So here's an initial, only lightly tested diff. >> >> Beware, this very well could eat your filesystems. >> >> To note any difference, you should use the -p

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Fri, Apr 01, 2011 at 06:54:37AM +0200, Otto Moerbeek wrote: > > A request to tech@, > > Can you please consider architecture wise bumps in filesystem defaults > > to data block size and fragment size such that inodes to be scanned > > are reduced? When you run fsck you really want it to be corr

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Thu, Mar 31, 2011 at 10:11:37PM -0500, Amit Kulkarni wrote: > Hi Otto, > > The speedup is very considerable! Thanks a ton for this diff! I didn't > find any problems running it on all my fs. I cannot get fsck -p, maybe > you have to run in single user mode? But still normal fsck works on > all

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Amit Kulkarni
Hi Otto, The speedup is very considerable! Thanks a ton for this diff! I didn't find any problems running it on all my fs. I cannot get fsck -p, maybe you have to run in single user mode? But still normal fsck works on all partitions and with much more speed than before. I changed all my large par

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Amit Kulkarni
> I dont think we want to change thed default density. Larger > parttitions already gets larger blocks and fragment, and as a > consequence lower number of inodes. > >> Otto, >> In my tests on AMD64, if FFS partition size increases beyond 30GB, >> fsck starts taking exponential time even if you ha

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Amit Kulkarni
>> If you really have a lot of used inodes, skipping the unused ones >> isn't going to help :-) >> >> You could always build your large-sized filesystems with a larger >> value of bytes-per-inode. newfs -i 32768 or 65536 is good for common >> filesystem use patterns with larger partitions (for spec

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Thu, Mar 31, 2011 at 10:12:07PM +0200, Otto Moerbeek wrote: > > So, if I read it correctly, setting just the block size higher to say > > 64Kb does auto tune frag size to 1/8 which is 8Kb (newfs complains > > appropriately) but the auto tune inode length to 4 times frag which is > > 32Kb is no

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Thu, Mar 31, 2011 at 10:14:46PM +0200, Otto Moerbeek wrote: > So here's an initial, only lightly tested diff. > > Beware, this very well could eat your filesystems. > > To note any difference, you should use the -p mode of fsck_ffs (rc > does that) and the fs should have been mounted with sof

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
So here's an initial, only lightly tested diff. Beware, this very well could eat your filesystems. To note any difference, you should use the -p mode of fsck_ffs (rc does that) and the fs should have been mounted with softdep. I have seen very nice speedups already. -Otto Index: dir.c

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Thu, Mar 31, 2011 at 02:50:36PM -0500, Amit Kulkarni wrote: > >> > >> If you really have a lot of used inodes, skipping the unused ones > >> isn't going to help :-) > >> > >> You could always build your large-sized filesystems with a larger > >> value of bytes-per-inode. newfs -i 32768 or 65536

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Marco Peereboom
On Thu, Mar 31, 2011 at 03:15:59PM +0100, Stuart Henderson wrote: > On 2011/03/31 08:29, Marco Peereboom wrote: > > On Thu, Mar 31, 2011 at 09:13:41AM +, Stuart Henderson wrote: > > > On 2011-03-31, Otto Moerbeek wrote: > > > > On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: > >

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Stuart Henderson
On 2011/03/31 08:29, Marco Peereboom wrote: > On Thu, Mar 31, 2011 at 09:13:41AM +, Stuart Henderson wrote: > > On 2011-03-31, Otto Moerbeek wrote: > > > On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: > > >> In fsck_ffs's pass1.c it just takes forever for large sized partitions

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Marco Peereboom
On Thu, Mar 31, 2011 at 09:13:41AM +, Stuart Henderson wrote: > On 2011-03-31, Otto Moerbeek wrote: > > On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: > >> In fsck_ffs's pass1.c it just takes forever for large sized partitions > >> and also if you have very high number of files

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Thu, Mar 31, 2011 at 12:30:29PM +0200, Benny Lofgren wrote: > For example, this is what one of my file systems looks like right now: > > skynet:~# df -ih /u0 > Filesystem SizeUsed Avail Capacity iused ifree %iused > Mounted on > /dev/raid1a 12.6T7.0T5.5T56% 881220 2

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Stuart Henderson
On 2011/03/31 12:46, Otto Moerbeek wrote: > > > > In general, the default values and algorithms for allocations could > > probably do with a tune-up, since of course today's disks are several > > magnitudes larger than only a few years ago (let alone than those that > > were around when the bulk o

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Thu, Mar 31, 2011 at 12:30:29PM +0200, Benny Lofgren wrote: > On 2011-03-31 11.13, Stuart Henderson wrote: > > On 2011-03-31, Otto Moerbeek wrote: > >> On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: > >>> In fsck_ffs's pass1.c it just takes forever for large sized partitions >

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Thu, Mar 31, 2011 at 09:13:41AM +, Stuart Henderson wrote: > On 2011-03-31, Otto Moerbeek wrote: > > On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: > >> In fsck_ffs's pass1.c it just takes forever for large sized partitions > >> and also if you have very high number of file

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Benny Lofgren
On 2011-03-31 11.13, Stuart Henderson wrote: > On 2011-03-31, Otto Moerbeek wrote: >> On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: >>> In fsck_ffs's pass1.c it just takes forever for large sized partitions >>> and also if you have very high number of files stored on that >>> part

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Stuart Henderson
On 2011-03-31, Otto Moerbeek wrote: > On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: >> In fsck_ffs's pass1.c it just takes forever for large sized partitions >> and also if you have very high number of files stored on that >> partition (used inodes count goes high). If you really

Re: horribly slow fsck_ffs pass1 performance

2011-03-31 Thread Otto Moerbeek
On Wed, Mar 30, 2011 at 03:45:02PM -0500, Amit Kulkarni wrote: > Hi, > > In fsck_ffs's pass1.c it just takes forever for large sized partitions > and also if you have very high number of files stored on that > partition (used inodes count goes high). > > fsck main limitation is in pass1.c. > >

horribly slow fsck_ffs pass1 performance

2011-03-30 Thread Amit Kulkarni
Hi, In fsck_ffs's pass1.c it just takes forever for large sized partitions and also if you have very high number of files stored on that partition (used inodes count goes high). fsck main limitation is in pass1.c. In pass1.c I found out that it in fact proceeded to check all inodes, but there's