Re: portage tree (Was: Re: reiser4 status (correction))

[Hans Reiser]

Thanks Christian.  You can go ahead and add something to our wiki
pointing to it if you would like.  This might help tide people over
until the repacker ships.

Hans

Re: future r4 maintenance question

[Hans Reiser]

Maciej Sołtysiak wrote:

Hello Hans,

Saturday, July 22, 2006, 8:03:28 PM, you wrote:

  

We are going to give changing the paradigm a try.  The difference
between 4.1-beta and 4.0 is that different plugins are the default, and
the experimental code is in the plugins you see when mounting with the
mount option 4.1-beta.   Let's see if it works in practice.


I Understand. This is good news. Hm, do you think that reiser4's pluggability
is enough to have this single kernel tree (fs/reiser4) for a longer period
of time.

Yes, reiser4 will have a much longer lifetime,  and improvements will
come out in small pieces rather than complete rewrites.  I think we can
add the enhanced semantics one feature at a time.  It was the storage
layer that was the big thing that had to be right before the rest could
proceed.  Now, what remains are a whole lot of incremental improvements
I hope.  If I can make enough money off the repacker or I get funding
from France or some other government and we are able to afford to keep
vs and zam working  on the storage layer, and Nate working on VFS
changes and Peter Foldiak and others at St. Andrews doing semantic
enhancements and Gorazd doing user space browsers things could get very
interesting.  We need to get into the kernel, and money will
manifest, and programmers can get to work

 I mean, can you predict a need of spawning something like reiser5
in the forseeable future or would fs/reiser4 + plugins be enough to do
away with the future vision and other future * stuff you've written
about ? eg. I remember reading about very granular security ACLs like
restricting a certain line in a file (like /etc/passwd)
  

Yes, I still want to do that stuff

Hans

Re: reiser4 status (correction)

[Hans Reiser]

Mike Benoit wrote:

On Sat, 2006-07-22 at 07:34 -0500, David Masover wrote:

  

The compression will probably mostly be about speed.  Remember, if
we're 
talking about people who want to see tangible, visceral results,
we're 
probably also talking about end-users.  And trust me, the vast
majority 
of most of my data (as an end-user) is not very compressible.




Sure, mine too. Between the many gigs of MP3s and movies I have stored
on my HD, only about 10-20GB is the OS, Email, and documents/source
code. Even just compressing that small portion though I could probably
save between 5-10GB. The difference is though I can do a df before, and
a df after, and I can instantly see I got my moneys worth. Same with
encryption. 
  

I am looking forward to the first user email complaining that he
compressed a file stored on reiser4 and it didn't save space (and then
someday maybe even an email saying that he compressed a file and it took
up more space but the user space compressor is sure that it saved space
and he does not understand).:)

With the repacker it is much more difficult (for average users) to time
how long a program takes to load some file (or launch), before the
repacker and after.

I think you are confusing repacking and compression?  Repacking, by
removing seeks, will make it more predictable not less.

 Especially since caching comes in to play. 

Also, according to this little poll on one of the compressed FUSE sites
you linked to, more people are looking to compression for space saving,
then for speed:
http://parallel.vub.ac.be/~johan/compFUSEd/index.php?option=pollstask=resultspollid=31


  

No, mostly we're talking about things like office documents, the 
majority of which fit in less than a gigabyte, and multimedia (music, 
movies, games) which will gain very little from compression.  If 
anything, the benefit would be mostly in compressing software.



less tangible like fragmentation percentages and minor I/O throughput
improvements. I used to work at a large, world wide web hosting company
and I could see making a case to management for purchasing Reiser4
compression would be pretty easy for our shared servers. Instantly
freeing up large amounts of disk space (where .html/.php files were the
vast majority) would save huge amounts of money on disk drives,
especially since most of the servers used RAID1 and adding new drives
was a huge pain in the neck. Making a case to purchase a repacker would
be much, much more difficult.
  

Hmm, the problem is, if storage space is really the big deal, it's been 
done before, and some of these efforts are still usable and free:

http://parallel.vub.ac.be/~johan/compFUSEd/
http://www.miio.net/fusecompress/
http://north.one.pl/~kazik/pub/LZOlayer/
http://apfs.humorgraficojr.com/apfs_ingles.html

And while we're on the topic, here's an FS that does unpacking of 
archives, probably about the same way we imagined it in Reiser4 
pseudofiles/magic:

http://www.nongnu.org/unpackfs/

But regardless, as far as I can tell, the only real, tangible benefit of 
using Reiser4 compression instead of one of those four FUSE filesystems 
is speed.  Reiser4 would compress/decompress when actually hitting the 
disk, not just the FS, and it would also probably use in-kernel 
compression, rather than calling out to userspace on every FS operation.


I think that compressing only on flush is a big issue.  It was a lot
harder to code it, but it really makes a difference.  You don't want a
machine with a working set that fits into RAM to be compressing, that
would be lethal to performance, and it is a very important case.

Hans

Re: future r4 maintenance question

[Hans Reiser]

I am going to do the enhanced semantics first, so that somebody does not
beat me to it.

David's examples are good.


 There's another note to kernel developers -- if Reiser5, 6, and 7 are
 implemented as suites of plugins on top of Reiser4, then the Reiser4
 code will be maintained for a very long time.  Kind of like ext2 vs
 ext3, only moreso -- a Reiser5 FS may well be a Reiser4 FS mounted
 with additional mount options.

 There is definitely a lot that can be done to move Reiser4 (as it is
 today) closer to the Reiser4 whitepaper on the homepage.  ACLs are one
 thing, files as a directory are another.  The idea of v4 is to do away
 with many cases where a separate namespace is created for no good
 reason -- for instance, where is the data in an id3 tag?  It's inside
 an mp3 file, and you can only get to it with tools written for id3
 tags of mp3 files.  The Reiser4 concept is to allow things like that
 to exist, but not require programs to know about libid3 or whatever. 
 Want to know what the artist of a particular file is?

 foo.mp3/.../id3/artist

 Or maybe a more generic way:

 foo.mp3/.../song-info/artist

 That way, you could have tools which don't even have to know if the
 file has an id3 tag, or something entirely new, or if the metadata is
 being stored outside the main file.  It'd be entirely possible to
 allow that file to be treated as a separate file entirely by the
 plugin, rather than something derived from foo.mp3.

 The advantages don't seem immediately obvious until you consider that
 the program which does this doesn't have to even know that it's
 dealing with song metadata.  Consider some of the one-line shell
 scripts possible:


 # Change the artist name for all songs in the directory:
 for i in *; do echo 'Jimi Hendrix'  foo.mp3/.../song-info/artist; done

 # Make a playlist of all files by Hendrix, mp3 or otherwise:
 for i in `find`; do
 if [ `cat $i/.../song-info/artist` == 'Jimi Hendrix' ]; then
 echo ../$i  playlists/hendrix.m3u;
 fi;
 done

 # Copy all files needed by said playlist to a USB device:
 cp playlist/hendrix.m3u/.../files/* /mnt/usb
 cp playlist/hendrix.m3u /mnt/usb


 I'm sure others can think of much more interesting examples.

 All of that is planned for v4, eventually.  It's very pluggable.

 Well, I think it is.  I don't work here...

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Hans Reiser]

Jeff, I think that a large part of what is going on is that any patch
that can be read in 15 minutes gets reviewed immediately, and any patch
that is worked on for 5 years and then takes a week to read gets
neglected.  This is true even if line for line the 1 week to read patch
is more valuable.What is more is that people know this is
irrational, but aren't able to cure it in themselves.  Even I have a
problem of paying too much attention to endless 5 minute emails when I
know I should instead, say, read the compression plugin from beginning
to end.

There is nothing about small patches that makes them better code.  There
is no reason we should favor them, if the developers are willing to work
on something for 5 years to escape a local optimum, that is often the
RIGHT thing to do.

It is importand that we embrace our diversity, and be happy for the
strength it gives us.  Some of us are good at small patches that evolve,
and some are good at escaping local optimums.  We all have value, both
trees and grass have their place in the world.




 With you in particular, you demonstrated NO interest in maintaining
 reiser3, once reiser4 began to make a splash.  Linux kernel code
 exists for DECADES, and as such, long term maintenance is a CRITICAL
 aspect of development.

You are rejecting the development model which is based on stable
branches getting only bugfixes.  V3 is a stable branch.  It just had a
feature added to it which added a bug that MythTV users are hitting. 
Some of them are responding to it by walking away from Reiser3, and no
doubt muttering about what an unstable pile of shit our code is.  On
monday one of my guys is stopping work on V4 to send in a bug fix for a
feature that should have gone into V4 first, and then maybe gotten
backported after it was proven in V4.

So, given that Jeff and Chris can often be gotten to fix bugs, do I ask
them to do it whenever there is a bug to fix and they will fix it?  Oh
yes!  The despiriting thing though is that there is usually another
reason to let them fix it, which is that almost all v3 bugs are in
features they have added to what ought to have been a stable branch, and
since it is their code, they should be the ones to fix it.  We might,
maybe, get one bug report a year in code written by Namesys before  I
announced code freeze on V3.

I just got an email from the programmer who wrote the MythTV bug saying
that he is just too busy to bother fixing the bug in his code.  so
my response is that a Namesys programmer is going to fix it on Monday.

All this talk about how you guys worry that code is going to be
abandoned, you know, try policing the kids in their 20's who do it, not
those who have been working since 1984 on developing the thing you
somehow are worried they will abandon.  I am not 20 something anymore, I
am getting fat no matter how much I exercise, and I stick with things,
and I only wish some things didn't stick so much with my middle


 Regardless of whatever new whiz-bang technology exists in reiser4,
 there is a very real worry that you will abandon reiser4 once its in
 the tree for a few years, just like what happened with reiser3.

And look at how Linus abandoned 2.4!  Users of 2.4 needed so many
features that were put into 2.6 instead, and they were just abandoned
and neglected and  Do you think he will abandon 2.6.18 also?

The stable branch of code getting only bugfixes and the development
branch getting all the new features model of development is something
most release management professionals agree is the right way to do
things.  I worked with release management teams some, and I have to say
that the dominant paradigm in the software industry is, in this case,
the best one yet.

Of course, I want to make it a little better, you know how I am, and as
I was just discussing on the reiserfs-list, with plugins we can now move
to a model in which if you mount reiser4 using the -o reiser4.1-beta
mount option, it changes what the default plugin is, and that is how we
do releases, we put our beta code in different plugins, and let the user
choose whether to upgrade to a new release by just choosing what plugins
to use as his default.  Now that we paid the 5 year development price
tag to get everything as plugins, we can now upgrade in littler pieces
than any other FS.  Hmm, I need a buzz phrase, its not extreme
programming, maybe moderate programming.  Does that sound exciting to
others.;-)  Seriously though, I am curious to see whether plugin based
release management works out as pleasantly for users as I am hoping it will.

Hans

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Matt Heler]

On Sunday 23 July 2006 12:20 am, Hans Reiser wrote:
 I just got an email from the programmer who wrote the MythTV bug saying
 that he is just too busy to bother fixing the bug in his code.  so
 my response is that a Namesys programmer is going to fix it on Monday.

The way you wrote this, makes it sound like a userspace issue, and _not_ a 
problem with reiserfs.

 And look at how Linus abandoned 2.4!  Users of 2.4 needed so many
 features that were put into 2.6 instead, and they were just abandoned
 and neglected and  Do you think he will abandon 2.6.18 also?

Not entirely true, he did not abandon the 2.4 kernel branch, he passed on 
maintainership to Marcelo. Similar to how he passed the torch on the 2.2 
kernel branch to Alan Cox. Also on a side note, many new features ( and a ton 
of bug fixes !! ) were added to the 2.4 series _after_ Linus started working 
on the 2.5 branch.

Re: future r4 maintenance question

[Sander Sweers]

On Sat, 2006-07-22 at 14:42 -0500, David Masover wrote:
 Mike Benoit wrote:
 
[...]
 
 You would hope.  The problem is, most of the ideas we have for 
 amazing/useless/practical plugins have already been done as specialized 
 filesystems in FUSE.  What's more, FUSE doesn't usually require root 
 privileges to run, causing some people to do such useless things as 
 support for various filesystems already in the kernel (so you can 
 loop-mount filesystems without using the loop device or having root 
 privileges).
 
 Although, there is one other thing to consider -- NTFS in the kernel has 
 stagnated for something like 10 years without write support, while 
 userspace tools like ntfsck and resize_ntfs, using libntfs, have 
 actually gotten relatively stable at dealing with all kinds of ntfs 
 oddities.  For awhile, we had captive-ntfs (also done in FUSE), but more 
 recently, someone wrote a FUSE driver for libntfs, making the first 
 reasonably stable fully featured read/write NTFS driver for Linux.
 
 It's also reasonably fast, sometimes faster than ext2/3, I'm told.

I am using the official libntfs fuse module but it is not all that fast
if you compare how much more cpu power is needed. And even then it is
NOT faster then ext2/3. I am copying a couple of 650mb iso's and I have
100% cpu utilisation and it still takes tens of minutes to copy just 1.

Fuse is nice for things like gmailfs or beaglefs but if you really want
performance, kernelspace imho is the only place to implement a
filesystems.

Greets
Sander

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Jan-Benedict Glaw]

On Sun, 2006-07-23 01:20:40 -0600, Hans Reiser [EMAIL PROTECTED] wrote:
 There is nothing about small patches that makes them better code.  There

Erm, a small patch is something which should _obviously_ fix one
issue. A small patch, containing at max some 100 lines, can easily be
read and understood.

A complete filesystem (I'm co-maintaining one for an ancient on-disk
format, too) isn't really easy to understand or to verify from looking
at it for 5min.

 is no reason we should favor them, if the developers are willing to work
 on something for 5 years to escape a local optimum, that is often the
 RIGHT thing to do.

I give a shit of nothing to some 5 year work if I cannot verify that
it won't hurt me at some point.

 It is importand that we embrace our diversity, and be happy for the
 strength it gives us.  Some of us are good at small patches that evolve,
 and some are good at escaping local optimums.  We all have value, both
 trees and grass have their place in the world.

Just put reiser4 in some GIT tree and publish it. Maybe you can place
it on git.kernel.org .

MfG, JBG

-- 
   Jan-Benedict Glaw   [EMAIL PROTECTED]+49-172-7608481
 Signature of: ...und wenn Du denkst, es geht nicht mehr,
 the second  :kommt irgendwo ein Lichtlein her.


signature.asc
Description: Digital signature

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Toby Thain]

On 23-Jul-06, at 7:48 AM, Jan-Benedict Glaw wrote:

On Sun, 2006-07-23 01:20:40 -0600, Hans Reiser [EMAIL PROTECTED]  
wrote:
There is nothing about small patches that makes them better code.   
There


Erm, a small patch is something which should _obviously_ fix one
issue. A small patch, containing at max some 100 lines, can easily be
read and understood.

A complete filesystem (I'm co-maintaining one for an ancient on-disk
format, too) isn't really easy to understand or to verify from looking
at it for 5min.


Nonetheless, There is nothing about small patches that makes them  
better code.


Hans is quite right. Long patches just take longer to read. This can  
make them harder to penetrate review, as he describes, with analogy.




is no reason we should favor them, if the developers are willing  
to work

on something for 5 years to escape a local optimum, that is often the
RIGHT thing to do.


I give a shit of nothing to some 5 year work if I cannot verify that
it won't hurt me at some point.


Do you really review all patches to ensure this? It is well  
understood that only once r4 reaches mainline will it get the wider  
testing it must have to shake down.


Lucky Namesys is not deterred by ingratitude or there would be no 5  
year work for us to contemplate at all.





It is importand that we embrace our diversity, and be happy for the
strength it gives us.  Some of us are good at small patches that  
evolve,
and some are good at escaping local optimums.  We all have value,  
both

trees and grass have their place in the world.


Just put reiser4 in some GIT tree and publish it. Maybe you can place
it on git.kernel.org .


Why should Hans give up the aspiration to have r4 in mainline due to  
a small number of regressive personalities (a.k.a. politics)?


To much of the Linux world R3 has been an extremely valuable  
contribution; r4 promises to be even more so.


--T



MfG, JBG

--
   Jan-Benedict Glaw   [EMAIL PROTECTED] 
+49-172-7608481
 Signature of: ...und wenn Du denkst, es geht  
nicht mehr,
 the second  :kommt irgendwo ein  
Lichtlein her.

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Jeff Mahoney]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hans Reiser wrote:
 I just got an email from the programmer who wrote the MythTV bug saying
 that he is just too busy to bother fixing the bug in his code.  so
 my response is that a Namesys programmer is going to fix it on Monday.


Hans -

I'll accept blame when it's my bug, but the MythTV one isn't. I've been
working with the bitmap code and did the analysis to track down what was
happening, but that doesn't make it a bug in my code.

That particular bug isn't in the bitmap scanning code, it's a side
effect of the write batching higher up. It's looking for a window of 32
blocks, and there's just no window that large available. It ends up
scanning all the bitmaps looking for the window, and then backs off to
single block allocations. The scanning code works fine, and it does skip
where there aren't enough free blocks available in a particular bitmap.

It's a pathological case when the file system is seriously fragmented. A
quick fix would be to set a flag indicating that future writes shouldn't
bother trying to find a window that large, but that's a hack. A better
allocation algorithm would keep track of free space extents in memory,
subject to getting dropped by memory pressure. Since that information
would be separate from the bitmaps themselves, we could get rid of that
nasty is this block free, but in the journal? check that we need to do
as well. It's invasive, and a quicker fix would just be to track the
largest window, and rescan when it gets used or a block in that bitmap
gets freed.

That said, I actually did start work on a fix for this one, but I really
just don't have the time right now.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFEw+BBLPWxlyuTD7IRAtG8AKCOWW/AH3NAen6gd6BToJGVfzdnNACfYkVS
j2/6yAAeWKAhs4ng9fdGW0Y=
=gB+v
-END PGP SIGNATURE-

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Hans Reiser]

Jeff Mahoney wrote:



 That particular bug isn't in the bitmap scanning code, it's a side
 effect of the write batching higher up.

Did you write the code that looks for a window of 32
blocks?  If not, and if this code has been around for a long time, I
apologize.   I thought you did write it and added it in recent months.



 It's a pathological case when the file system is seriously fragmented.

Most bugs are pathological cases.;-)

 A
 quick fix would be to set a flag indicating that future writes shouldn't
 bother trying to find a window that large,

There are lots of quick fixes.  1) The quickest is to not scan for the
window at all.  2) The second quickest is to limit the number of bitmaps
that will be scanned to some number like 3.  3) The not at all quickest
is to track free extents like XFS does, which is not a hack, but it
belongs in a development branch.  I am not sure it is worth the
complexity, but my mind is not closed.

On monday we will do 1) or 2), probably 1).   After the repacker is
done, we should review all our block allocation algorithms.  I have an
idea for how to do things more optimally for streaming media that will
avoid fragmentation over time, and when combined with the repacker may
make 3 not worthwhile.

I am grateful that you and Chris do bug fixes, but when you guys are too
busy, (and that can and will happen to any of us), the baton needs to
get passed.  V3 needs to be a zero defect product, and once we know it
is a bug I don't want bugs in V3 to remain unfixed for more than a day
plus the time it takes to fix it.If you do add code, I want any bugs
that show up in the aftermath of mainstream merging to get jumped on.

Hans

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Jeff Mahoney]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hans Reiser wrote:
 Jeff Mahoney wrote:
 

 That particular bug isn't in the bitmap scanning code, it's a side
 effect of the write batching higher up.
 
 Did you write the code that looks for a window of 32
 blocks?  If not, and if this code has been around for a long time, I
 apologize.   I thought you did write it and added it in recent months.

Nope. The scan_bitmap_block() code that looks for windows was added in
a changeset from a bk merge against 2.5.33 in September 2002. The change
to want a minimum size window was added in June 2004 to 2.6.8-rc2. That
patch is actually credited to both Mason and I, but I don't recall who
wrote that bit. It may well have been my code, after all, but it's
certainly not a new bug. *shrug* I guess MythTV might just be generating
an i/o pattern that hadn't been seen before.

 A
 quick fix would be to set a flag indicating that future writes shouldn't
 bother trying to find a window that large,
 
 There are lots of quick fixes.  1) The quickest is to not scan for the
 window at all.  2) The second quickest is to limit the number of bitmaps
 that will be scanned to some number like 3.  3) The not at all quickest
 is to track free extents like XFS does, which is not a hack, but it
 belongs in a development branch.  I am not sure it is worth the
 complexity, but my mind is not closed.
 
 On monday we will do 1) or 2), probably 1).   After the repacker is
 done, we should review all our block allocation algorithms.  I have an
 idea for how to do things more optimally for streaming media that will
 avoid fragmentation over time, and when combined with the repacker may
 make 3 not worthwhile.

If you want to go the 1) route, it's trivial. See patch below. It will
restore the one-block-at-a-time behavior.

 I am grateful that you and Chris do bug fixes, but when you guys are too
 busy, (and that can and will happen to any of us), the baton needs to
 get passed.  V3 needs to be a zero defect product, and once we know it
 is a bug I don't want bugs in V3 to remain unfixed for more than a day
 plus the time it takes to fix it.If you do add code, I want any bugs
 that show up in the aftermath of mainstream merging to get jumped on.

Anyone up for it? :) There are changes I'd like to see in reiser3,
particularly ones that address the severe problems observed in David
Chinner's high bandwidth file system talk this year at OLS. Specifically,
it ended up making very little progress and spending the majority of the
time in the journal when the workload is streaming data at the disk at a
very high rate on a very large file system. Yes, that is certainly XFS's
sweet spot, but barely making progress at all is a bit more severe than
poor performance. Perhaps mkreiserfs should be a bit saner about choosing
journal sizes, since a 32 MB journal is not a good fit for all cases. Also,
I'd like to see the usage of the BKL gone as it severely limits performance
when more than one thread is writing to the file system, or even another
reiserfs file system. It's not entirely low hanging fruit since the nested
cases need to be audited, but it shouldn't be too hard to eliminate the
inter-filesystem lock contention by replacing the BKL with a per-sb mutex.
I have some more things, but I have nowhere near the time to do them,
and other file systems will perform fine.

- -Jeff

Patch:

- --- linux-2.6.17.orig/fs/reiserfs/bitmap.c2006-01-02 22:21:10.0 
-0500
+++ linux-2.6.17.orig.devel/fs/reiserfs/bitmap.c2006-07-23 
19:10:57.0 -0400
@@ -1020,7 +1020,6 @@
b_blocknr_t finish = SB_BLOCK_COUNT(s) - 1;
int passno = 0;
int nr_allocated = 0;
- - int bigalloc = 0;
 
determine_prealloc_size(hint);
if (!hint-formatted_node) {
@@ -1047,28 +1046,9 @@
hint-preallocate = hint-prealloc_size = 0;
}
/* for unformatted nodes, force large allocations */
- - bigalloc = amount_needed;
}
 
do {
- - /* in bigalloc mode, nr_allocated should stay zero until
- -  * the entire allocation is filled
- -  */
- - if (unlikely(bigalloc  nr_allocated)) {
- - reiserfs_warning(s, bigalloc is %d, nr_allocated %d\n,
- -  bigalloc, nr_allocated);
- - /* reset things to a sane value */
- - bigalloc = amount_needed - nr_allocated;
- - }
- - /*
- -  * try pass 0 and pass 1 looking for a nice big
- -  * contiguous allocation.  Then reset and look
- -  * for anything you can find.
- -  */
- - if (passno == 2  bigalloc) {
- - passno = 0;
- - bigalloc = 0;
- - }
switch (passno++) {
case 0: /* Search from hint-search_start to end 

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Hans Reiser]

Jeff Mahoney wrote:



 Anyone up for it? :) There are changes I'd like to see in reiser3,
 particularly ones that address the severe problems observed in David
 Chinner's high bandwidth file system talk this year at OLS. Specifically,
 it ended up making very little progress and spending the majority of the
 time in the journal when the workload is streaming data at the disk at a
 very high rate on a very large file system. Yes, that is certainly XFS's
 sweet spot, but barely making progress at all is a bit more severe than
 poor performance. Perhaps mkreiserfs should be a bit saner about
 choosing
 journal sizes, since a 32 MB journal is not a good fit for all cases.
 Also,
 I'd like to see the usage of the BKL gone as it severely limits
 performance
 when more than one thread is writing to the file system, or even another
 reiserfs file system. It's not entirely low hanging fruit since the nested
 cases need to be audited, but it shouldn't be too hard to eliminate the
 inter-filesystem lock contention by replacing the BKL with a per-sb mutex.

Getting rid of the BKL is a huge task that was done in V4 for a reason. 
You are talking about 6+ man-months, and years of shake-out to fully
debug.  Actually, it is a tribute to Zam's skill that V4's locking got
debugged so fast: I gave him the task knowing it was going to be the
hardest code to debug, and he did it very well.

These things you discuss, except for the journal size, are not things to
fix in a stable branch.

My apologies that I thought this was a new bug.  Let us be glad that a
user gave us enough detail we saw it.

 I have some more things, but I have nowhere near the time to do them,
 and other file systems will perform fine.

Re: the 'official' point of view expressed by kernelnewbies.org regarding reiser4 inclusion

[Hans Reiser]

Matt Heler wrote:

On Sunday 23 July 2006 12:20 am, Hans Reiser wrote:
  


The way you wrote this, makes it sound like a userspace issue, and _not_ a 
problem with reiserfs.
  

It was a problem with reiserfs.  Code was added to search for the
perfect spot to fit a file.  If there is no perfect spot, it searches
every bitmap for that spot before giving up.  However, Jeff kindly gave
us a little patch to fix this and made the whole issue moot.  It also
seems I was in error, and we actually have had this problem since 2002. 
Now some past remarks from users about fragmentation make more sense. 
What can I say, since I have no MP3s I never get anywhere near full on
my personal hard drive.

  

And look at how Linus abandoned 2.4!  Users of 2.4 needed so many
features that were put into 2.6 instead, and they were just abandoned
and neglected and  Do you think he will abandon 2.6.18 also?



Not entirely true, he did not abandon the 2.4 kernel branch, he passed on 
maintainership to Marcelo. Similar to how he passed the torch on the 2.2 
kernel branch to Alan Cox. Also on a side note, many new features ( and a ton 
of bug fixes !! ) were added to the 2.4 series _after_ Linus started working 
on the 2.5 branch.
  

You missed the sarcasm in my voice, my apologies, it is the trouble I
have with email.

Just to balance everything with some nuance, let me add that when a
development branch is first opened, there is usually a bit of gray as to
whether particular small features should go into the development branch
or the stable branch.  As the stable branch gets more stable the
incentive to not destabilize it increases, and as a development branch
becomes usable, the delay to users due to putting features only there
reduces.

I want reiserfs to be the filesystem that professional system
administrators view as the one with both the fastest technological pace,
and the most conservative release management.

I apologize to users  that the technology required a 5 year gap between
releases.   It just did, an outsider may not realize how deep the
changes we made were.  Things like per node locking based on a whole new
approach to tree locking that goes bottom up instead of the usual top
down are big tasks.Dancing trees are a big change, getting rid of
blobs is a big change, wandering logs.  We did a lot of things like
that, and got very fortunate with them.  If we had tried to add such
changes to V3, the code would have been unstable the whole 5 years, and
would not have come out right.

Experienced writers know that often, if you want to fix a passage, even
a passage that is quite good in some parts, sometimes it is better to
write the whole passage again without looking at the text of the first
draft of the old passage, because sometimes your muse just needs the
freedom, and without the freedom the awkwardness of the old passage is
incurable.  Probably there is some very sophisticated neurological
reason why that is.  Code can be the same.  Sometimes.  I knew that
reiser4 HAD to be written from scratch without reference to the old code
if it was to come out right. 

If I cannot be a great artist, at least I can try to have the
temperament of one, yes? :-)

I sincerely hope that using mount options to select default plugins, and
making development code go into new plugins means that releases after
this can be roughly quarterly, and that we can start doing a whole bunch
of quick little plugins.  Technically, I think it is going to be
downhill skiing from here, and some very visible bits of functionality
will get added much more easily than this difficult infrastructure we
just coded.

Hans