Re: Which version will be merged into mainline kernel?
Only a fool argues against a fool ;)
Re: Which version will be merged into mainline kernel?
David Masover wrote: Christopher Chan wrote: Anyway I guess you are smarter than Hans then since you obviously feel that his use of fsync is a bit of unneeded integrity. I'm not going to respond to this until you phrase it in a way which doesn't gossip about Hans. Unless something has changed, Hans is still unable to communicate on this list, so it's entirely unfair for either of us to put words in his mouth as long as he isn't here to defend himself. Look, the reason why reiser4 uses fsync is to guarantee data and filesystem integrity which from what I have heard is far better than that of ext3 which is at the moment the leader in this regard. Anyway, this rant went on longer than I wanted it to, but since I've brought it down to a question of economics, I don't really know the answer. I do think, however, that arguing always for or against fsync is absolutist and ultimately wrong. Obviously we should take this question of fsync to its creator and all those who use it since they are all crippling their software implementing this dreadful useless function and actually making use of it. I'm sorry, did I say it was useless? And I quote: "arguing always for OR AGAINST fsync is absolutist and ultimately wrong." I'm not saying fsync is always the wrong choice, either. I'm simply saying it isn't always the right one. When ANY piece of software be it a database, mta or a filesystem wants to know that what was written is safe on disk, it uses fsync, end of story. NO filesystem worth its salt will not not use fsync or fsyncdata when its come to the part of wanting a guarantee that data/metadata has been written to disk. And no, I'm not impressed by what the majority thinks. On a larger scale, the majority thinks Windows is their only real option, and that computers are inherently unstable and insecure, and that software engineers will never listen to them and always make it their fault. The majority really have no choice. Most of the software they need are on Windows. If Linux had come out earlier or GNU had a viable working solution earlier or they had got more exposure, Unix might have been the dominant desktop OS instead of DOS/Windows/M$. I do not believe the majority chose to think Windows is their only real option. You can put in something else and they will happily use it until they find out they cannot use 'necessary software that runs on Windows only' and then they switch back. If you look at how Apple has been rather successful with their desktops/laptops you can see that some people are prepared to dump Windows for what it is, a piece of unstable and insecure crap. Open source too has to share part of the blame when it comes to the desktop. If you're going to debate this, please try to debate it with just you and me. You shouldn't need to invoke Hans-who-isn't-here, or every software engineer in the past 20 years. If your point is really that solid, it should stand on its own, without you having to intimidate me about how stupid I am for questioning established practices. If you had brought up a solid point against fsync instead of completely irrelevant comparisons and sticking to your faulty points then I won't have bothered pointing out that you are being stupid for questioning established practices without grounds. I'd rather drop the debate, because, as I said, we mostly agree. I am not sure we do. Until you drop the fsync is unnecessary for data integrity part, we don't. How could I ever forget that banks use unreliable media? So how do they deal with that issue? (Or are you being sarcastic?) Obviously, fsync won't save them, so how do they deal with it? Come on, David, stop acting like some ignorant admin. Media problems are migitated by having redundant failovers available. I won't speculate on Hans' opinions on whether fsync is really a good design decision -- at least, not without him able to respond. You don't have to. He did not create fsync and so you don't have to worry about whether fsync is a good design decision. He could have as easily removed it, the way I patched it out. Pretended to implement it, but simply have a stub that always returns success. Heh. Did you know that there was a time that Linux pretended to sync when it did not actually do it? The result? fsck, fsck, fsck He certainly has opinions about whether fsync is a good idea, why he wants reiser4 to have better fsync performance, and many other things that we should not discuss in his absence. The only way you can get better fsync performance is to use RAID under the filesystem or do full journaling on a NVRAM block device. Get this through your head. fsync is not written/created by Hans. fsync was already part of the Unix 98, Unix 95, 4.3BSD and SVID 3 APIs. (see http://www.unix.org/apis/1.r.html) I doubt very much that Hans has any opinions whatsoever on a standard API call. So you can stop the Hans is not here nonsense. Until H
Re: Which version will be merged into mainline kernel?
So, you have to have a way to insure data integrity beyond flushing it to disk. Flushing it to disk is just a bit of unneeded integrity, that costs a little bit of performance, and would probably save your ass in one or two edge cases, but those cases shouldn't normally happen, and if you're properly prepared for the hardware failing, you're properly prepared for those edge cases, also. fsync costs a lot of performance. Disk I/O is orders of magnitude off RAM. Anyway I guess you are smarter than Hans then since you obviously feel that his use of fsync is a bit of unneeded integrity. Anyway, this rant went on longer than I wanted it to, but since I've brought it down to a question of economics, I don't really know the answer. I do think, however, that arguing always for or against fsync is absolutist and ultimately wrong. Obviously we should take this question of fsync to its creator and all those who use it since they are all crippling their software implementing this dreadful useless function and actually making use of it. Anyway, are we done here? I don't think I disagree with your points, maybe just your absolutism. Ha! Got tell that to a bank or any body who uses a database to store important data. Hans knows what he is doing providing data guarantee with fsync. Yes, he's providing warm fuzzies for the banks. What are the banks doing about hard disk failure? Remember, fsync only guarantees the data was successfully written to a physical media -- it doesn't guarantee the data will still be there when you want to go read it. How could I ever forget that banks use unreliable media? I won't speculate on Hans' opinions on whether fsync is really a good design decision -- at least, not without him able to respond. You don't have to. He did not create fsync and so you don't have to worry about whether fsync is a good design decision.
Re: Which version will be merged into mainline kernel?
Hi there, First of all this discussion is about ... well isn't it of topic here at all? journaling support has been late on Linux. Everything seems to be late on Linux. Real-time support, preemptive support... Yes Linux is late on everything because things are done when there is _real_ demand for it. I don't see a problem with that ;-) lg Clemens
Re: Which version will be merged into mainline kernel?
Which is why data important software use fsync/fsyncdata to minimize data loss from crashes. If (and only if) you actually read and accepted the paragraph you're quoting there, the point is that using fsync and fdatasync on these is kind of like doing bad block relocation in software, when the hard disk is already doing it for you. There are situations where it's useful, but mostly, it's just a performance drain without a point. You have no idea what fsync/fsyncdata does. They cannot be compared to bad block relocation at all. That is an apples to oranges comparison. I don't necessarily agree with this point of view, but there you go. could probably argue the other side just as effectively. Besides, at least on my experimental/gaming rig, I like a little adrenaline in my admin work! Try arguing why you lost thousands of mails when the box crashed and justifying it. There aren't thousands of mails on my experimental dev/gaming rig. Also, it hasn't crashed in a month or two (I seem to have upgraded/hacked my way out of its last major crasher), and it's running all kinds of experimental stuff. Sure, individual programs crash from time to time (especially my hacks), but that's irrelevant to the fsync discussion. I was not referring to your gaming rig. Did you say that you don't use fsync for your production boxes? So, as far as I can tell, this mailserver (that I'm sending this from right now), which is amd64/reiser4/no_fsync, hasn't lost a single message. And I don't even have an APM on it (although that reminds me, I need to buy one now, as in, "tomorrow" now.) Heh. Just wait till you get a crash shortly after a write with fsync disabled. Anyway, are we done here? I don't think I disagree with your points, maybe just your absolutism. Ha! Got tell that to a bank or any body who uses a database to store important data. Hans knows what he is doing providing data guarantee with fsync. I was very surprised when I heard all those reports about zero data loss on reiser4 boxes after a crash. Now I have an idea why.
Re: Which version will be merged into mainline kernel?
So the bad case is: 1. mails are tiny 2. RAM cache is big 3. cache doesn't get flushed to disk for 1000s of mails 4. crash happens seldomly, but severely :) -> 1000s of mail lost by the crash Now I see :) :). So fsync/fsyncdata takes care of those rare crashes which is why mail servers should rarely lose mail. I say should because the journal mode available/used affects fsync/fsyncdata too.
Re: Which version will be merged into mainline kernel?
While we are at it: I take it that this is the reason journalling support is only picking up now: the disks are so big that even in the unlikely event that some of the hardware failsafes fail, one just cannot fsck all the disks completely anymore, ever. Only picking up now?? reiserfs was the first filesystem available for Linux with journaling support. NTFS for Windows had it a long time ago. Other Unix's had journaling a long time ago too. FreeBSD, Solaris and the other BSD's had another theory for file system reliability and speed implemented under softupdates a long time ago too. journaling support has been late on Linux. Everything seems to be late on Linux. Real-time support, preemptive support... So the choice is really 'borked once, borked forever' / 'journal it and at least somehow get it back online without fscking / copying-to-new-disks-if-at-all-possible for 5 days straight'. No. Choices are: sync only (no write caching), turn on softupdates if supported, turn on various degrees of journaling if supported or async only (who cares?) If you have fsync/fsyncdata available, databases, mtas and other software should not care less how the filesystem is mounted.
Re: Which version will be merged into mainline kernel?
Hi, On Sat, 11 Nov 2006 14:28:42 -0500, Valdis.Kletnieks wrote: [...] > I've never understood this kind of attitude some filesystems have. Usually > the hardware would make sure that stuff doesn't disappear, and not some > weird software workarounds like journalling or write barriers that > complicate and slow down everything. > > Now as you were saying? Hehehe. Well, you turned it all around... insightful :) While we are at it: I take it that this is the reason journalling support is only picking up now: the disks are so big that even in the unlikely event that some of the hardware failsafes fail, one just cannot fsck all the disks completely anymore, ever. So the choice is really 'borked once, borked forever' / 'journal it and at least somehow get it back online without fscking / copying-to-new-disks-if-at-all-possible for 5 days straight'. cheers, Danny
Re: Which version will be merged into mainline kernel?
Hi, On Mon, 13 Nov 2006 17:34:59 +0800, Christopher Chan wrote: > David Masover wrote: >> Christopher Chan wrote: > [...smartdrv...] Yeah, smartdrv was nice :) [...] > Try arguing why you lost thousands of mails when the box crashed and > justifying it. Oh. Good point. So the bad case is: 1. mails are tiny 2. RAM cache is big 3. cache doesn't get flushed to disk for 1000s of mails 4. crash happens seldomly, but severely :) -> 1000s of mail lost by the crash Now I see :) cheers, Danny
Re: Which version will be merged into mainline kernel?
David Masover wrote: Christopher Chan wrote: Danny Milosavljevic wrote: The only possibility one could still lose mail when having proper hardware failsafes would be if the kernel had a bug and crashed (and that's so bad, it doesn't really warrant any working around it). Ever used DOS with smartdrv? smartdrv gave a performance boost by storing recently touched files in memory and writing to disk later. This is called a disk cache. You would be explicitly told to NOT just turn off the computer each time smartdrv was loaded. You had to first clear the cache and then you could power off the box otherwise you would lose any data sitting in the cache that had not been flushed. Which means that, if you have proper hardware failsafes, you will only lose data if you don't properly clear the cache before shutting down (which Linux shutdown scripts will do for you), or if there's a crash. I'm sure many people used smartdrv without problems, and appreciated the performance boost. Which is why data important software use fsync/fsyncdata to minimize data loss from crashes. Moral of the story: Data loss is a Bad Thing. Those who would give up essential data safety for a little temporary performance deserve neither. (With apologies to Ben Franklin.) And while reading this, it's important to keep in mind that I do not practice what I've just preached. I'm addicted to the performance, and I could probably argue the other side just as effectively. Besides, at least on my experimental/gaming rig, I like a little adrenaline in my admin work! Try arguing why you lost thousands of mails when the box crashed and justifying it.
Re: Which version will be merged into mainline kernel?
Danny Milosavljevic wrote: Hi, On Wed, 08 Nov 2006 03:21:36 -0700, Andreas Dilger wrote: On Nov 08, 2006 10:15 +0100, Francesco Biscani wrote: I think the slow performance you're experiencing are caused by the fsync() call not being well-optimized in reiser4. I've commented out the function in fs/buffer.c, and I'm having much better performance on my / partition. I don't think this can be advocated as a real solution to performance problems. This can mean data loss for some applications like email that expect "fsync return" to mean "this data is safely on disk". I've never understood this kind of attitude some MTAs have. Usually the hardware would make sure that stuff doesn't disappear (UPS, powered RAM, harddisk condenser) and not some weird software workaround that complicates and slows down everything. The only possibility one could still lose mail when having proper hardware failsafes would be if the kernel had a bug and crashed (and that's so bad, it doesn't really warrant any working around it). Ever used DOS with smartdrv? smartdrv gave a performance boost by storing recently touched files in memory and writing to disk later. This is called a disk cache. You would be explicitly told to NOT just turn off the computer each time smartdrv was loaded. You had to first clear the cache and then you could power off the box otherwise you would lose any data sitting in the cache that had not been flushed. All current Unices have disk cache capabilities built-in and the only way to bypass the disk cache for those applications that needed to have their data definitely written to disk and not sitting the cache was the fsync and fsyncdata calls.
Re: Which version will be merged into mainline kernel?
On Sat, 11 Nov 2006 15:14:18 GMT, Danny Milosavljevic said: > I've never understood this kind of attitude some MTAs have. Usually the > hardware would make sure that stuff doesn't disappear (UPS, powered RAM, > harddisk condenser) and not some weird software workaround that complicates > and slows down everything. I've never understood this kind of attitude some filesystems have. Usually the hardware would make sure that stuff doesn't disappear, and not some weird software workarounds like journalling or write barriers that complicate and slow down everything. Now as you were saying? pgpHqTi1lKSH4.pgp Description: PGP signature
Re: Which version will be merged into mainline kernel?
Hi, On Wed, 08 Nov 2006 03:21:36 -0700, Andreas Dilger wrote: > On Nov 08, 2006 10:15 +0100, Francesco Biscani wrote: >> I think the slow performance you're experiencing are caused by the fsync() >> call not being well-optimized in reiser4. I've commented out the function in >> fs/buffer.c, and I'm having much better performance on my / partition. > > I don't think this can be advocated as a real solution to performance > problems. This can mean data loss for some applications like email that > expect "fsync return" to mean "this data is safely on disk". I've never understood this kind of attitude some MTAs have. Usually the hardware would make sure that stuff doesn't disappear (UPS, powered RAM, harddisk condenser) and not some weird software workaround that complicates and slows down everything. The only possibility one could still lose mail when having proper hardware failsafes would be if the kernel had a bug and crashed (and that's so bad, it doesn't really warrant any working around it). But maybe that's just me. cheers, Danny
Re: Which version will be merged into mainline kernel?
sorry, my last message was not really clear: I didn't change the kernel, I just disable the preload-scripts executed by fedora. I first tried to inlucde the whole KDE stuff but found it to be even slower so I disabled them completly and now everything works fine :-) I guess this could be maybe because of the reduced space available for caching in RAM, because compressed and uncompressed data is cached. lg Clemens 2006/11/9, Clemens Eisserer <[EMAIL PROTECTED]>: Hi again, Don't know what its worth: I disabled readahead completly and now KDE starts almost as fast as on EXT3. lg Clemens
Re: Which version will be merged into mainline kernel?
Hi again, Don't know what its worth: I disabled readahead completly and now KDE starts almost as fast as on EXT3. lg Clemens
Re: Which version will be merged into mainline kernel?
Hi again, > I think the slow performance you're experiencing are caused by the fsync() > call not being well-optimized in reiser4. I've commented out the function in > fs/buffer.c, and I'm having much better performance on my / partition. I don't think this can be advocated as a real solution to performance problems. This can mean data loss for some applications like email that expect "fsync return" to mean "this data is safely on disk". May as well say "I improved the performance of my backups by writing to /dev/null". To be honest I don't really know what fsync does (I guess it forces something to be written or something like this) and if yes - why it gets called that often. However am I wrong or is fsync mostly calles by applications writing data, whereas starting xorg+kde is mostly a read job? Thank you for your patience, lg Clemens
Re: Which version will be merged into mainline kernel?
On 11/8/06, Clemens Eisserer <[EMAIL PROTECTED]> wrote: Hello again, Thanks a lot for all the help, well ... I now use reiser4 as root-partition :-) Everything works ok and I haven't seen any crashes or compatibility problems, simply wonderful :-). If something bad happens I'll let you know ;) However performance is so-and-so, the system was faster with ext3, although I compiled the kernel with "-O3 -march=pentium4 -mtune=pentium4 -mno-sse -mno-sse3". Booting the system into KDE takes about twice the time, although I have to admit I was using another harddrive - I now use a 60GB/4200RPM instead an 40GB/4200RPM which both show quite the same hdparm-results, I use the lzo1 plugin (default ccreg40). Are there some settings (or ext3 optimizations) in FC6, or could it be that some of my kernel-settings slow things down for FC6 (the same compiled kernel-binary was already used for ext3). The reiser4-partition is mounted with "noatime". These are some features I could imagine causing problems: * Inotify * Adaptive file readahead + hit feedback + fine grained readahead aging * Timer with 1kHz * Readahead-script of the distribution noatime and nodiratime mount options can also improve Reiser4 performance a lot. better solution for the fsync problem is a library like http://ftp.die.net/pub/qmail-tools/libnosync.c , which can be used only when needed. -- Jindrich Makovicka
Re: Which version will be merged into mainline kernel?
On Nov 08, 2006 10:15 +0100, Francesco Biscani wrote: > I think the slow performance you're experiencing are caused by the fsync() > call not being well-optimized in reiser4. I've commented out the function in > fs/buffer.c, and I'm having much better performance on my / partition. I don't think this can be advocated as a real solution to performance problems. This can mean data loss for some applications like email that expect "fsync return" to mean "this data is safely on disk". May as well say "I improved the performance of my backups by writing to /dev/null". Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Re: Which version will be merged into mainline kernel?
On Wednesday 08 November 2006 08:21, Clemens Eisserer wrote: > Hello again, > > Thanks a lot for all the help, well ... I now use reiser4 as root-partition > :-) > > Everything works ok and I haven't seen any crashes or compatibility > problems, simply wonderful :-). If something bad happens I'll let you > know ;) > > However performance is so-and-so, the system was faster with ext3, > although I compiled the kernel with "-O3 -march=pentium4 > -mtune=pentium4 -mno-sse -mno-sse3". Booting the system into KDE takes > about twice the time, although I have to admit I was using another > harddrive - I now use a 60GB/4200RPM instead an 40GB/4200RPM which > both show quite the same hdparm-results, > I use the lzo1 plugin (default ccreg40). > > Are there some settings (or ext3 optimizations) in FC6, or could it be > that some of my kernel-settings slow things down for FC6 (the same > compiled kernel-binary was already used for ext3). The > reiser4-partition is mounted with "noatime". > > These are some features I could imagine causing problems: > * Inotify > * Adaptive file readahead + hit feedback + fine grained readahead aging > * Timer with 1kHz > * Readahead-script of the distribution I think the slow performance you're experiencing are caused by the fsync() call not being well-optimized in reiser4. I've commented out the function in fs/buffer.c, and I'm having much better performance on my / partition. HTH, Francesco -- Dr. Francesco Biscani Dipartimento di Astronomia Università di Padova [EMAIL PROTECTED]
Re: Which version will be merged into mainline kernel?
Hello again, Thanks a lot for all the help, well ... I now use reiser4 as root-partition :-) Everything works ok and I haven't seen any crashes or compatibility problems, simply wonderful :-). If something bad happens I'll let you know ;) However performance is so-and-so, the system was faster with ext3, although I compiled the kernel with "-O3 -march=pentium4 -mtune=pentium4 -mno-sse -mno-sse3". Booting the system into KDE takes about twice the time, although I have to admit I was using another harddrive - I now use a 60GB/4200RPM instead an 40GB/4200RPM which both show quite the same hdparm-results, I use the lzo1 plugin (default ccreg40). Are there some settings (or ext3 optimizations) in FC6, or could it be that some of my kernel-settings slow things down for FC6 (the same compiled kernel-binary was already used for ext3). The reiser4-partition is mounted with "noatime". These are some features I could imagine causing problems: * Inotify * Adaptive file readahead + hit feedback + fine grained readahead aging * Timer with 1kHz * Readahead-script of the distribution Thanks a lot, lg Clemens
Re: Which version will be merged into mainline kernel?
Hi again, How much RAM? 512mb, DDR1 "PC2700" Thanks, but the status of this account seems to be unclear Ok, then I'll wait till there are other ways to donate. Thanks a lot, Clemens
Re: Which version will be merged into mainline kernel?
Hello, ok, a bit later No need to hurry ;) both compressed and decompressed data are cached, it means that cryptcompress file requires *(2-R) more memory then usual file (R - compression ratio) Oh ... thats a drawback. It means the performance advantage compression will do will be destroyed at least partially :-/ However I am sure there are fine reasons to do so (or the current VFS design forces to do so) and I also don't understand the background. I am happy with what I get :-) What is hardware configuration of your laptop? Its a Northwood-P4-2.6ghz with a very slow (Hitachy?) drive: "Timing buffered disk reads: 48 MB in 3.08 seconds = 15.59 MB/sec" Thanks a lot for this great FS, lg Clemens Btw. If you send me the setup - this could be seen as support service ;) Would it help Reiser4 development if I pay the 25$ - or is the locked?
Re: Which version will be merged into mainline kernel?
Hi, If you want to try out in the latest -mm kernel, then we will send you the setup. I'll setup my laptop within the next days and will compile the latest -mm kernel. Could you send me the "setup", please? (Do you mean the kernel-setup, or a howto?) Thanks a lot in advance, lg Clemens Btw. a very last question about compression: Will the disk-cache held in RAM be also compressed? Would be great for machines with slow disk, little RAM but fast processor... (e.g. my laptop ;) )
Re: Which version will be merged into mainline kernel?
Hello Clemens, Tuesday, October 17, 2006, 11:14:33 PM, you wrote: > somewhere that first 2.6.19 was the goal but now it is 20 or 21. Is it > now more or less a predictable thing, only when exactly is still open > or can kernel devs still change their opinion? It is largely up to Andrew Morton. I trust him and I think that when he says r4 is ready, it is ready. Andrew keeps knows that there are both sides of the story here and he does his best to push r4 into mainline while keeping to it that the code adhers to requirements and guidelines stated by kernel developers. I belive r4 is soon to be merged, if not in december/january then surely before easter. (would be a good xmas present,hehe) Regards, Maciej
Re: Which version will be merged into mainline kernel?
Hi again and thanks for the response, > Will reiser-4.0 or reiser-4.1 be merged into mainline kernel, I ask I don't know, why it should be merged partially. I.e. "all or nothing". I thought that compression belongs exclusivly to reiser4-4.1, and maybe because of stability issues only reiser4-4.0 will maybe merged. Great to hear to maybe soon get the full power at once :-) If you want to try out in the latest -mm kernel, then we will send you the setup. I am currently really busy - I've just started university. But I would be happy if I would be allowed to ask for help again in a few weeks :-) Btw. Is there anything new about includion into mainline? I read somewhere that first 2.6.19 was the goal but now it is 20 or 21. Is it now more or less a predictable thing, only when exactly is still open or can kernel devs still change their opinion? lg Clemens
Which version will be merged into mainline kernel?
Hello, Will reiser-4.0 or reiser-4.1 be merged into mainline kernel, I ask because I am quite interrested in trying out the compression plugin. Sorry about what happend in the last few days :-/ lg Clemens
